[
https://issues.apache.org/jira/browse/CASSANDRA-10371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15148893#comment-15148893
]
pavel commented on CASSANDRA-10371:
-----------------------------------
Logs. Dead node is 10.0.1.32
{noformat}
{"time":"2016-02-16T16:31:24.209Z","level":"TRACE","msg":"requestAll for
/10.0.1.32"}
{"time":"2016-02-16T16:31:24.210Z","level":"TRACE","msg":"Received a
GossipDigestAck2Message from /10.0.2.128"}
{"time":"2016-02-16T16:31:24.210Z","level":"TRACE","msg":"reporting
/10.0.2.128"}
{"time":"2016-02-16T16:31:24.210Z","level":"TRACE","msg":"/10.0.2.128local
generation 1454668510, remote generation 1454668510"}
{"time":"2016-02-16T16:31:24.210Z","level":"TRACE","msg":"Updating heartbeat
state version to 2947566 from 2947563 for /10.0.2.128 ..."}
{"time":"2016-02-16T16:31:24.210Z","level":"TRACE","msg":"Ignoring gossip for
/10.0.1.32 because it is quarantined"}
{"time":"2016-02-16T16:31:24.351Z","level":"TRACE","msg":"My heartbeat is now
2947086"}
{"time":"2016-02-16T16:31:24.351Z","level":"TRACE","msg":"Gossip Digests are :
/10.0.1.88:1454668666:2947086 /10.0.1.129:1454674252:2930159
/10.0.2.128:1454668510:2947566 /10.0.1.12:1454675536:2926261
/10.0.2.117:1454676714:2922695 /10.0.2.168:1455284954:1077795 "}
{"time":"2016-02-16T16:31:24.351Z","level":"TRACE","msg":"Sending a
GossipDigestSyn to /10.0.1.12 ..."}
{"time":"2016-02-16T16:31:24.351Z","level":"TRACE","msg":"Performing status
check ..."}
{"time":"2016-02-16T16:31:24.351Z","level":"TRACE","msg":"PHI for /10.0.2.128 :
0.13901063851144269"}
{"time":"2016-02-16T16:31:24.351Z","level":"TRACE","msg":"PHI for /10.0.1.129 :
0.21683730070438223"}
{"time":"2016-02-16T16:31:24.351Z","level":"TRACE","msg":"PHI for /10.0.2.168 :
0.2206211272280096"}
{"time":"2016-02-16T16:31:24.351Z","level":"TRACE","msg":"PHI for /10.0.1.12 :
0.2149248608862009"}
{"time":"2016-02-16T16:31:24.352Z","level":"TRACE","msg":"PHI for /10.0.2.117 :
0.2130329987137985"}
{"time":"2016-02-16T16:31:24.352Z","level":"DEBUG","msg":"60000 elapsed,
/10.0.1.32 gossip quarantine over"}
{"time":"2016-02-16T16:31:24.352Z","level":"TRACE","msg":"Received a
GossipDigestAckMessage from /10.0.1.12"}
{"time":"2016-02-16T16:31:24.352Z","level":"TRACE","msg":"Received ack with 2
digests and 0 states"}
{"time":"2016-02-16T16:31:24.352Z","level":"TRACE","msg":"local heartbeat
version 2947086 greater than 2947085 for /10.0.1.88"}
{"time":"2016-02-16T16:31:24.352Z","level":"TRACE","msg":"local heartbeat
version 2947566 greater than 2947565 for /10.0.2.128"}
{"time":"2016-02-16T16:31:24.352Z","level":"TRACE","msg":"Sending a
GossipDigestAck2Message to /10.0.1.12"}
{"time":"2016-02-16T16:31:24.542Z","level":"TRACE","msg":"local heartbeat
version 2926261 greater than 2926258 for /10.0.1.12"}
{"time":"2016-02-16T16:31:24.542Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:24.542Z","level":"TRACE","msg":"local heartbeat
version 2947086 greater than 2947080 for /10.0.1.88"}
{"time":"2016-02-16T16:31:24.542Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:24.542Z","level":"TRACE","msg":"local heartbeat
version 2947566 greater than 2947563 for /10.0.2.128"}
{"time":"2016-02-16T16:31:24.542Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:24.543Z","level":"TRACE","msg":"Received a
GossipDigestAck2Message from /10.0.1.129"}
{"time":"2016-02-16T16:31:24.543Z","level":"TRACE","msg":"reporting
/10.0.1.129"}
{"time":"2016-02-16T16:31:24.543Z","level":"TRACE","msg":"/10.0.1.129local
generation 1454674252, remote generation 1454674252"}
{"time":"2016-02-16T16:31:24.543Z","level":"TRACE","msg":"Updating heartbeat
state version to 2930162 from 2930159 for /10.0.1.129 ..."}
{"time":"2016-02-16T16:31:25.209Z","level":"TRACE","msg":"local heartbeat
version 2930162 greater than 2930159 for /10.0.1.129"}
{"time":"2016-02-16T16:31:25.209Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:25.209Z","level":"TRACE","msg":"local heartbeat
version 2947086 greater than 2947085 for /10.0.1.88"}
{"time":"2016-02-16T16:31:25.209Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:25.209Z","level":"TRACE","msg":"requestAll for
/10.0.1.32"}
{"time":"2016-02-16T16:31:25.210Z","level":"TRACE","msg":"Received a
GossipDigestAck2Message from /10.0.2.128"}
{"time":"2016-02-16T16:31:25.210Z","level":"TRACE","msg":"reporting
/10.0.2.128"}
{"time":"2016-02-16T16:31:25.210Z","level":"TRACE","msg":"reporting
/10.0.2.117"}
{"time":"2016-02-16T16:31:25.210Z","level":"TRACE","msg":"/10.0.2.128local
generation 1454668510, remote generation 1454668510"}
{"time":"2016-02-16T16:31:25.210Z","level":"TRACE","msg":"Updating heartbeat
state version to 2947569 from 2947566 for /10.0.2.128 ..."}
{"time":"2016-02-16T16:31:25.210Z","level":"TRACE","msg":"reporting /10.0.1.32"}
{"time":"2016-02-16T16:31:25.210Z","level":"TRACE","msg":"Adding endpoint state
for /10.0.1.32"}
{"time":"2016-02-16T16:31:25.210Z","level":"DEBUG","msg":"Not marking
/10.0.1.32 alive due to dead state"}
{"time":"2016-02-16T16:31:25.211Z","level":"TRACE","msg":"marking as down
/10.0.1.32"}
{"time":"2016-02-16T16:31:25.211Z","level":"INFO","msg":"InetAddress /10.0.1.32
is now DOWN"}
{"time":"2016-02-16T16:31:25.211Z","level":"TRACE","msg":"Notified
[org.apache.cassandra.service.StorageService@2a92d5f9,
org.apache.cassandra.locator.ReconnectableSnitchHelper@63e5499,
org.apache.cassandra.service.LoadBroadcaster@7a3e4f9f]"}
{"time":"2016-02-16T16:31:25.211Z","level":"DEBUG","msg":"Ignoring state change
for dead or unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.211Z","level":"DEBUG","msg":"Ignoring state change
for dead or unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.211Z","level":"DEBUG","msg":"Ignoring state change
for dead or unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.211Z","level":"DEBUG","msg":"Ignoring state change
for dead or unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.211Z","level":"DEBUG","msg":"Ignoring state change
for dead or unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.211Z","level":"DEBUG","msg":"Ignoring state change
for dead or unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.211Z","level":"DEBUG","msg":"Ignoring state change
for dead or unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.211Z","level":"DEBUG","msg":"Ignoring state change
for dead or unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.212Z","level":"DEBUG","msg":"Ignoring state change
for dead or unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.212Z","level":"DEBUG","msg":"Ignoring state change
for dead or unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.212Z","level":"DEBUG","msg":"Node /10.0.1.32 state
left, tokens [-1183123795032699037, -1237172198872744999, -1270149265263593564,
-1389400073696907418, -1402231454860468978, -1420312602012480180,
-1627335600868165275, -1713896876933182948, -1714549883939918780,
-1741474774932965377, -1792210053935902256, -1807601543024917385,
-1854566658540932161, -1912378675308644559, -1963516450452847956,
-2081077201955154820, -2099259848054237265, -2129680696766320789,
-2221883494221453885, -2310616694711668360, -2331548641412448139,
-2334622440252028474, -2411111763982847025, -245865888987490824,
-246385138307167507, -2473543956019538623, -2490169864633659751,
-2532042510174406079, -2540038371895843059, -2577021784711590342,
-2593326939593610701, -2705600331335690795, -2770464252902532656,
-2839608713233822191, -2912475620975565625, -2943366603912110567,
-2949295810903492735, -2954629576212069889, -3095752208331422097,
-3121066151673234082, -3185818042633507184, -3219579857569752331,
-3286343234413829618, -3289689335761352230, -3361565551667832402,
-3383031027800371008, -3412816001810159961, -3457683958466098497,
-3508695836421455440, -3532532777067013075, -3543699029518234044,
-3580715661159129955, -3599252547477456660, -3661459590777641370,
-3667538041993955650, -3690186824731493069, -3814115795769795724,
-409080988277749690, -4156007243077529438, -4168774893038102265,
-4213854018229603756, -4249034660629583574, -4281296366845033280,
-4320926472067799077, -4545355489460541553, -4630628418422675470,
-4654983718222492637, -4768547837932983920, -5159507483120816494,
-51965338274886423, -5199723923113410828, -5223817829501441469,
-5267064983314818217, -5348036139422617554, -5364643938379720849,
-5495473223858010026, -5538859875930354329, -554259440667407064,
-5547533646691145209, -5599903631185515707, -5779719603131964183,
-5932457424237549800, -6205324647087801173, -6229187725024886147,
-6231270541967013773, -6235596621005793282, -6243196665962158041,
-6264073286569207959, -6427783773364474000, -6501124205004483449,
-6515615874324067226, -6573166604041811188, -6637220677879215528,
-6651782973192407164, -6660806062319128258, -6745026564913028874,
-6812331251174808386, -7084494716757599690, -7149011211591734620,
-7274798955118474541, -731652465373161574, -7403663263352230504,
-7447942789148217722, -7449824614964680985, -7587057746419387993,
-7641766861615411515, -7685202236673643074, -7707429459189222115,
-7742454891260223055, -7833962528790251697, -7931725553821464164,
-7939773289897241357, -8098012638292660116, -8219265643204532541,
-8235265212011847570, -8252620209307958750, -8332424581430025253,
-8349765834148048802, -836487426771203231, -8505914076204635715,
-851663313261657661, -8519826107865925909, -8695372492829159498,
-8942575831111517062, -8976797677584625277, -9051099539809741903,
-963992846195142625, 1003830146651642498, 1178027849963644975,
1224976256801927215, 1244352078631038965, 1417164721674217220,
1451341591539912399, 1491082438042287951, 1514848536642096395,
1779901662209221643, 1854090933062166699, 1943140502461231414,
1994380979939543148, 2007534562233247106, 217709005954998613,
2235989942794609624, 2241300505721899121, 2246701380267263353,
231183097291070161, 2426913079401922388, 2451699806267654588,
2506037868809622944, 2539177368085335242, 2585691867824138041,
2615916489656464006, 2629622650576085045, 2657709892891143999,
269267015073834956, 2978608649954009655, 2994749269564046800,
3002378570488766614, 3121394566209939036, 3141180417712080576,
3154166401188000671, 3295643071378191280, 3367253256599281972,
3373165743103532333, 3410319489097671220, 341173714064808701,
35319237546365581, 358168701228763255, 3695798865479120909,
3784007090181772945, 3859215587032554422, 388414320421460408,
3958884439975359895, 4116791292047135794, 4157923384507172767,
4177479403175856644, 4192843956567907187, 4237835213950914050,
4283455088657318777, 4437755738583404917, 4464954521637384793,
4479350931740982844, 448168435071015674, 4483893029099418629,
4495279466361191724, 4626894508809678069, 4913067912592229158,
5055224510936602124, 5146899889661568824, 5333227006937586923,
5344713819757688419, 5354664536118782383, 5407856766112305920,
5524147618224097934, 5530799914723849578, 5594217902784391863,
5740424123774088150, 5773813700926694122, 5797583289244413653,
596003237801815323, 6051342740892152639, 6053457919557951562,
6063733117952809262, 6188792405330710479, 6215325692940778409,
6255014464234787541, 6307265543356112469, 6331411861541807584,
6428394292228979654, 6440759262443849168, 6519598944774579329,
6559591672744555181, 6637797466881599715, 6662404025200321225,
6668777379548711499, 6828109123001551836, 6861072227483193626,
6876892389046410599, 6982428538201939129, 7004630263067699041,
703529510861716420, 7035478043169974728, 713068522342900631,
7143695289303246630, 7149047549904215441, 7302396691607427712,
7315761441890909840, 7410347447605570455, 754235745869667971,
7544573809574793144, 7586097788690287546, 7597038064500920869,
7614889355576709597, 7624148922826428440, 7627724472890172915,
7896536432019982357, 8070449930340402428, 8100741102346464407,
8206476157175887120, 8277001814877694995, 8323125797400536081,
8331290716255307824, 8338939509697372178, 834182466138639995,
8388253326997854443, 8414277759665359090, 8670923111306643765,
8769994749009616915, 8770706585661243637, 8812518846508548955,
8847966629758817746, 8906298870427014138, 8987438727642595319,
9050092026849126476, 933979495781858352, 962613924858074837,
985754492835172659]"}
{"time":"2016-02-16T16:31:25.212Z","level":"DEBUG","msg":"adding expire time
for endpoint : /10.0.1.32 (1454932471326)"}
{"time":"2016-02-16T16:31:25.212Z","level":"INFO","msg":"Removing tokens
[-1183123795032699037, -1237172198872744999, -1270149265263593564,
-1389400073696907418, -1402231454860468978, -1420312602012480180,
-1627335600868165275, -1713896876933182948, -1714549883939918780,
-1741474774932965377, -1792210053935902256, -1807601543024917385,
-1854566658540932161, -1912378675308644559, -1963516450452847956,
-2081077201955154820, -2099259848054237265, -2129680696766320789,
-2221883494221453885, -2310616694711668360, -2331548641412448139,
-2334622440252028474, -2411111763982847025, -245865888987490824,
-246385138307167507, -2473543956019538623, -2490169864633659751,
-2532042510174406079, -2540038371895843059, -2577021784711590342,
-2593326939593610701, -2705600331335690795, -2770464252902532656,
-2839608713233822191, -2912475620975565625, -2943366603912110567,
-2949295810903492735, -2954629576212069889, -3095752208331422097,
-3121066151673234082, -3185818042633507184, -3219579857569752331,
-3286343234413829618, -3289689335761352230, -3361565551667832402,
-3383031027800371008, -3412816001810159961, -3457683958466098497,
-3508695836421455440, -3532532777067013075, -3543699029518234044,
-3580715661159129955, -3599252547477456660, -3661459590777641370,
-3667538041993955650, -3690186824731493069, -3814115795769795724,
-409080988277749690, -4156007243077529438, -4168774893038102265,
-4213854018229603756, -4249034660629583574, -4281296366845033280,
-4320926472067799077, -4545355489460541553, -4630628418422675470,
-4654983718222492637, -4768547837932983920, -5159507483120816494,
-51965338274886423, -5199723923113410828, -5223817829501441469,
-5267064983314818217, -5348036139422617554, -5364643938379720849,
-5495473223858010026, -5538859875930354329, -554259440667407064,
-5547533646691145209, -5599903631185515707, -5779719603131964183,
-5932457424237549800, -6205324647087801173, -6229187725024886147,
-6231270541967013773, -6235596621005793282, -6243196665962158041,
-6264073286569207959, -6427783773364474000, -6501124205004483449,
-6515615874324067226, -6573166604041811188, -6637220677879215528,
-6651782973192407164, -6660806062319128258, -6745026564913028874,
-6812331251174808386, -7084494716757599690, -7149011211591734620,
-7274798955118474541, -731652465373161574, -7403663263352230504,
-7447942789148217722, -7449824614964680985, -7587057746419387993,
-7641766861615411515, -7685202236673643074, -7707429459189222115,
-7742454891260223055, -7833962528790251697, -7931725553821464164,
-7939773289897241357, -8098012638292660116, -8219265643204532541,
-8235265212011847570, -8252620209307958750, -8332424581430025253,
-8349765834148048802, -836487426771203231, -8505914076204635715,
-851663313261657661, -8519826107865925909, -8695372492829159498,
-8942575831111517062, -8976797677584625277, -9051099539809741903,
-963992846195142625, 1003830146651642498, 1178027849963644975,
1224976256801927215, 1244352078631038965, 1417164721674217220,
1451341591539912399, 1491082438042287951, 1514848536642096395,
1779901662209221643, 1854090933062166699, 1943140502461231414,
1994380979939543148, 2007534562233247106, 217709005954998613,
2235989942794609624, 2241300505721899121, 2246701380267263353,
231183097291070161, 2426913079401922388, 2451699806267654588,
2506037868809622944, 2539177368085335242, 2585691867824138041,
2615916489656464006, 2629622650576085045, 2657709892891143999,
269267015073834956, 2978608649954009655, 2994749269564046800,
3002378570488766614, 3121394566209939036, 3141180417712080576,
3154166401188000671, 3295643071378191280, 3367253256599281972,
3373165743103532333, 3410319489097671220, 341173714064808701,
35319237546365581, 358168701228763255, 3695798865479120909,
3784007090181772945, 3859215587032554422, 388414320421460408,
3958884439975359895, 4116791292047135794, 4157923384507172767,
4177479403175856644, 4192843956567907187, 4237835213950914050,
4283455088657318777, 4437755738583404917, 4464954521637384793,
4479350931740982844, 448168435071015674, 4483893029099418629,
4495279466361191724, 4626894508809678069, 4913067912592229158,
5055224510936602124, 5146899889661568824, 5333227006937586923,
5344713819757688419, 5354664536118782383, 5407856766112305920,
5524147618224097934, 5530799914723849578, 5594217902784391863,
5740424123774088150, 5773813700926694122, 5797583289244413653,
596003237801815323, 6051342740892152639, 6053457919557951562,
6063733117952809262, 6188792405330710479, 6215325692940778409,
6255014464234787541, 6307265543356112469, 6331411861541807584,
6428394292228979654, 6440759262443849168, 6519598944774579329,
6559591672744555181, 6637797466881599715, 6662404025200321225,
6668777379548711499, 6828109123001551836, 6861072227483193626,
6876892389046410599, 6982428538201939129, 7004630263067699041,
703529510861716420, 7035478043169974728, 713068522342900631,
7143695289303246630, 7149047549904215441, 7302396691607427712,
7315761441890909840, 7410347447605570455, 754235745869667971,
7544573809574793144, 7586097788690287546, 7597038064500920869,
7614889355576709597, 7624148922826428440, 7627724472890172915,
7896536432019982357, 8070449930340402428, 8100741102346464407,
8206476157175887120, 8277001814877694995, 8323125797400536081,
8331290716255307824, 8338939509697372178, 834182466138639995,
8388253326997854443, 8414277759665359090, 8670923111306643765,
8769994749009616915, 8770706585661243637, 8812518846508548955,
8847966629758817746, 8906298870427014138, 8987438727642595319,
9050092026849126476, 933979495781858352, 962613924858074837,
985754492835172659] for /10.0.1.32"}
{"time":"2016-02-16T16:31:25.218Z","level":"DEBUG","msg":"removing endpoint
/10.0.1.32"}
{"time":"2016-02-16T16:31:25.218Z","level":"DEBUG","msg":"Ignoring state change
for dead or unknown endpoint: /10.0.1.32"}
{"time":"2016-02-16T16:31:25.221Z","level":"TRACE","msg":"/10.0.2.168local
generation 1455284954, remote generation 1455284954"}
{"time":"2016-02-16T16:31:25.221Z","level":"TRACE","msg":"Updating heartbeat
state version to 1077795 from 1077795 for /10.0.2.168 ..."}
{"time":"2016-02-16T16:31:25.221Z","level":"TRACE","msg":"/10.0.2.117local
generation 1454676714, remote generation 1454676714"}
{"time":"2016-02-16T16:31:25.221Z","level":"TRACE","msg":"Updating heartbeat
state version to 2922701 from 2922695 for /10.0.2.117 ..."}
{"time":"2016-02-16T16:31:25.352Z","level":"TRACE","msg":"My heartbeat is now
2947089"}
{"time":"2016-02-16T16:31:25.352Z","level":"TRACE","msg":"Gossip Digests are :
/10.0.2.117:1454676714:2922701 /10.0.2.168:1455284954:1077797
/10.0.1.129:1454674252:2930162 /10.0.2.128:1454668510:2947569
/10.0.1.32:1447859696:20667192 /10.0.1.12:1454675536:2926261
/10.0.1.88:1454668666:2947089 "}
{"time":"2016-02-16T16:31:25.354Z","level":"TRACE","msg":"Sending a
GossipDigestSyn to /10.0.2.128 ..."}
{"time":"2016-02-16T16:31:25.354Z","level":"TRACE","msg":"Performing status
check ..."}
{"time":"2016-02-16T16:31:25.354Z","level":"TRACE","msg":"PHI for /10.0.2.128 :
0.14132872197630086"}
{"time":"2016-02-16T16:31:25.354Z","level":"DEBUG","msg":"time is expiring for
endpoint : /10.0.1.32 (1454932471326)"}
{"time":"2016-02-16T16:31:25.354Z","level":"DEBUG","msg":"evicting /10.0.1.32
from gossip"}
{"time":"2016-02-16T16:31:25.354Z","level":"TRACE","msg":"PHI for /10.0.1.129 :
0.7902186262859834"}
{"time":"2016-02-16T16:31:25.354Z","level":"TRACE","msg":"PHI for /10.0.2.168 :
1.2129498281352311"}
{"time":"2016-02-16T16:31:25.354Z","level":"TRACE","msg":"PHI for /10.0.1.12 :
1.1816270155595487"}
{"time":"2016-02-16T16:31:25.354Z","level":"TRACE","msg":"PHI for /10.0.2.117 :
0.13743310334362183"}
{"time":"2016-02-16T16:31:25.355Z","level":"TRACE","msg":"Received a
GossipDigestAckMessage from /10.0.2.128"}
{"time":"2016-02-16T16:31:25.355Z","level":"TRACE","msg":"Received ack with 1
digests and 1 states"}
{"time":"2016-02-16T16:31:25.355Z","level":"TRACE","msg":"/10.0.2.128local
generation 1454668510, remote generation 1454668510"}
{"time":"2016-02-16T16:31:25.355Z","level":"TRACE","msg":"Updating heartbeat
state version to 2947569 from 2947569 for /10.0.2.128 ..."}
{"time":"2016-02-16T16:31:25.355Z","level":"TRACE","msg":"local heartbeat
version 2947089 greater than 2947088 for /10.0.1.88"}
{"time":"2016-02-16T16:31:25.355Z","level":"TRACE","msg":"Sending a
GossipDigestAck2Message to /10.0.2.128"}
{"time":"2016-02-16T16:31:25.542Z","level":"TRACE","msg":"local heartbeat
version 2922701 greater than 2922698 for /10.0.2.117"}
{"time":"2016-02-16T16:31:25.542Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:25.542Z","level":"TRACE","msg":"local heartbeat
version 2947089 greater than 2947086 for /10.0.1.88"}
{"time":"2016-02-16T16:31:25.542Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:25.542Z","level":"TRACE","msg":"local heartbeat
version 2947569 greater than 2947566 for /10.0.2.128"}
{"time":"2016-02-16T16:31:25.542Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:25.543Z","level":"TRACE","msg":"Received a
GossipDigestAck2Message from /10.0.1.129"}
{"time":"2016-02-16T16:31:25.543Z","level":"TRACE","msg":"reporting
/10.0.2.168"}
{"time":"2016-02-16T16:31:25.543Z","level":"TRACE","msg":"reporting
/10.0.1.129"}
{"time":"2016-02-16T16:31:25.543Z","level":"TRACE","msg":"/10.0.2.168local
generation 1455284954, remote generation 1455284954"}
{"time":"2016-02-16T16:31:25.543Z","level":"TRACE","msg":"Updating heartbeat
state version to 1077801 from 1077795 for /10.0.2.168 ..."}
{"time":"2016-02-16T16:31:25.543Z","level":"TRACE","msg":"/10.0.1.129local
generation 1454674252, remote generation 1454674252"}
{"time":"2016-02-16T16:31:25.543Z","level":"TRACE","msg":"Updating heartbeat
state version to 2930165 from 2930162 for /10.0.1.129 ..."}
{"time":"2016-02-16T16:31:26.128Z","level":"TRACE","msg":"local heartbeat
version 1077801 greater than 1077798 for /10.0.2.168"}
{"time":"2016-02-16T16:31:26.128Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.128Z","level":"TRACE","msg":"local heartbeat
version 2922701 greater than 2922700 for /10.0.2.117"}
{"time":"2016-02-16T16:31:26.128Z","level":"TRACE","msg":"local heartbeat
version 2930165 greater than 2930159 for /10.0.1.129"}
{"time":"2016-02-16T16:31:26.128Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.129Z","level":"TRACE","msg":"local heartbeat
version 2947089 greater than 2947086 for /10.0.1.88"}
{"time":"2016-02-16T16:31:26.129Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.129Z","level":"TRACE","msg":"local heartbeat
version 2947569 greater than 2947566 for /10.0.2.128"}
{"time":"2016-02-16T16:31:26.129Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.129Z","level":"TRACE","msg":"Received a
GossipDigestAck2Message from /10.0.1.12"}
{"time":"2016-02-16T16:31:26.129Z","level":"TRACE","msg":"reporting /10.0.1.12"}
{"time":"2016-02-16T16:31:26.129Z","level":"TRACE","msg":"/10.0.1.12local
generation 1454675536, remote generation 1454675536"}
{"time":"2016-02-16T16:31:26.129Z","level":"TRACE","msg":"Updating heartbeat
state version to 2926267 from 2926261 for /10.0.1.12 ..."}
{"time":"2016-02-16T16:31:26.130Z","level":"TRACE","msg":"local heartbeat
version 1077801 greater than 1077798 for /10.0.2.168"}
{"time":"2016-02-16T16:31:26.130Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.130Z","level":"TRACE","msg":"local heartbeat
version 2926267 greater than 2926264 for /10.0.1.12"}
{"time":"2016-02-16T16:31:26.130Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.130Z","level":"TRACE","msg":"local heartbeat
version 2930165 greater than 2930159 for /10.0.1.129"}
{"time":"2016-02-16T16:31:26.130Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.130Z","level":"TRACE","msg":"local heartbeat
version 2947089 greater than 2947086 for /10.0.1.88"}
{"time":"2016-02-16T16:31:26.130Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.130Z","level":"TRACE","msg":"local heartbeat
version 2947569 greater than 2947568 for /10.0.2.128"}
{"time":"2016-02-16T16:31:26.130Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.131Z","level":"TRACE","msg":"Received a
GossipDigestAck2Message from /10.0.2.117"}
{"time":"2016-02-16T16:31:26.131Z","level":"TRACE","msg":"reporting
/10.0.2.117"}
{"time":"2016-02-16T16:31:26.131Z","level":"TRACE","msg":"/10.0.2.117local
generation 1454676714, remote generation 1454676714"}
{"time":"2016-02-16T16:31:26.131Z","level":"TRACE","msg":"Updating heartbeat
state version to 2922704 from 2922701 for /10.0.2.117 ..."}
{"time":"2016-02-16T16:31:26.354Z","level":"TRACE","msg":"My heartbeat is now
2947092"}
{"time":"2016-02-16T16:31:26.354Z","level":"TRACE","msg":"Gossip Digests are :
/10.0.2.168:1455284954:1077801 /10.0.1.88:1454668666:2947092
/10.0.2.128:1454668510:2947571 /10.0.1.129:1454674252:2930165
/10.0.1.12:1454675536:2926267 /10.0.2.117:1454676714:2922704 "}
{"time":"2016-02-16T16:31:26.354Z","level":"TRACE","msg":"Sending a
GossipDigestSyn to /10.0.1.129 ..."}
{"time":"2016-02-16T16:31:26.354Z","level":"TRACE","msg":"Performing status
check ..."}
{"time":"2016-02-16T16:31:26.354Z","level":"TRACE","msg":"PHI for /10.0.2.128 :
1.1254232596265088"}
{"time":"2016-02-16T16:31:26.354Z","level":"TRACE","msg":"PHI for /10.0.1.129 :
0.7904647939808764"}
{"time":"2016-02-16T16:31:26.355Z","level":"TRACE","msg":"PHI for /10.0.2.168 :
0.803410322347404"}
{"time":"2016-02-16T16:31:26.355Z","level":"TRACE","msg":"PHI for /10.0.1.12 :
0.21743102018805321"}
{"time":"2016-02-16T16:31:26.355Z","level":"TRACE","msg":"PHI for /10.0.2.117 :
0.21360712465395978"}
{"time":"2016-02-16T16:31:26.355Z","level":"TRACE","msg":"Received a
GossipDigestAckMessage from /10.0.1.129"}
{"time":"2016-02-16T16:31:26.355Z","level":"TRACE","msg":"Received ack with 2
digests and 2 states"}
{"time":"2016-02-16T16:31:26.355Z","level":"TRACE","msg":"reporting
/10.0.2.128"}
{"time":"2016-02-16T16:31:26.355Z","level":"TRACE","msg":"/10.0.2.128local
generation 1454668510, remote generation 1454668510"}
{"time":"2016-02-16T16:31:26.355Z","level":"TRACE","msg":"Updating heartbeat
state version to 2947572 from 2947569 for /10.0.2.128 ..."}
{"time":"2016-02-16T16:31:26.355Z","level":"TRACE","msg":"/10.0.1.129local
generation 1454674252, remote generation 1454674252"}
{"time":"2016-02-16T16:31:26.356Z","level":"TRACE","msg":"Updating heartbeat
state version to 2930165 from 2930165 for /10.0.1.129 ..."}
{"time":"2016-02-16T16:31:26.356Z","level":"TRACE","msg":"local heartbeat
version 2926267 greater than 2926264 for /10.0.1.12"}
{"time":"2016-02-16T16:31:26.356Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.356Z","level":"TRACE","msg":"local heartbeat
version 2947092 greater than 2947089 for /10.0.1.88"}
{"time":"2016-02-16T16:31:26.356Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.356Z","level":"TRACE","msg":"Sending a
GossipDigestAck2Message to /10.0.1.129"}
{"time":"2016-02-16T16:31:26.501Z","level":"TRACE","msg":"local heartbeat
version 2922704 greater than 2922701 for /10.0.2.117"}
{"time":"2016-02-16T16:31:26.501Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.501Z","level":"TRACE","msg":"local heartbeat
version 2930165 greater than 2930164 for /10.0.1.129"}
{"time":"2016-02-16T16:31:26.501Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.251889169216156"}
{"time":"2016-02-16T16:31:26.501Z","level":"TRACE","msg":"local heartbeat
version 2947092 greater than 2947086 for /10.0.1.88"}
{"time":"2016-02-16T16:31:26.501Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.501Z","level":"TRACE","msg":"local heartbeat
version 2947572 greater than 2947566 for /10.0.2.128"}
{"time":"2016-02-16T16:31:26.501Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:26.502Z","level":"TRACE","msg":"Received a
GossipDigestAck2Message from /10.0.2.168"}
{"time":"2016-02-16T16:31:26.502Z","level":"TRACE","msg":"reporting
/10.0.2.168"}
{"time":"2016-02-16T16:31:26.502Z","level":"TRACE","msg":"/10.0.2.168local
generation 1455284954, remote generation 1455284954"}
{"time":"2016-02-16T16:31:26.502Z","level":"TRACE","msg":"Updating heartbeat
state version to 1077804 from 1077801 for /10.0.2.168 ..."}
{"time":"2016-02-16T16:31:27.130Z","level":"TRACE","msg":"local heartbeat
version 1077804 greater than 1077801 for /10.0.2.168"}
{"time":"2016-02-16T16:31:27.130Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:27.130Z","level":"TRACE","msg":"local heartbeat
version 2947092 greater than 2947091 for /10.0.1.88"}
{"time":"2016-02-16T16:31:27.130Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:27.130Z","level":"TRACE","msg":"local heartbeat
version 2947572 greater than 2947571 for /10.0.2.128"}
{"time":"2016-02-16T16:31:27.131Z","level":"TRACE","msg":"Received a
GossipDigestAck2Message from /10.0.2.117"}
{"time":"2016-02-16T16:31:27.131Z","level":"TRACE","msg":"reporting
/10.0.2.117"}
{"time":"2016-02-16T16:31:27.131Z","level":"TRACE","msg":"/10.0.2.117local
generation 1454676714, remote generation 1454676714"}
{"time":"2016-02-16T16:31:27.131Z","level":"TRACE","msg":"Updating heartbeat
state version to 2922707 from 2922704 for /10.0.2.117 ..."}
{"time":"2016-02-16T16:31:27.209Z","level":"TRACE","msg":"local heartbeat
version 1077804 greater than 1077801 for /10.0.2.168"}
{"time":"2016-02-16T16:31:27.209Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:27.209Z","level":"TRACE","msg":"local heartbeat
version 2922707 greater than 2922704 for /10.0.2.117"}
{"time":"2016-02-16T16:31:27.209Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:27.209Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-16T16:31:27.209Z","level":"TRACE","msg":"requestAll for
/10.0.1.32"}
{"time":"2016-02-16T16:31:27.211Z","level":"TRACE","msg":"Received a
GossipDigestAck2Message from /10.0.2.128"}
{"time":"2016-02-16T16:31:27.211Z","level":"TRACE","msg":"reporting
/10.0.2.128"}
{"time":"2016-02-16T16:31:27.211Z","level":"TRACE","msg":"reporting
/10.0.1.129"}
{"time":"2016-02-16T16:31:27.211Z","level":"TRACE","msg":"/10.0.2.128local
generation 1454668510, remote generation 1454668510"}
{"time":"2016-02-16T16:31:27.211Z","level":"TRACE","msg":"Updating heartbeat
state version to 2947575 from 2947572 for /10.0.2.128 ..."}
{"time":"2016-02-16T16:31:27.211Z","level":"TRACE","msg":"Ignoring gossip for
/10.0.1.32 because it is quarantined"}
{noformat}
> Decommissioned nodes can remain in gossip
> -----------------------------------------
>
> Key: CASSANDRA-10371
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10371
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata
> Reporter: Brandon Williams
> Assignee: Stefania
> Priority: Minor
>
> This may apply to other dead states as well. Dead states should be expired
> after 3 days. In the case of decom we attach a timestamp to let the other
> nodes know when it should be expired. It has been observed that sometimes a
> subset of nodes in the cluster never expire the state, and through heap
> analysis of these nodes it is revealed that the epstate.isAlive check returns
> true when it should return false, which would allow the state to be evicted.
> This may have been affected by CASSANDRA-8336.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
