[
https://issues.apache.org/jira/browse/CASSANDRA-10371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15152664#comment-15152664
]
pavel commented on CASSANDRA-10371:
-----------------------------------
Latest logs from 10.0.2.128 are:
{noformat}
{"time":"2016-02-18T17:23:24.532Z","level":"TRACE","msg":"Adding state LOAD:
1.417907115E9"}
{"time":"2016-02-18T17:23:24.532Z","level":"TRACE","msg":"Sending a
GossipDigestAck2Message to /10.0.1.12"}
{"time":"2016-02-18T17:23:25.530Z","level":"TRACE","msg":"My heartbeat is now
3481162"}
{"time":"2016-02-18T17:23:25.530Z","level":"TRACE","msg":"Gossip Digests are :
/10.0.2.168:1455284954:1611388 /10.0.2.128:1454668510:3481162
/10.0.1.129:1454674252:3463753 /10.0.1.88:1454668666:3480679
/10.0.1.12:1454675536:3459852 /10.0.2.117:1454676714:3456286
/10.0.1.32:1447859696:20667192 "}
{"time":"2016-02-18T17:23:25.530Z","level":"TRACE","msg":"Sending a
GossipDigestSyn to /10.0.1.12 ..."}
{"time":"2016-02-18T17:23:25.530Z","level":"TRACE","msg":"Performing status
check ..."}
{"time":"2016-02-18T17:23:25.531Z","level":"TRACE","msg":"PHI for /10.0.1.129 :
0.9763986442368152"}
{"time":"2016-02-18T17:23:25.531Z","level":"TRACE","msg":"PHI for /10.0.2.168 :
1.0035592404405616"}
{"time":"2016-02-18T17:23:25.531Z","level":"TRACE","msg":"PHI for /10.0.1.12 :
0.9730840000367144"}
{"time":"2016-02-18T17:23:25.531Z","level":"TRACE","msg":"PHI for /10.0.2.117 :
0.9806746365769617"}
{"time":"2016-02-18T17:23:25.531Z","level":"TRACE","msg":"PHI for /10.0.1.88 :
0.9813799722457983"}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"Received a
GossipDigestAckMessage from /10.0.1.12"}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"Received ack with 2
digests and 4 states"}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"reporting /10.0.1.88"}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"reporting
/10.0.1.129"}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"reporting /10.0.1.12"}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"reporting
/10.0.2.117"}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"/10.0.1.88local
generation 1454668666, remote generation 1454668666"}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"Updating heartbeat
state version to 3480682 from 3480679 for /10.0.1.88 ..."}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"/10.0.1.129local
generation 1454674252, remote generation 1454674252"}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"Updating heartbeat
state version to 3463756 from 3463753 for /10.0.1.129 ..."}
{"time":"2016-02-18T17:23:25.532Z","level":"TRACE","msg":"/10.0.1.12local
generation 1454675536, remote generation 1454675536"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Updating heartbeat
state version to 3459855 from 3459852 for /10.0.1.12 ..."}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"/10.0.2.117local
generation 1454676714, remote generation 1454676714"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Updating heartbeat
state version to 3456289 from 3456286 for /10.0.2.117 ..."}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"local heartbeat
version 3481162 greater than 3481159 for /10.0.2.128"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"local heartbeat
version 20667192 greater than 0 for /10.0.1.32"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state
RELEASE_VERSION: 2.1.11"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state HOST_ID:
88290568-e6ca-4674-a597-bad21e8a794c"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state STATUS:
LEFT,-1183123795032699037,1454932471326"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.5167958736419678"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state SCHEMA:
612b0652-7783-384f-b412-a5288429ce5a"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state
NET_VERSION: 8"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state DC:
cassandra"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state
INTERNAL_IP: 10.0.1.32"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state RACK:
RAC1"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state
RPC_ADDRESS: 10.10.65.1"}
?j??????7???6-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state
TOKENS:???-G?????p?3{?_?=w?????d???E??)?J 6?
?4?????? ?>?7? ?s?DQ??D?D?CAs1bs?u??27??01B??O?L????Ja|??q???<??*H???e??
[??????????B?I??? s???F???????D
?<?W'?????sj?s??b????Up??????????????$?'??????
4K????=??V&???f??N??5???'???@? A?
????F????}???^????Q?-w?Z?d?=v??X??????YQ????
?N?N?????
}???L?[?R?1V??R?n?i??%?Rg???b?re}???????DS?'???C??'?????g?f,??
?????)?e?G??.?Ga??~
????(?~??E\m6????M????w_?p?????l???(8
??"I????N??????3?Yr?I%Q&?k??O???$???gw1??BVZ???z?????t??H?v??*R??[???l(????W??????/??]???????.???k????
????!?????????-A?1?d?m? ?u??n?????"????.?*?
?R3)??Y2?????jT?pD??fRo???O?$3|?|??c??U??????X?H???
?????FL4???H,??w?????I??BBr?T??i0??Mq?8???u????x????B?]G??!c??6y???d38????6????.H??????K|?v?S???7????Q???l?Q??^?S??5?$?P
?{!?
?Vqh?k???1v?@?2 ?
H;eu/?Y????c??^-?w5?5S?X!? ?^??"/??zA"?<*#<?\?p
4?~fe?5??!
??c??:eY6??lO?u9!??P?9??I??9?cbJ?:/?]%??:????/;q??Z?C=?T?7=??{(xr>)????~86?3?Z>9????}z???R?x?>gH3J|?,?
R>bp????@6
??lD.?<gnF'???9V
Gmv.!?gJm??J,<???6JO?~??K
??L????JL?Y8?5M???
?*O?S??3P
??S?Pu'H?'Em?z(S??)U?T3???T&?$?YYU??,_OVAE???V?F???W???'?^W??????Y6>??t?Yb,l???ZzD??L?Y?E??\1??w(\u?)x2?\?A??)J^?Q
?_7l??_o?1G??`?????a5r??a ?p\???a?
44?? ?T
??c#?F?'c6???j?eWS???e??EUh>f???????
i???,%m?"?-k?iG;??B?in?k??i???
{o??s??pk???lq?B??.r??L
s?????s?????Hs??Mn?`
?ti?A?t?? ????
?R?xUN?(Iy?G?@?y??+???zL[3>??z?J?oH?{????C?|??????}?b????L
/?3"}? ?)g)?|0
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Adding state LOAD:
1.417907115E9"}
{"time":"2016-02-18T17:23:25.533Z","level":"TRACE","msg":"Sending a
GossipDigestAck2Message to /10.0.1.12"}
{"time":"2016-02-18T17:23:26.531Z","level":"TRACE","msg":"My heartbeat is now
3481165"}
{"time":"2016-02-18T17:23:26.531Z","level":"TRACE","msg":"Gossip Digests are :
/10.0.1.129:1454674252:3463756 /10.0.1.32:1447859696:20667192
/10.0.2.168:1455284954:1611388 /10.0.1.12:1454675536:3459855
/10.0.1.88:1454668666:3480682 /10.0.2.128:1454668510:3481165
/10.0.2.117:1454676714:3456289 "}
{"time":"2016-02-18T17:23:26.531Z","level":"TRACE","msg":"Sending a
GossipDigestSyn to /10.0.1.88 ..."}
{"time":"2016-02-18T17:23:26.531Z","level":"TRACE","msg":"Performing status
check ..."}
{"time":"2016-02-18T17:23:26.531Z","level":"TRACE","msg":"PHI for /10.0.1.129 :
0.9761823849067737"}
{"time":"2016-02-18T17:23:26.531Z","level":"TRACE","msg":"PHI for /10.0.2.168 :
2.0081343786762047"}
{"time":"2016-02-18T17:23:26.531Z","level":"TRACE","msg":"PHI for /10.0.1.12 :
0.9725689805550068"}
{"time":"2016-02-18T17:23:26.531Z","level":"TRACE","msg":"PHI for /10.0.2.117 :
0.9793344483809482"}
{"time":"2016-02-18T17:23:26.531Z","level":"TRACE","msg":"PHI for /10.0.1.88 :
0.9800589923703654"}
{"time":"2016-02-18T17:23:26.532Z","level":"TRACE","msg":"Received a
GossipDigestAckMessage from /10.0.1.88"}
{"time":"2016-02-18T17:23:26.532Z","level":"TRACE","msg":"Received ack with 2
digests and 3 states"}
{"time":"2016-02-18T17:23:26.532Z","level":"TRACE","msg":"reporting
/10.0.2.168"}
{"time":"2016-02-18T17:23:26.532Z","level":"TRACE","msg":"reporting /10.0.1.88"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"reporting
/10.0.2.117"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"/10.0.2.168local
generation 1455284954, remote generation 1455284954"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Updating heartbeat
state version to 1611392 from 1611386 for /10.0.2.168 ..."}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"/10.0.1.88local
generation 1454668666, remote generation 1454668666"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Updating heartbeat
state version to 3480685 from 3480682 for /10.0.1.88 ..."}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"/10.0.2.117local
generation 1454676714, remote generation 1454676714"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Updating heartbeat
state version to 3456292 from 3456289 for /10.0.2.117 ..."}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"local heartbeat
version 3481165 greater than 3481162 for /10.0.2.128"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"local heartbeat
version 20667192 greater than 0 for /10.0.1.32"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state
RELEASE_VERSION: 2.1.11"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state HOST_ID:
88290568-e6ca-4674-a597-bad21e8a794c"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state STATUS:
LEFT,-1183123795032699037,1454932471326"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.5167958736419678"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state SCHEMA:
612b0652-7783-384f-b412-a5288429ce5a"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state
NET_VERSION: 8"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state DC:
cassandra"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state
INTERNAL_IP: 10.0.1.32"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state RACK:
RAC1"}
{"time":"2016-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state
RPC_ADDRESS: 10.10.65.1"}
?j??????7???6-02-18T17:23:26.533Z","level":"TRACE","msg":"Adding state
TOKENS:???-G?????p?3{?_?=w?????d???E??)?J 6?
?4?????? ?>?7? ?s?DQ??D?D?CAs1bs?u??27??01B??O?L????Ja|??q???<??*H???e??
[??????????B?I??? s???F???????D
?<?W'?????sj?s??b????Up??????????????$?'??????
4K????=??V&???f??N??5???'???@? A?
????F????}???^????Q?-w?Z?d?=v??X??????YQ????
?N?N?????
}???L?[?R?1V??R?n?i??%?Rg???b?re}???????DS?'???C??'?????g?f,??
?????)?e?G??.?Ga??~
????(?~??E\m6????M????w_?p?????l???(8
??"I????N??????3?Yr?I%Q&?k??O???$???gw1??BVZ???z?????t??H?v??*R??[???l(????W??????/??]???????.???k????
????!?????????-A?1?d?m? ?u??n?????"????.?*?
?R3)??Y2?????jT?pD??fRo???O?$3|?|??c??U??????X?H???
?????FL4???H,??w?????I??BBr?T??i0??Mq?8???u????x????B?]G??!c??6y???d38????6????.H??????K|?v?S???7????Q???l?Q??^?S??5?$?P
?{!?
?Vqh?k???1v?@?2 ?
H;eu/?Y????c??^-?w5?5S?X!? ?^??"/??zA"?<*#<?\?p
4?~fe?5??!
??c??:eY6??lO?u9!??P?9??I??9?cbJ?:/?]%??:????/;q??Z?C=?T?7=??{(xr>)????~86?3?Z>9????}z???R?x?>gH3J|?,?
R>bp????@6
??lD.?<gnF'???9V
Gmv.!?gJm??J,<???6JO?~??K
??L????JL?Y8?5M???
?*O?S??3P
??S?Pu'H?'Em?z(S??)U?T3???T&?$?YYU??,_OVAE???V?F???W???'?^W??????Y6>??t?Yb,l???ZzD??L?Y?E??\1??w(\u?)x2?\?A??)J^?Q
?_7l??_o?1G??`?????a5r??a ?p\???a?
44?? ?T
??c#?F?'c6???j?eWS???e??EUh>f???????
i???,%m?"?-k?iG;??B?in?k??i???
{o??s??pk???lq?B??.r??L
s?????s?????Hs??Mn?`
?ti?A?t?? ????
?R?xUN?(Iy?G?@?y??+???zL[3>??z?J?oH?{????C?|??????}?b????L
/?3"}? ?)g)?|0
{"time":"2016-02-18T17:23:26.534Z","level":"TRACE","msg":"Adding state LOAD:
1.417907115E9"}
{"time":"2016-02-18T17:23:26.534Z","level":"TRACE","msg":"Sending a
GossipDigestAck2Message to /10.0.1.88"}
{"time":"2016-02-18T17:23:27.441Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.25641027092933655"}
{"time":"2016-02-18T17:23:27.442Z","level":"TRACE","msg":"Received a
GossipDigestAck2Message from /10.0.1.88"}
{"time":"2016-02-18T17:23:27.442Z","level":"TRACE","msg":"reporting
/10.0.2.168"}
{"time":"2016-02-18T17:23:27.442Z","level":"TRACE","msg":"reporting /10.0.1.88"}
{"time":"2016-02-18T17:23:27.442Z","level":"TRACE","msg":"reporting
/10.0.1.129"}
{"time":"2016-02-18T17:23:27.443Z","level":"TRACE","msg":"reporting /10.0.1.12"}
{"time":"2016-02-18T17:23:27.443Z","level":"TRACE","msg":"/10.0.2.168local
generation 1455284954, remote generation 1455284954"}
{"time":"2016-02-18T17:23:27.443Z","level":"TRACE","msg":"Updating heartbeat
state version to 1611395 from 1611392 for /10.0.2.168 ..."}
{"time":"2016-02-18T17:23:27.443Z","level":"TRACE","msg":"/10.0.1.88local
generation 1454668666, remote generation 1454668666"}
{"time":"2016-02-18T17:23:27.443Z","level":"TRACE","msg":"Updating heartbeat
state version to 3480688 from 3480685 for /10.0.1.88 ..."}
{"time":"2016-02-18T17:23:27.443Z","level":"TRACE","msg":"/10.0.1.129local
generation 1454674252, remote generation 1454674252"}
{"time":"2016-02-18T17:23:27.443Z","level":"TRACE","msg":"Updating heartbeat
state version to 3463762 from 3463756 for /10.0.1.129 ..."}
{"time":"2016-02-18T17:23:27.443Z","level":"TRACE","msg":"/10.0.1.12local
generation 1454675536, remote generation 1454675536"}
{"time":"2016-02-18T17:23:27.443Z","level":"TRACE","msg":"Updating heartbeat
state version to 3459858 from 3459855 for /10.0.1.12 ..."}
{"time":"2016-02-18T17:23:27.531Z","level":"TRACE","msg":"My heartbeat is now
3481168"}
{"time":"2016-02-18T17:23:27.531Z","level":"TRACE","msg":"Gossip Digests are :
/10.0.2.128:1454668510:3481168 /10.0.1.32:1447859696:20667192
/10.0.1.88:1454668666:3480688 /10.0.2.168:1455284954:1611395
/10.0.1.129:1454674252:3463762 /10.0.2.117:1454676714:3456294
/10.0.1.12:1454675536:3459858 "}
{"time":"2016-02-18T17:23:27.531Z","level":"TRACE","msg":"Sending a
GossipDigestSyn to /10.0.1.129 ..."}
{"time":"2016-02-18T17:23:27.531Z","level":"TRACE","msg":"Performing status
check ..."}
{"time":"2016-02-18T17:23:27.532Z","level":"TRACE","msg":"PHI for /10.0.1.129 :
0.08694391920660251"}
{"time":"2016-02-18T17:23:27.532Z","level":"TRACE","msg":"PHI for /10.0.2.168 :
0.08943066263688543"}
{"time":"2016-02-18T17:23:27.532Z","level":"TRACE","msg":"PHI for /10.0.1.12 :
0.08651621637584873"}
{"time":"2016-02-18T17:23:27.532Z","level":"TRACE","msg":"PHI for /10.0.2.117 :
0.9803039634425629"}
{"time":"2016-02-18T17:23:27.532Z","level":"TRACE","msg":"PHI for /10.0.1.88 :
0.08748333029792915"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Received a
GossipDigestAckMessage from /10.0.1.129"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Received ack with 3
digests and 0 states"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"local heartbeat
version 3480688 greater than 3480685 for /10.0.1.88"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"local heartbeat
version 3481168 greater than 3481165 for /10.0.2.128"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.25641027092933655"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"local heartbeat
version 20667192 greater than 0 for /10.0.1.32"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state
RELEASE_VERSION: 2.1.11"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state HOST_ID:
88290568-e6ca-4674-a597-bad21e8a794c"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state STATUS:
LEFT,-1183123795032699037,1454932471326"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.5167958736419678"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state SCHEMA:
612b0652-7783-384f-b412-a5288429ce5a"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state
NET_VERSION: 8"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state DC:
cassandra"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state
INTERNAL_IP: 10.0.1.32"}
{"time":"2016-02-18T17:23:27.533Z","level":"TRACE","msg":"Adding state RACK:
RAC1"}
{"time":"2016-02-18T17:23:27.534Z","level":"TRACE","msg":"Adding state
RPC_ADDRESS: 10.10.65.1"}
?j??????7???6-02-18T17:23:27.534Z","level":"TRACE","msg":"Adding state
TOKENS:???-G?????p?3{?_?=w?????d???E??)?J 6?
?4?????? ?>?7? ?s?DQ??D?D?CAs1bs?u??27??01B??O?L????Ja|??q???<??*H???e??
[??????????B?I??? s???F???????D
?<?W'?????sj?s??b????Up??????????????$?'??????
4K????=??V&???f??N??5???'???@? A?
????F????}???^????Q?-w?Z?d?=v??X??????YQ????
?N?N?????
}???L?[?R?1V??R?n?i??%?Rg???b?re}???????DS?'???C??'?????g?f,??
?????)?e?G??.?Ga??~
????(?~??E\m6????M????w_?p?????l???(8
??"I????N??????3?Yr?I%Q&?k??O???$???gw1??BVZ???z?????t??H?v??*R??[???l(????W??????/??]???????.???k????
????!?????????-A?1?d?m? ?u??n?????"????.?*?
?R3)??Y2?????jT?pD??fRo???O?$3|?|??c??U??????X?H???
?????FL4???H,??w?????I??BBr?T??i0??Mq?8???u????x????B?]G??!c??6y???d38????6????.H??????K|?v?S???7????Q???l?Q??^?S??5?$?P
?{!?
?Vqh?k???1v?@?2 ?
H;eu/?Y????c??^-?w5?5S?X!? ?^??"/??zA"?<*#<?\?p
4?~fe?5??!
??c??:eY6??lO?u9!??P?9??I??9?cbJ?:/?]%??:????/;q??Z?C=?T?7=??{(xr>)????~86?3?Z>9????}z???R?x?>gH3J|?,?
R>bp????@6
??lD.?<gnF'???9V
Gmv.!?gJm??J,<???6JO?~??K
??L????JL?Y8?5M???
?*O?S??3P
??S?Pu'H?'Em?z(S??)U?T3???T&?$?YYU??,_OVAE???V?F???W???'?^W??????Y6>??t?Yb,l???ZzD??L?Y?E??\1??w(\u?)x2?\?A??)J^?Q
?_7l??_o?1G??`?????a5r??a ?p\???a?
44?? ?T
??c#?F?'c6???j?eWS???e??EUh>f???????
i???,%m?"?-k?iG;??B?in?k??i???
{o??s??pk???lq?B??.r??L
s?????s?????Hs??Mn?`
?ti?A?t?? ????
?R?xUN?(Iy?G?@?y??+???zL[3>??z?J?oH?{????C?|??????}?b????L
/?3"}? ?)g)?|0
{"time":"2016-02-18T17:23:27.534Z","level":"TRACE","msg":"Adding state LOAD:
1.417907115E9"}
{"time":"2016-02-18T17:23:27.534Z","level":"TRACE","msg":"Sending a
GossipDigestAck2Message to /10.0.1.129"}
{"time":"2016-02-18T17:23:28.532Z","level":"TRACE","msg":"My heartbeat is now
3481171"}
{"time":"2016-02-18T17:23:28.532Z","level":"TRACE","msg":"Gossip Digests are :
/10.0.1.88:1454668666:3480688 /10.0.2.117:1454676714:3456294
/10.0.2.168:1455284954:1611395 /10.0.2.128:1454668510:3481171
/10.0.1.129:1454674252:3463762 /10.0.1.12:1454675536:3459858
/10.0.1.32:1447859696:20667192 "}
{"time":"2016-02-18T17:23:28.532Z","level":"TRACE","msg":"Sending a
GossipDigestSyn to /10.0.2.117 ..."}
{"time":"2016-02-18T17:23:28.532Z","level":"TRACE","msg":"Performing status
check ..."}
{"time":"2016-02-18T17:23:28.532Z","level":"TRACE","msg":"PHI for /10.0.1.129 :
1.0647911374060737"}
{"time":"2016-02-18T17:23:28.532Z","level":"TRACE","msg":"PHI for /10.0.2.168 :
1.0934171825795906"}
{"time":"2016-02-18T17:23:28.532Z","level":"TRACE","msg":"PHI for /10.0.1.12 :
1.0591266224847133"}
{"time":"2016-02-18T17:23:28.532Z","level":"TRACE","msg":"PHI for /10.0.2.117 :
1.962043900044831"}
{"time":"2016-02-18T17:23:28.532Z","level":"TRACE","msg":"PHI for /10.0.1.88 :
1.069004979529513"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Received a
GossipDigestAckMessage from /10.0.2.117"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Received ack with 2
digests and 5 states"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"reporting
/10.0.2.168"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"reporting /10.0.1.88"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"reporting
/10.0.1.129"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"reporting /10.0.1.12"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"reporting
/10.0.2.117"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"/10.0.2.168local
generation 1455284954, remote generation 1455284954"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Updating heartbeat
state version to 1611398 from 1611395 for /10.0.2.168 ..."}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"/10.0.1.88local
generation 1454668666, remote generation 1454668666"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Updating heartbeat
state version to 3480691 from 3480688 for /10.0.1.88 ..."}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"/10.0.1.129local
generation 1454674252, remote generation 1454674252"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Updating heartbeat
state version to 3463765 from 3463762 for /10.0.1.129 ..."}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"/10.0.1.12local
generation 1454675536, remote generation 1454675536"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Updating heartbeat
state version to 3459864 from 3459858 for /10.0.1.12 ..."}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"/10.0.2.117local
generation 1454676714, remote generation 1454676714"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Updating heartbeat
state version to 3456298 from 3456292 for /10.0.2.117 ..."}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"local heartbeat
version 3481171 greater than 3481168 for /10.0.2.128"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.0"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"local heartbeat
version 20667192 greater than 0 for /10.0.1.32"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state
RELEASE_VERSION: 2.1.11"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state HOST_ID:
88290568-e6ca-4674-a597-bad21e8a794c"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state STATUS:
LEFT,-1183123795032699037,1454932471326"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state
SEVERITY: 0.5167958736419678"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state SCHEMA:
612b0652-7783-384f-b412-a5288429ce5a"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state
NET_VERSION: 8"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state DC:
cassandra"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state
INTERNAL_IP: 10.0.1.32"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state RACK:
RAC1"}
{"time":"2016-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state
RPC_ADDRESS: 10.10.65.1"}
?j??????7???6-02-18T17:23:28.533Z","level":"TRACE","msg":"Adding state
TOKENS:???-G?????p?3{?_?=w?????d???E??)?J 6?
?4?????? ?>?7? ?s?DQ??D?D?CAs1bs?u??27??01B??O?L????Ja|??q???<??*H???e??
[??????????B?I??? s???F???????D
?<?W'?????sj?s??b????Up??????????????$?'??????
4K????=??V&???f??N??5???'???@? A?
????F????}???^????Q?-w?Z?d?=v??X??????YQ????
?N?N?????
}???L?[?R?1V??R?n?i??%?Rg???b?re}???????DS?'???C??'?????g?f,??
?????)?e?G??.?Ga??~
????(?~??E\m6????M????w_?p?????l???(8
??"I????N??????3?Yr?I%Q&?k??O???$???gw1??BVZ???z?????t??H?v??*R??[???l(????W??????/??]???????.???k????
????!?????????-A?1?d?m? ?u??n?????"????.?*?
?R3)??Y2?????jT?pD??fRo???O?$3|?|??c??U??????X?H???
?????FL4???H,??w?????I??BBr?T??i0??Mq?8???u????x????B?]G??!c??6y???d38????6????.H??????K|?v?S???7????Q???l?Q??^?S??5?$?P
?{!?
?Vqh?k???1v?@?2 ?
H;eu/?Y????c??^-?w5?5S?X!? ?^??"/??zA"?<*#<?\?p
4?~fe?5??!
??c??:eY6??lO?u9!??P?9??I??9?cbJ?:/?]%??:????/;q??Z?C=?T?7=??{(xr>)????~86?3?Z>9????}z???R?x?>gH3J|?,?
R>bp????@6
??lD.?<gnF'???9V
Gmv.!?gJm??J,<???6JO?~??K
??L????JL?Y8?5M???
?*O?S??3P
??S?Pu'H?'Em?z(S??)U?T3???T&?$?YYU??,_OVAE???V?F???W???'?^W??????Y6>??t?Yb,l???ZzD??L?Y?E??\1??w(\u?)x2?\?A??)J^?Q
?_7l??_o?1G??`?????a5r??a ?p\???a?
44?? ?T
??c#?F?'c6???j?eWS???e??EUh>f???????
i???,%m?"?-k?iG;??B?in?k??i???
{o??s??pk???lq?B??.r??L
s?????s?????Hs??Mn?`
?ti?A?t?? ????
?R?xUN?(Iy?G?@?y??+???zL[3>??z?J?oH?{????C?|??????}?b????L
/?3"}? ?)g)?|0
{noformat}
> Decommissioned nodes can remain in gossip
> -----------------------------------------
>
> Key: CASSANDRA-10371
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10371
> Project: Cassandra
> Issue Type: Bug
> Components: Distributed Metadata
> Reporter: Brandon Williams
> Assignee: Joel Knighton
> Priority: Minor
>
> This may apply to other dead states as well. Dead states should be expired
> after 3 days. In the case of decom we attach a timestamp to let the other
> nodes know when it should be expired. It has been observed that sometimes a
> subset of nodes in the cluster never expire the state, and through heap
> analysis of these nodes it is revealed that the epstate.isAlive check returns
> true when it should return false, which would allow the state to be evicted.
> This may have been affected by CASSANDRA-8336.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
