[jira] [Commented] (CASSANDRA-2961) Expire dead gossip states based on time
[ https://issues.apache.org/jira/browse/CASSANDRA-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13113778#comment-13113778 ] paul cannon commented on CASSANDRA-2961: +1 Expire dead gossip states based on time --- Key: CASSANDRA-2961 URL: https://issues.apache.org/jira/browse/CASSANDRA-2961 Project: Cassandra Issue Type: Improvement Affects Versions: 1.0.0 Reporter: Brandon Williams Assignee: Jérémy Sevellec Priority: Minor Fix For: 1.0.1 Attachments: trunk-2961-v2.patch, trunk-2961-v3.patch, trunk-2961-v4.patch, trunk-2961.patch Currently dead states are held until aVeryLongTime, 3 days. The problem is that if a node reboots within this period, it begins a new 3 days and will repopulate the ring with the dead state. While mostly harmless, perpetuating the state forever is at least wasting a small amount of bandwidth. Instead, we can expire states based on a ttl, which will require that the cluster be loosely time synced; within the quarantine period of 60s. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2961) Expire dead gossip states based on time
[ https://issues.apache.org/jira/browse/CASSANDRA-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13104867#comment-13104867 ] paul cannon commented on CASSANDRA-2961: So it looks like expireTimeEndpointMap is meant to contain absolute timestamps, not durations; however, the default expiretime value (as returned by o.a.c.gms.Gossiper.getExpireTimeForEndpoint()) is aVeryLongTime. That should probably be (System.currentTimeMillis() + aVeryLongTime). Also, I think it would make more sense to pass expire times into the VersionedValue constructors, not call into the Gossiper from inside them. Expire dead gossip states based on time --- Key: CASSANDRA-2961 URL: https://issues.apache.org/jira/browse/CASSANDRA-2961 Project: Cassandra Issue Type: Improvement Affects Versions: 1.0.0 Reporter: Brandon Williams Assignee: Jérémy Sevellec Priority: Minor Fix For: 1.0.1 Attachments: trunk-2961-v2.patch, trunk-2961-v3.patch, trunk-2961.patch Currently dead states are held until aVeryLongTime, 3 days. The problem is that if a node reboots within this period, it begins a new 3 days and will repopulate the ring with the dead state. While mostly harmless, perpetuating the state forever is at least wasting a small amount of bandwidth. Instead, we can expire states based on a ttl, which will require that the cluster be loosely time synced; within the quarantine period of 60s. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2961) Expire dead gossip states based on time
[ https://issues.apache.org/jira/browse/CASSANDRA-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100123#comment-13100123 ] Jérémy Sevellec commented on CASSANDRA-2961: I found why i have make 2 call of addExpireTimeIfFound into SS instead of calling it into excise : There is 3 calls to excise in SS : handleStateLeft, handleStateRemoving and... removeToken. In removeToken, we don't have the pieces of the VV which contain expireTime. So we can't extract an expireTime. so there is 3 possibilities : - modify excise to add pieces parameter, set it to null in the removeToken call and manage the case if pieces is null. I find this solution not so beautiful but will work. - refactor and create 2 method signature of excise : one with pieces parameter and one without. - keep as it is. It's as you want. Tell me what is your preference (or another). Expire dead gossip states based on time --- Key: CASSANDRA-2961 URL: https://issues.apache.org/jira/browse/CASSANDRA-2961 Project: Cassandra Issue Type: Improvement Affects Versions: 1.0 Reporter: Brandon Williams Priority: Minor Fix For: 1.0 Attachments: trunk-2961-v2.patch, trunk-2961.patch Currently dead states are held until aVeryLongTime, 3 days. The problem is that if a node reboots within this period, it begins a new 3 days and will repopulate the ring with the dead state. While mostly harmless, perpetuating the state forever is at least wasting a small amount of bandwidth. Instead, we can expire states based on a ttl, which will require that the cluster be loosely time synced; within the quarantine period of 60s. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2961) Expire dead gossip states based on time
[ https://issues.apache.org/jira/browse/CASSANDRA-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100468#comment-13100468 ] Brandon Williams commented on CASSANDRA-2961: - bq. hamscrest : In my case, It's true, I just use hamcrest with is into assert. There is a lot of other verb which interesting to make asserting more readable. Tt was for help for next but if you want I can remove it. tell me you do you prefer. I don't think it's providing enough utility yet to justify another dependency. bq. VersionedValue.getExpireTime : It's true, I put it in the Gossiper? a utility class? I think I would pass the expire times to the VV constructors and put the actual generation of the times in Gossiper. bq. There is 3 calls to excise in SS : handleStateLeft, handleStateRemoving and... removeToken. In removeToken, we don't have the pieces of the VV which contain expireTime. So we can't extract an expireTime. That's because in removeToken, we are responsible for the generation of the expireTime (we are the removal coordinator.) Expire dead gossip states based on time --- Key: CASSANDRA-2961 URL: https://issues.apache.org/jira/browse/CASSANDRA-2961 Project: Cassandra Issue Type: Improvement Affects Versions: 1.0 Reporter: Brandon Williams Priority: Minor Fix For: 1.1 Attachments: trunk-2961-v2.patch, trunk-2961.patch Currently dead states are held until aVeryLongTime, 3 days. The problem is that if a node reboots within this period, it begins a new 3 days and will repopulate the ring with the dead state. While mostly harmless, perpetuating the state forever is at least wasting a small amount of bandwidth. Instead, we can expire states based on a ttl, which will require that the cluster be loosely time synced; within the quarantine period of 60s. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2961) Expire dead gossip states based on time
[ https://issues.apache.org/jira/browse/CASSANDRA-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100637#comment-13100637 ] Jérémy Sevellec commented on CASSANDRA-2961: You can find an new version of the patch : - without hamcrest dependency - compute the generation of expireTime into gossiper and calling it into the constructor of VV - modify SS to be more readable I hope it's ok :-) Expire dead gossip states based on time --- Key: CASSANDRA-2961 URL: https://issues.apache.org/jira/browse/CASSANDRA-2961 Project: Cassandra Issue Type: Improvement Affects Versions: 1.0 Reporter: Brandon Williams Priority: Minor Fix For: 1.1 Attachments: trunk-2961-v2.patch, trunk-2961-v3.patch, trunk-2961.patch Currently dead states are held until aVeryLongTime, 3 days. The problem is that if a node reboots within this period, it begins a new 3 days and will repopulate the ring with the dead state. While mostly harmless, perpetuating the state forever is at least wasting a small amount of bandwidth. Instead, we can expire states based on a ttl, which will require that the cluster be loosely time synced; within the quarantine period of 60s. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2961) Expire dead gossip states based on time
[ https://issues.apache.org/jira/browse/CASSANDRA-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13098984#comment-13098984 ] Brandon Williams commented on CASSANDRA-2961: - Can you rebase without the line reformatting? Expire dead gossip states based on time --- Key: CASSANDRA-2961 URL: https://issues.apache.org/jira/browse/CASSANDRA-2961 Project: Cassandra Issue Type: Improvement Affects Versions: 1.0 Reporter: Brandon Williams Labels: patch Fix For: 1.0 Attachments: trunk-2961.patch Currently dead states are held until aVeryLongTime, 3 days. The problem is that if a node reboots within this period, it begins a new 3 days and will repopulate the ring with the dead state. While mostly harmless, perpetuating the state forever is at least wasting a small amount of bandwidth. Instead, we can expire states based on a ttl, which will require that the cluster be loosely time synced; within the quarantine period of 60s. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2961) Expire dead gossip states based on time
[ https://issues.apache.org/jira/browse/CASSANDRA-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13099052#comment-13099052 ] Jérémy Sevellec commented on CASSANDRA-2961: ok. I thought it was a good idea but I realize that to inspect the patch, it is difficult. I will publish a new patch version as soon as possible. Expire dead gossip states based on time --- Key: CASSANDRA-2961 URL: https://issues.apache.org/jira/browse/CASSANDRA-2961 Project: Cassandra Issue Type: Improvement Affects Versions: 1.0 Reporter: Brandon Williams Labels: patch Fix For: 1.0 Attachments: trunk-2961.patch Currently dead states are held until aVeryLongTime, 3 days. The problem is that if a node reboots within this period, it begins a new 3 days and will repopulate the ring with the dead state. While mostly harmless, perpetuating the state forever is at least wasting a small amount of bandwidth. Instead, we can expire states based on a ttl, which will require that the cluster be loosely time synced; within the quarantine period of 60s. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2961) Expire dead gossip states based on time
[ https://issues.apache.org/jira/browse/CASSANDRA-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13099062#comment-13099062 ] Brandon Williams commented on CASSANDRA-2961: - We typically don't adhere to 80 columns anyway, java is too verbose :) Expire dead gossip states based on time --- Key: CASSANDRA-2961 URL: https://issues.apache.org/jira/browse/CASSANDRA-2961 Project: Cassandra Issue Type: Improvement Affects Versions: 1.0 Reporter: Brandon Williams Labels: patch Fix For: 1.0 Attachments: trunk-2961.patch Currently dead states are held until aVeryLongTime, 3 days. The problem is that if a node reboots within this period, it begins a new 3 days and will repopulate the ring with the dead state. While mostly harmless, perpetuating the state forever is at least wasting a small amount of bandwidth. Instead, we can expire states based on a ttl, which will require that the cluster be loosely time synced; within the quarantine period of 60s. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2961) Expire dead gossip states based on time
[ https://issues.apache.org/jira/browse/CASSANDRA-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13099067#comment-13099067 ] Jérémy Sevellec commented on CASSANDRA-2961: yes it's true and it'is my code style configuration :-( Expire dead gossip states based on time --- Key: CASSANDRA-2961 URL: https://issues.apache.org/jira/browse/CASSANDRA-2961 Project: Cassandra Issue Type: Improvement Affects Versions: 1.0 Reporter: Brandon Williams Labels: patch Fix For: 1.0 Attachments: trunk-2961.patch Currently dead states are held until aVeryLongTime, 3 days. The problem is that if a node reboots within this period, it begins a new 3 days and will repopulate the ring with the dead state. While mostly harmless, perpetuating the state forever is at least wasting a small amount of bandwidth. Instead, we can expire states based on a ttl, which will require that the cluster be loosely time synced; within the quarantine period of 60s. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2961) Expire dead gossip states based on time
[ https://issues.apache.org/jira/browse/CASSANDRA-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13099466#comment-13099466 ] Brandon Williams commented on CASSANDRA-2961: - A few things: * I don't think it's worth pulling in the hamcrest dependency for 'is' instead of writing assertEquals(1L, expireTime) * VersionedValue.getExpireTime feels like the wrong place to me for that logic, but I could be wrong * rather than having multiple calls to addExpireTimeIfFound let's put this in excise() * some DEBUG logging to know when an endpoint is going to be expired (and whether a timestamp was supplied or not) could be helpful in the future Expire dead gossip states based on time --- Key: CASSANDRA-2961 URL: https://issues.apache.org/jira/browse/CASSANDRA-2961 Project: Cassandra Issue Type: Improvement Affects Versions: 1.0 Reporter: Brandon Williams Labels: patch Fix For: 1.0 Attachments: trunk-2961-v2.patch, trunk-2961.patch Currently dead states are held until aVeryLongTime, 3 days. The problem is that if a node reboots within this period, it begins a new 3 days and will repopulate the ring with the dead state. While mostly harmless, perpetuating the state forever is at least wasting a small amount of bandwidth. Instead, we can expire states based on a ttl, which will require that the cluster be loosely time synced; within the quarantine period of 60s. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2961) Expire dead gossip states based on time
[ https://issues.apache.org/jira/browse/CASSANDRA-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100089#comment-13100089 ] Jérémy Sevellec commented on CASSANDRA-2961: ok I like it, it's few things :-) : - hamscrest : In my case, It's true, I just use hamcrest with is into assert. There is a lot of other verb which interesting to make asserting more readable. Tt was for help for next but if you want I can remove it. tell me you do you prefer. - VersionedValue.getExpireTime : It's true, I put it in the Gossiper? a utility class? - addExpireTimeIfFound : ok i put one call in excise but i keep the method to isolate the thinking. if you're ok. - DEBUG log : ho there was (to make my test), but i remove it before creating the patch... I add them again Expire dead gossip states based on time --- Key: CASSANDRA-2961 URL: https://issues.apache.org/jira/browse/CASSANDRA-2961 Project: Cassandra Issue Type: Improvement Affects Versions: 1.0 Reporter: Brandon Williams Priority: Minor Fix For: 1.0 Attachments: trunk-2961-v2.patch, trunk-2961.patch Currently dead states are held until aVeryLongTime, 3 days. The problem is that if a node reboots within this period, it begins a new 3 days and will repopulate the ring with the dead state. While mostly harmless, perpetuating the state forever is at least wasting a small amount of bandwidth. Instead, we can expire states based on a ttl, which will require that the cluster be loosely time synced; within the quarantine period of 60s. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2961) Expire dead gossip states based on time
[ https://issues.apache.org/jira/browse/CASSANDRA-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096789#comment-13096789 ] Jérémy Sevellec commented on CASSANDRA-2961: In addition, I have include a lib for test (hamcrest) to simplify the writing of assertion into Junit. Expire dead gossip states based on time --- Key: CASSANDRA-2961 URL: https://issues.apache.org/jira/browse/CASSANDRA-2961 Project: Cassandra Issue Type: Improvement Affects Versions: 1.0 Reporter: Brandon Williams Labels: patch Fix For: 1.0 Attachments: trunk-2961.patch Currently dead states are held until aVeryLongTime, 3 days. The problem is that if a node reboots within this period, it begins a new 3 days and will repopulate the ring with the dead state. While mostly harmless, perpetuating the state forever is at least wasting a small amount of bandwidth. Instead, we can expire states based on a ttl, which will require that the cluster be loosely time synced; within the quarantine period of 60s. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2961) Expire dead gossip states based on time
[ https://issues.apache.org/jira/browse/CASSANDRA-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095967#comment-13095967 ] Jérémy Sevellec commented on CASSANDRA-2961: What do you think about : - create a Map of expireTimeEndpointMap into Gossiper wich store endpoints as key and expireTime as value. - SS, when a state change : - if STATUS is REMOVED_TOKEN or STATUS_LEFT, extract the expireTime in the string a the end of the VV and call the Gossiper to add the endpoint/expireTime into the expireTimeEndpointMap. For all other state - for all other STATUS, call the gossiper to remove the endpoint into expireTimeEndpointMap if it is present. - Gossiper, when doing status check for each endpoint, verifying if there is an expireTime in expireTimeEndpointMap for this endpoint, if so, we have an expireTime, if not, expireTime is set with aVeryLongTime. test and evict if necessary the endpoint. It makes sense for you? (I describe a lot... sorry but i would like to be sure of good understanding all aspect of the problem...) Expire dead gossip states based on time --- Key: CASSANDRA-2961 URL: https://issues.apache.org/jira/browse/CASSANDRA-2961 Project: Cassandra Issue Type: Improvement Reporter: Brandon Williams Fix For: 1.0 Currently dead states are held until aVeryLongTime, 3 days. The problem is that if a node reboots within this period, it begins a new 3 days and will repopulate the ring with the dead state. While mostly harmless, perpetuating the state forever is at least wasting a small amount of bandwidth. Instead, we can expire states based on a ttl, which will require that the cluster be loosely time synced; within the quarantine period of 60s. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2961) Expire dead gossip states based on time
[ https://issues.apache.org/jira/browse/CASSANDRA-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096057#comment-13096057 ] Brandon Williams commented on CASSANDRA-2961: - This sounds good, a couple of points though: bq. for all other STATUS, call the gossiper to remove the endpoint into expireTimeEndpointMap if it is present. REMOVED_TOKEN and STATUS_LEFT are the only states we need to worry about expiring. bq. Gossiper, when doing status check for each endpoint, verifying if there is an expireTime in expireTimeEndpointMap for this endpoint, if so, we have an expireTime, if not, expireTime is set with aVeryLongTime. test and evict if necessary the endpoint. I think if there is no expireTime we should just respect aVLT as we currently do, instead of populating expireTimeEndpointMap. That way when debugging we can tell if the remote side gave us an expireTime or not, otherwise it won't be distinguishable. Expire dead gossip states based on time --- Key: CASSANDRA-2961 URL: https://issues.apache.org/jira/browse/CASSANDRA-2961 Project: Cassandra Issue Type: Improvement Reporter: Brandon Williams Fix For: 1.0 Currently dead states are held until aVeryLongTime, 3 days. The problem is that if a node reboots within this period, it begins a new 3 days and will repopulate the ring with the dead state. While mostly harmless, perpetuating the state forever is at least wasting a small amount of bandwidth. Instead, we can expire states based on a ttl, which will require that the cluster be loosely time synced; within the quarantine period of 60s. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2961) Expire dead gossip states based on time
[ https://issues.apache.org/jira/browse/CASSANDRA-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096117#comment-13096117 ] Jérémy Sevellec commented on CASSANDRA-2961: I agree with you about the fact that : REMOVED_TOKEN and STATUS_LEFT are the only states we need to worry about expiring. But if SS : - first receive a change for an endpoint with the status by REMOVED_TOKEN or STATUS_LEFT - and then for this same endpoint receiving an other change with one of other status we have to delete the expireTime because the gossiper will remove this endpoint when expireTime will be exceeded and it must not? no? Expire dead gossip states based on time --- Key: CASSANDRA-2961 URL: https://issues.apache.org/jira/browse/CASSANDRA-2961 Project: Cassandra Issue Type: Improvement Reporter: Brandon Williams Fix For: 1.0 Currently dead states are held until aVeryLongTime, 3 days. The problem is that if a node reboots within this period, it begins a new 3 days and will repopulate the ring with the dead state. While mostly harmless, perpetuating the state forever is at least wasting a small amount of bandwidth. Instead, we can expire states based on a ttl, which will require that the cluster be loosely time synced; within the quarantine period of 60s. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2961) Expire dead gossip states based on time
[ https://issues.apache.org/jira/browse/CASSANDRA-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096124#comment-13096124 ] Brandon Williams commented on CASSANDRA-2961: - Yes, we'll need to perform housekeeping on expireTimeEndpointMap. The simplest thing to do is delete the entry when marking it alive, since there is no other way for a dead state to change. Expire dead gossip states based on time --- Key: CASSANDRA-2961 URL: https://issues.apache.org/jira/browse/CASSANDRA-2961 Project: Cassandra Issue Type: Improvement Reporter: Brandon Williams Fix For: 1.0 Currently dead states are held until aVeryLongTime, 3 days. The problem is that if a node reboots within this period, it begins a new 3 days and will repopulate the ring with the dead state. While mostly harmless, perpetuating the state forever is at least wasting a small amount of bandwidth. Instead, we can expire states based on a ttl, which will require that the cluster be loosely time synced; within the quarantine period of 60s. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2961) Expire dead gossip states based on time
[ https://issues.apache.org/jira/browse/CASSANDRA-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13083281#comment-13083281 ] Brandon Williams commented on CASSANDRA-2961: - It looks like as a benefit of our ghetto string concatenation/delimiting in VersionedValue, we can just append timestamps to states that need to expire, then respect those in the gossiper (if present) instead of aVLT, but we still need to use aVLT if not. This way, everything will stay backward-compatible, since older nodes will just get extra 'pieces' in SS notifications and not use them. We probably only need to do this for the removed and left states, since the fat client removal logic is already solid and a timestamp won't really help there anyway. Expire dead gossip states based on time --- Key: CASSANDRA-2961 URL: https://issues.apache.org/jira/browse/CASSANDRA-2961 Project: Cassandra Issue Type: Improvement Reporter: Brandon Williams Assignee: Patricio Echague Currently dead states are held until aVeryLongTime, 3 days. The problem is that if a node reboots within this period, it begins a new 3 days and will repopulate the ring with the dead state. While mostly harmless, perpetuating the state forever is at least wasting a small amount of bandwidth. Instead, we can expire states based on a ttl, which will require that the cluster be loosely time synced; within the quarantine period of 60s. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
