Re: Consistent vs inconsistent range movements
Also https://issues.apache.org/jira/browse/CASSANDRA-11005 On Fri, Mar 3, 2017 at 3:57 PM, Brandon Williamswrote: > https://issues.apache.org/jira/browse/CASSANDRA-11610 > > Disable only if adding single nodes to different DCs or racks. > > On Fri, Mar 3, 2017 at 5:44 PM, Jeff Jirsa wrote: > > > Imagine you have a cluster with RF=3, and you write a key with CL:QUORUM, > > it goes to nodes 1 and 3, but node 2 is offline. > > > > Some time later, node 2 comes online. > > > > Then you want to add node 2.5 in between nodes 1 and 3. > > > > If you stream data from node 2, you violate consistency guarantees > (quorum) > > - for data on node 2.5, you'll have data equal to node 2, which means a > > query with CL:QUORUM may query nodes 2 and 2.5 and return "no data", even > > though it's on node 1 (and was on node 3, before 3 stopped being a > > replica). > > > > To solve this problem, https://issues.apache.org/ > > jira/browse/CASSANDRA-2434 > > made it so we always try to do "consistent range movements" - that is, > when > > we add node 2.5, we stream from the replica it's replacing (3) - now when > > you query with quorum, you'll be guaranteed that 2 of the 3 remaining > > replicas with the data (if you wrote with quorum), and everything works > as > > expected. > > > > This guarantee isn't free - strict range movements can limit your ability > > to move (or join, or remove) multiple nodes at the same time (especially > in > > VNode clusters), so there's a system property to turn it off if you > really > > really really really really know what you're doing (which I imagine you > see > > in the code, which is why you're asking). > > > > Definitely read CASSANDRA-2434 . That's probably the best documentation > of > > this feature. > > > > > > > > > > On Fri, Mar 3, 2017 at 2:04 PM, benjamin roth wrote: > > > > > Hi, > > > > > > Can anyone tell the difference between consistent + inconsistent range > > > movements? > > > What exactly makes them consistent or inconsistent? > > > In what situations can both of them occur? > > > > > > It would be great to get a correct and deep understanding of that for > > > further MV improvments. My intuition tells me that a rebuild / > removenode > > > can break MV consistency, but to prove it I need more information. > > > I am also happy about code references - it's just very tedious to read > > all > > > through the code to get an overview of all that without some prose > > > information. > > > > > > Thanks in advance > > > > > >
Re: Consistent vs inconsistent range movements
https://issues.apache.org/jira/browse/CASSANDRA-11610 Disable only if adding single nodes to different DCs or racks. On Fri, Mar 3, 2017 at 5:44 PM, Jeff Jirsawrote: > Imagine you have a cluster with RF=3, and you write a key with CL:QUORUM, > it goes to nodes 1 and 3, but node 2 is offline. > > Some time later, node 2 comes online. > > Then you want to add node 2.5 in between nodes 1 and 3. > > If you stream data from node 2, you violate consistency guarantees (quorum) > - for data on node 2.5, you'll have data equal to node 2, which means a > query with CL:QUORUM may query nodes 2 and 2.5 and return "no data", even > though it's on node 1 (and was on node 3, before 3 stopped being a > replica). > > To solve this problem, https://issues.apache.org/ > jira/browse/CASSANDRA-2434 > made it so we always try to do "consistent range movements" - that is, when > we add node 2.5, we stream from the replica it's replacing (3) - now when > you query with quorum, you'll be guaranteed that 2 of the 3 remaining > replicas with the data (if you wrote with quorum), and everything works as > expected. > > This guarantee isn't free - strict range movements can limit your ability > to move (or join, or remove) multiple nodes at the same time (especially in > VNode clusters), so there's a system property to turn it off if you really > really really really really know what you're doing (which I imagine you see > in the code, which is why you're asking). > > Definitely read CASSANDRA-2434 . That's probably the best documentation of > this feature. > > > > > On Fri, Mar 3, 2017 at 2:04 PM, benjamin roth wrote: > > > Hi, > > > > Can anyone tell the difference between consistent + inconsistent range > > movements? > > What exactly makes them consistent or inconsistent? > > In what situations can both of them occur? > > > > It would be great to get a correct and deep understanding of that for > > further MV improvments. My intuition tells me that a rebuild / removenode > > can break MV consistency, but to prove it I need more information. > > I am also happy about code references - it's just very tedious to read > all > > through the code to get an overview of all that without some prose > > information. > > > > Thanks in advance > > >
Re: Consistent vs inconsistent range movements
Imagine you have a cluster with RF=3, and you write a key with CL:QUORUM, it goes to nodes 1 and 3, but node 2 is offline. Some time later, node 2 comes online. Then you want to add node 2.5 in between nodes 1 and 3. If you stream data from node 2, you violate consistency guarantees (quorum) - for data on node 2.5, you'll have data equal to node 2, which means a query with CL:QUORUM may query nodes 2 and 2.5 and return "no data", even though it's on node 1 (and was on node 3, before 3 stopped being a replica). To solve this problem, https://issues.apache.org/jira/browse/CASSANDRA-2434 made it so we always try to do "consistent range movements" - that is, when we add node 2.5, we stream from the replica it's replacing (3) - now when you query with quorum, you'll be guaranteed that 2 of the 3 remaining replicas with the data (if you wrote with quorum), and everything works as expected. This guarantee isn't free - strict range movements can limit your ability to move (or join, or remove) multiple nodes at the same time (especially in VNode clusters), so there's a system property to turn it off if you really really really really really know what you're doing (which I imagine you see in the code, which is why you're asking). Definitely read CASSANDRA-2434 . That's probably the best documentation of this feature. On Fri, Mar 3, 2017 at 2:04 PM, benjamin rothwrote: > Hi, > > Can anyone tell the difference between consistent + inconsistent range > movements? > What exactly makes them consistent or inconsistent? > In what situations can both of them occur? > > It would be great to get a correct and deep understanding of that for > further MV improvments. My intuition tells me that a rebuild / removenode > can break MV consistency, but to prove it I need more information. > I am also happy about code references - it's just very tedious to read all > through the code to get an overview of all that without some prose > information. > > Thanks in advance >