RE: vnode and NetworkTopologyStrategy: not playing well together ?
The discussion about racks NTS is also mentioned in this recent article : planetcassandra.org/multi-data-center-replication-in-nosql-databases/ The last section may be of interest for you Thanks DuyHai. Note that this section is also part of C* anti-patterns http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architecturePlanningAntiPatterns_c.html But I think it's missing some advice for vnodes (something like due to randomly-generated tokens, define one single rack when using vnodes ??). D. [@@ THALES GROUP INTERNAL @@] De : DuyHai Doan [mailto:doanduy...@gmail.com] Envoyé : mardi 5 août 2014 20:07 À : user@cassandra.apache.org Objet : RE: vnode and NetworkTopologyStrategy: not playing well together ? The discussion about racks NTS is also mentioned in this recent article : planetcassandra.org/multi-data-center-replication-in-nosql-databases/http://planetcassandra.org/multi-data-center-replication-in-nosql-databases/ The last section may be of interest for you Le 5 août 2014 18:14, DE VITO Dominique dominique.dev...@thalesgroup.commailto:dominique.dev...@thalesgroup.com a écrit : Jonathan wrote: Yes, if you have only 1 machine in a rack then your cluster will be imbalanced. You're going to be able to dream up all sorts of weird failure cases when you choose a scenario like RF=2 totally imbalanced network arch. Vnodes attempt to solve the problem of imbalanced rings by choosing so many tokens that it's improbable that the ring will be imbalanced. Storage/load distro = function(1st replica placement, other replica placement) vnode solves the balancing pb for 1st replica placement // so, yes, I agree with you, but for 1st replica placement only But NetworkTopologyStrategy (NTS) influences other (2+) replica placement = as NTS best behavior relies on token distro, and you have no control on tokens with vnodes, the best option I see with **vnode** is to use only one rack with NTS. Dominique -Message d'origine- De : jonathan.had...@gmail.commailto:jonathan.had...@gmail.com [mailto:jonathan.had...@gmail.commailto:jonathan.had...@gmail.com] De la part de Jonathan Haddad Envoyé : mardi 5 août 2014 18:04 À : user@cassandra.apache.orgmailto:user@cassandra.apache.org Objet : Re: vnode and NetworkTopologyStrategy: not playing well together ? Yes, if you have only 1 machine in a rack then your cluster will be imbalanced. You're going to be able to dream up all sorts of weird failure cases when you choose a scenario like RF=2 totally imbalanced network arch. Vnodes attempt to solve the problem of imbalanced rings by choosing so many tokens that it's improbable that the ring will be imbalanced. On Tue, Aug 5, 2014 at 8:57 AM, DE VITO Dominique dominique.dev...@thalesgroup.commailto:dominique.dev...@thalesgroup.com wrote: First, thanks for your answer. This is incorrect. Network Topology w/ Vnodes will be fine, assuming you've got RF= # of racks. IMHO, it's not a good enough condition. Let's use an example with RF=2 N1/rack_1 N2/rack_1 N3/rack_1 N4/rack_2 Here, you have RF= # of racks And due to NetworkTopologyStrategy, N4 will store *all* the cluster data, leading to a completely imbalanced cluster. IMHO, it happens when using nodes *or* vnodes. As well-balanced clusters with NetworkTopologyStrategy rely on carefully chosen token distribution/path along the ring *and* as tokens are randomly-generated with vnodes, my guess is that with vnodes and NetworkTopologyStrategy, it's better to define a single (logical) rack // due to carefully chosen tokens vs randomly-generated token clash. I don't see other options left. Do you see other ones ? Regards, Dominique -Message d'origine- De : jonathan.had...@gmail.commailto:jonathan.had...@gmail.com [mailto:jonathan.had...@gmail.commailto:jonathan.had...@gmail.com] De la part de Jonathan Haddad Envoyé : mardi 5 août 2014 17:43 À : user@cassandra.apache.orgmailto:user@cassandra.apache.org Objet : Re: vnode and NetworkTopologyStrategy: not playing well together ? This is incorrect. Network Topology w/ Vnodes will be fine, assuming you've got RF= # of racks. For each token, replicas are chosen based on the strategy. Essentially, you could have a wild imbalance in token ownership, but it wouldn't matter because the replicas would be distributed across the rest of the machines. http://www.datastax.com/documentation/cassandra/2.0/cassandra/architec ture/architectureDataDistributeReplication_c.html On Tue, Aug 5, 2014 at 8:19 AM, DE VITO Dominique dominique.dev...@thalesgroup.commailto:dominique.dev...@thalesgroup.com wrote: Hi, My understanding is that NetworkTopologyStrategy does NOT play well with vnodes, due to: · Vnode = tokens are (usually) randomly generated (AFAIK) · NetworkTopologyStrategy = required carefully choosen tokens for all nodes in order to not to get a VERY unbalanced ring
Re: vnode and NetworkTopologyStrategy: not playing well together ?
This is incorrect. Network Topology w/ Vnodes will be fine, assuming you've got RF= # of racks. For each token, replicas are chosen based on the strategy. Essentially, you could have a wild imbalance in token ownership, but it wouldn't matter because the replicas would be distributed across the rest of the machines. http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureDataDistributeReplication_c.html On Tue, Aug 5, 2014 at 8:19 AM, DE VITO Dominique dominique.dev...@thalesgroup.com wrote: Hi, My understanding is that NetworkTopologyStrategy does NOT play well with vnodes, due to: · Vnode = tokens are (usually) randomly generated (AFAIK) · NetworkTopologyStrategy = required carefully choosen tokens for all nodes in order to not to get a VERY unbalanced ring like in https://issues.apache.org/jira/browse/CASSANDRA-3810 When playing with vnodes, is the recommendation to define one rack for the entire cluster ? Thanks. Regards, Dominique -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade
Re: vnode and NetworkTopologyStrategy: not playing well together ?
* When I say wild imbalance, I do not mean all tokens on 1 node in the cluster, I really should have said slightly imbalanced On Tue, Aug 5, 2014 at 8:43 AM, Jonathan Haddad j...@jonhaddad.com wrote: This is incorrect. Network Topology w/ Vnodes will be fine, assuming you've got RF= # of racks. For each token, replicas are chosen based on the strategy. Essentially, you could have a wild imbalance in token ownership, but it wouldn't matter because the replicas would be distributed across the rest of the machines. http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureDataDistributeReplication_c.html On Tue, Aug 5, 2014 at 8:19 AM, DE VITO Dominique dominique.dev...@thalesgroup.com wrote: Hi, My understanding is that NetworkTopologyStrategy does NOT play well with vnodes, due to: · Vnode = tokens are (usually) randomly generated (AFAIK) · NetworkTopologyStrategy = required carefully choosen tokens for all nodes in order to not to get a VERY unbalanced ring like in https://issues.apache.org/jira/browse/CASSANDRA-3810 When playing with vnodes, is the recommendation to define one rack for the entire cluster ? Thanks. Regards, Dominique -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade
RE: vnode and NetworkTopologyStrategy: not playing well together ?
First, thanks for your answer. This is incorrect. Network Topology w/ Vnodes will be fine, assuming you've got RF= # of racks. IMHO, it's not a good enough condition. Let's use an example with RF=2 N1/rack_1 N2/rack_1 N3/rack_1 N4/rack_2 Here, you have RF= # of racks And due to NetworkTopologyStrategy, N4 will store *all* the cluster data, leading to a completely imbalanced cluster. IMHO, it happens when using nodes *or* vnodes. As well-balanced clusters with NetworkTopologyStrategy rely on carefully chosen token distribution/path along the ring *and* as tokens are randomly-generated with vnodes, my guess is that with vnodes and NetworkTopologyStrategy, it's better to define a single (logical) rack // due to carefully chosen tokens vs randomly-generated token clash. I don't see other options left. Do you see other ones ? Regards, Dominique -Message d'origine- De : jonathan.had...@gmail.com [mailto:jonathan.had...@gmail.com] De la part de Jonathan Haddad Envoyé : mardi 5 août 2014 17:43 À : user@cassandra.apache.org Objet : Re: vnode and NetworkTopologyStrategy: not playing well together ? This is incorrect. Network Topology w/ Vnodes will be fine, assuming you've got RF= # of racks. For each token, replicas are chosen based on the strategy. Essentially, you could have a wild imbalance in token ownership, but it wouldn't matter because the replicas would be distributed across the rest of the machines. http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureDataDistributeReplication_c.html On Tue, Aug 5, 2014 at 8:19 AM, DE VITO Dominique dominique.dev...@thalesgroup.com wrote: Hi, My understanding is that NetworkTopologyStrategy does NOT play well with vnodes, due to: · Vnode = tokens are (usually) randomly generated (AFAIK) · NetworkTopologyStrategy = required carefully choosen tokens for all nodes in order to not to get a VERY unbalanced ring like in https://issues.apache.org/jira/browse/CASSANDRA-3810 When playing with vnodes, is the recommendation to define one rack for the entire cluster ? Thanks. Regards, Dominique -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade
Re: vnode and NetworkTopologyStrategy: not playing well together ?
If your nodes are not actually evenly distributed across physical racks for redundancy, don't use multiple racks. On Tue, Aug 5, 2014 at 10:57 AM, DE VITO Dominique dominique.dev...@thalesgroup.com wrote: First, thanks for your answer. This is incorrect. Network Topology w/ Vnodes will be fine, assuming you've got RF= # of racks. IMHO, it's not a good enough condition. Let's use an example with RF=2 N1/rack_1 N2/rack_1 N3/rack_1 N4/rack_2 Here, you have RF= # of racks And due to NetworkTopologyStrategy, N4 will store *all* the cluster data, leading to a completely imbalanced cluster. IMHO, it happens when using nodes *or* vnodes. As well-balanced clusters with NetworkTopologyStrategy rely on carefully chosen token distribution/path along the ring *and* as tokens are randomly-generated with vnodes, my guess is that with vnodes and NetworkTopologyStrategy, it's better to define a single (logical) rack // due to carefully chosen tokens vs randomly-generated token clash. I don't see other options left. Do you see other ones ? Regards, Dominique -Message d'origine- De : jonathan.had...@gmail.com [mailto:jonathan.had...@gmail.com] De la part de Jonathan Haddad Envoyé : mardi 5 août 2014 17:43 À : user@cassandra.apache.org Objet : Re: vnode and NetworkTopologyStrategy: not playing well together ? This is incorrect. Network Topology w/ Vnodes will be fine, assuming you've got RF= # of racks. For each token, replicas are chosen based on the strategy. Essentially, you could have a wild imbalance in token ownership, but it wouldn't matter because the replicas would be distributed across the rest of the machines. http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureDataDistributeReplication_c.html On Tue, Aug 5, 2014 at 8:19 AM, DE VITO Dominique dominique.dev...@thalesgroup.com wrote: Hi, My understanding is that NetworkTopologyStrategy does NOT play well with vnodes, due to: · Vnode = tokens are (usually) randomly generated (AFAIK) · NetworkTopologyStrategy = required carefully choosen tokens for all nodes in order to not to get a VERY unbalanced ring like in https://issues.apache.org/jira/browse/CASSANDRA-3810 When playing with vnodes, is the recommendation to define one rack for the entire cluster ? Thanks. Regards, Dominique -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade
Re: vnode and NetworkTopologyStrategy: not playing well together ?
Yes, if you have only 1 machine in a rack then your cluster will be imbalanced. You're going to be able to dream up all sorts of weird failure cases when you choose a scenario like RF=2 totally imbalanced network arch. Vnodes attempt to solve the problem of imbalanced rings by choosing so many tokens that it's improbable that the ring will be imbalanced. On Tue, Aug 5, 2014 at 8:57 AM, DE VITO Dominique dominique.dev...@thalesgroup.com wrote: First, thanks for your answer. This is incorrect. Network Topology w/ Vnodes will be fine, assuming you've got RF= # of racks. IMHO, it's not a good enough condition. Let's use an example with RF=2 N1/rack_1 N2/rack_1 N3/rack_1 N4/rack_2 Here, you have RF= # of racks And due to NetworkTopologyStrategy, N4 will store *all* the cluster data, leading to a completely imbalanced cluster. IMHO, it happens when using nodes *or* vnodes. As well-balanced clusters with NetworkTopologyStrategy rely on carefully chosen token distribution/path along the ring *and* as tokens are randomly-generated with vnodes, my guess is that with vnodes and NetworkTopologyStrategy, it's better to define a single (logical) rack // due to carefully chosen tokens vs randomly-generated token clash. I don't see other options left. Do you see other ones ? Regards, Dominique -Message d'origine- De : jonathan.had...@gmail.com [mailto:jonathan.had...@gmail.com] De la part de Jonathan Haddad Envoyé : mardi 5 août 2014 17:43 À : user@cassandra.apache.org Objet : Re: vnode and NetworkTopologyStrategy: not playing well together ? This is incorrect. Network Topology w/ Vnodes will be fine, assuming you've got RF= # of racks. For each token, replicas are chosen based on the strategy. Essentially, you could have a wild imbalance in token ownership, but it wouldn't matter because the replicas would be distributed across the rest of the machines. http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureDataDistributeReplication_c.html On Tue, Aug 5, 2014 at 8:19 AM, DE VITO Dominique dominique.dev...@thalesgroup.com wrote: Hi, My understanding is that NetworkTopologyStrategy does NOT play well with vnodes, due to: · Vnode = tokens are (usually) randomly generated (AFAIK) · NetworkTopologyStrategy = required carefully choosen tokens for all nodes in order to not to get a VERY unbalanced ring like in https://issues.apache.org/jira/browse/CASSANDRA-3810 When playing with vnodes, is the recommendation to define one rack for the entire cluster ? Thanks. Regards, Dominique -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade
RE: vnode and NetworkTopologyStrategy: not playing well together ?
Jonathan wrote: Yes, if you have only 1 machine in a rack then your cluster will be imbalanced. You're going to be able to dream up all sorts of weird failure cases when you choose a scenario like RF=2 totally imbalanced network arch. Vnodes attempt to solve the problem of imbalanced rings by choosing so many tokens that it's improbable that the ring will be imbalanced. Storage/load distro = function(1st replica placement, other replica placement) vnode solves the balancing pb for 1st replica placement // so, yes, I agree with you, but for 1st replica placement only But NetworkTopologyStrategy (NTS) influences other (2+) replica placement = as NTS best behavior relies on token distro, and you have no control on tokens with vnodes, the best option I see with **vnode** is to use only one rack with NTS. Dominique -Message d'origine- De : jonathan.had...@gmail.com [mailto:jonathan.had...@gmail.com] De la part de Jonathan Haddad Envoyé : mardi 5 août 2014 18:04 À : user@cassandra.apache.org Objet : Re: vnode and NetworkTopologyStrategy: not playing well together ? Yes, if you have only 1 machine in a rack then your cluster will be imbalanced. You're going to be able to dream up all sorts of weird failure cases when you choose a scenario like RF=2 totally imbalanced network arch. Vnodes attempt to solve the problem of imbalanced rings by choosing so many tokens that it's improbable that the ring will be imbalanced. On Tue, Aug 5, 2014 at 8:57 AM, DE VITO Dominique dominique.dev...@thalesgroup.com wrote: First, thanks for your answer. This is incorrect. Network Topology w/ Vnodes will be fine, assuming you've got RF= # of racks. IMHO, it's not a good enough condition. Let's use an example with RF=2 N1/rack_1 N2/rack_1 N3/rack_1 N4/rack_2 Here, you have RF= # of racks And due to NetworkTopologyStrategy, N4 will store *all* the cluster data, leading to a completely imbalanced cluster. IMHO, it happens when using nodes *or* vnodes. As well-balanced clusters with NetworkTopologyStrategy rely on carefully chosen token distribution/path along the ring *and* as tokens are randomly-generated with vnodes, my guess is that with vnodes and NetworkTopologyStrategy, it's better to define a single (logical) rack // due to carefully chosen tokens vs randomly-generated token clash. I don't see other options left. Do you see other ones ? Regards, Dominique -Message d'origine- De : jonathan.had...@gmail.com [mailto:jonathan.had...@gmail.com] De la part de Jonathan Haddad Envoyé : mardi 5 août 2014 17:43 À : user@cassandra.apache.org Objet : Re: vnode and NetworkTopologyStrategy: not playing well together ? This is incorrect. Network Topology w/ Vnodes will be fine, assuming you've got RF= # of racks. For each token, replicas are chosen based on the strategy. Essentially, you could have a wild imbalance in token ownership, but it wouldn't matter because the replicas would be distributed across the rest of the machines. http://www.datastax.com/documentation/cassandra/2.0/cassandra/architec ture/architectureDataDistributeReplication_c.html On Tue, Aug 5, 2014 at 8:19 AM, DE VITO Dominique dominique.dev...@thalesgroup.com wrote: Hi, My understanding is that NetworkTopologyStrategy does NOT play well with vnodes, due to: · Vnode = tokens are (usually) randomly generated (AFAIK) · NetworkTopologyStrategy = required carefully choosen tokens for all nodes in order to not to get a VERY unbalanced ring like in https://issues.apache.org/jira/browse/CASSANDRA-3810 When playing with vnodes, is the recommendation to define one rack for the entire cluster ? Thanks. Regards, Dominique -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade
RE: vnode and NetworkTopologyStrategy: not playing well together ?
The discussion about racks NTS is also mentioned in this recent article : planetcassandra.org/multi-data-center-replication-in-nosql-databases/ The last section may be of interest for you Le 5 août 2014 18:14, DE VITO Dominique dominique.dev...@thalesgroup.com a écrit : Jonathan wrote: Yes, if you have only 1 machine in a rack then your cluster will be imbalanced. You're going to be able to dream up all sorts of weird failure cases when you choose a scenario like RF=2 totally imbalanced network arch. Vnodes attempt to solve the problem of imbalanced rings by choosing so many tokens that it's improbable that the ring will be imbalanced. Storage/load distro = function(1st replica placement, other replica placement) vnode solves the balancing pb for 1st replica placement // so, yes, I agree with you, but for 1st replica placement only But NetworkTopologyStrategy (NTS) influences other (2+) replica placement = as NTS best behavior relies on token distro, and you have no control on tokens with vnodes, the best option I see with **vnode** is to use only one rack with NTS. Dominique -Message d'origine- De : jonathan.had...@gmail.com [mailto:jonathan.had...@gmail.com] De la part de Jonathan Haddad Envoyé : mardi 5 août 2014 18:04 À : user@cassandra.apache.org Objet : Re: vnode and NetworkTopologyStrategy: not playing well together ? Yes, if you have only 1 machine in a rack then your cluster will be imbalanced. You're going to be able to dream up all sorts of weird failure cases when you choose a scenario like RF=2 totally imbalanced network arch. Vnodes attempt to solve the problem of imbalanced rings by choosing so many tokens that it's improbable that the ring will be imbalanced. On Tue, Aug 5, 2014 at 8:57 AM, DE VITO Dominique dominique.dev...@thalesgroup.com wrote: First, thanks for your answer. This is incorrect. Network Topology w/ Vnodes will be fine, assuming you've got RF= # of racks. IMHO, it's not a good enough condition. Let's use an example with RF=2 N1/rack_1 N2/rack_1 N3/rack_1 N4/rack_2 Here, you have RF= # of racks And due to NetworkTopologyStrategy, N4 will store *all* the cluster data, leading to a completely imbalanced cluster. IMHO, it happens when using nodes *or* vnodes. As well-balanced clusters with NetworkTopologyStrategy rely on carefully chosen token distribution/path along the ring *and* as tokens are randomly-generated with vnodes, my guess is that with vnodes and NetworkTopologyStrategy, it's better to define a single (logical) rack // due to carefully chosen tokens vs randomly-generated token clash. I don't see other options left. Do you see other ones ? Regards, Dominique -Message d'origine- De : jonathan.had...@gmail.com [mailto:jonathan.had...@gmail.com] De la part de Jonathan Haddad Envoyé : mardi 5 août 2014 17:43 À : user@cassandra.apache.org Objet : Re: vnode and NetworkTopologyStrategy: not playing well together ? This is incorrect. Network Topology w/ Vnodes will be fine, assuming you've got RF= # of racks. For each token, replicas are chosen based on the strategy. Essentially, you could have a wild imbalance in token ownership, but it wouldn't matter because the replicas would be distributed across the rest of the machines. http://www.datastax.com/documentation/cassandra/2.0/cassandra/architec ture/architectureDataDistributeReplication_c.html On Tue, Aug 5, 2014 at 8:19 AM, DE VITO Dominique dominique.dev...@thalesgroup.com wrote: Hi, My understanding is that NetworkTopologyStrategy does NOT play well with vnodes, due to: · Vnode = tokens are (usually) randomly generated (AFAIK) · NetworkTopologyStrategy = required carefully choosen tokens for all nodes in order to not to get a VERY unbalanced ring like in https://issues.apache.org/jira/browse/CASSANDRA-3810 When playing with vnodes, is the recommendation to define one rack for the entire cluster ? Thanks. Regards, Dominique -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade
