Yes, I'm against it. We should be using the default value that benefits the most people, rather than an arbitrary compromise.
Most clusters don't shrink, they stay the same size or grow. I'd say 90% or more fall in this category. Let's do the right thing by default and include good comments that help people make the right decision if they think they'll be outside the usual case. On Thu, Jan 30, 2020, 8:07 PM Joseph Lynch <joe.e.ly...@gmail.com> wrote: > Any objections to the compromise of 16 as proposed in Chris's original > patch? > > -Joey > > On Thu, Jan 30, 2020, 3:47 PM Anthony Grasso <anthony.gra...@gmail.com> > wrote: > > > I think lowering the number of tokens is a great idea! Similar to Jon, > when > > I have reduced the number of tokens for clients it has been improvement > in > > repair performance. > > > > I am concerned that the proposed default value for num_tokens is too low. > > If you set up a cluster using the proposed defaults, you will get a > > balanced cluster. However, if you decommission nodes you will start to > see > > large imbalances especially for small clusters (< 20 nodes). This is > > because the allocate_tokens_for_local_replication_factor setting is only > > applied during the bootstrap process. > > > > I have recommended very low values for num_tokens to clients. This was > > because it was very unlikely that they would reduce their cluster size > and > > I warned them of the caveats with using a small value for num_tokens. > > > > The proposed num_token default value is fine for devs and operators that > > know what they are doing. However, the general Cassandra community will > be > > unaware of the potential issue with such a low value. We should consider > > setting num_tokens to 16 - 32 as the default. This will at least help > > reduce the severity of the imbalance when decommissioning a node whilst > > still providing the benefits of having a low number of tokens. In > addition, > > we can add a comment to num_tokens that clusters over 100 nodes (per > > datacenter) should consider reducing it down to 4. > > > > Cheers, > > Anthony > > > > On Fri, 31 Jan 2020 at 01:58, Jon Haddad <j...@jonhaddad.com> wrote: > > > > > Larger clusters is where high token counts do the most damage. That's > why > > > it's such a problem. You start out with a small cluster using 256, as > you > > > grow into the hundreds it becomes more and more unstable. > > > > > > > > > On Thu, Jan 30, 2020, 8:19 AM onmstester onmstester > > > <onmstes...@zoho.com.invalid> wrote: > > > > > > > Shouldn't we consider the cluster size to configure num_tokens? > > > > > > > > For example is it OK to use num_tokens=4 for a cluster of more than > 100 > > > of > > > > nodes? > > > > > > > > > > > > > > > > Another question that is not so much relevant to this : > > > > > > > > When we use the token assignment algorithm (the new/non-random one) > > for a > > > > specific keyspace, why should we use initial token for all the seeds, > > > isn't > > > > one seed enough and then just set the keyspace for all other nodes? > > > > > > > > > > > > > > > > Also i do not understand why should we consider rack topology and > > number > > > > of racks for configuration of num_tokens? > > > > > > > > > > > > > > > > Sent using https://www.zoho.com/mail/ > > > > > > > > > > > > > > > > > > > > ---- On Thu, 30 Jan 2020 04:33:57 +0330 Jeremy Hanna < > > > > jeremy.hanna1...@gmail.com> wrote ---- > > > > > > > > > > > > The new default wouldn't be retroactively set for 3.x, but the same > > > > principles apply. The new algorithm is in 3.x as well as the > > > > simplification of the configuration. So no reason not to use the > same > > > > configuration on 3.x. > > > > > > > > > On Jan 30, 2020, at 4:34 AM, Chen-Becker, Derek <mailto: > > > > dchen...@amazon.com.INVALID> wrote: > > > > > > > > > > Does the same guidance apply to 3.x clusters? I read through the > JIRA > > > > ticket linked below, along with tickets that it links to, but it's > not > > > > clear that the new allocation algorithm is available in 3.x or if > there > > > are > > > > other reasons that this would be problematic. > > > > > > > > > > Thanks, > > > > > > > > > > Derek > > > > > > > > > > On 1/29/20, 9:54 AM, "Jon Haddad" <mailto:j...@jonhaddad.com> > wrote: > > > > > > > > > > Ive put a lot of my previous clients on 4 tokens, all of which > > have > > > > > resulted in a major improvement. > > > > > > > > > > I wouldn't use any more than 4 except under some pretty unusual > > > > > circumstances. > > > > > > > > > > Jon > > > > > > > > > > On Wed, Jan 29, 2020, 11:18 AM Ben Bromhead <mailto: > > > > b...@instaclustr.com> wrote: > > > > > > > > > >> +1 to reducing the number of tokens as low as possible for > > > availability > > > > >> issues. 4 lgtm > > > > >> > > > > >> On Wed, Jan 29, 2020 at 1:14 AM Dinesh Joshi <mailto: > > > djo...@apache.org> > > > > wrote: > > > > >> > > > > >>> Thanks for restarting this discussion Jeremy. I personally think > 4 > > is > > > > a > > > > >>> good number as a default. I think whatever we pick, we should > have > > > > enough > > > > >>> documentation for operators to make sense of the new defaults in > > 4.0. > > > > >>> > > > > >>> Dinesh > > > > >>> > > > > >>>> On Jan 28, 2020, at 9:25 PM, Jeremy Hanna <mailto: > > > > jeremy.hanna1...@gmail.com> > > > > >>> wrote: > > > > >>>> > > > > >>>> I wanted to start a discussion about the default for num_tokens > > that > > > > >>> we'd like for people starting in Cassandra 4.0. This is for > ticket > > > > >>> CASSANDRA-13701 < > > > https://issues.apache.org/jira/browse/CASSANDRA-13701> > > > > > > > > >>> (which has been duplicated a number of times, most recently by > me). > > > > >>>> > > > > >>>> TLDR, based on availability concerns, skew concerns, operational > > > > >>> concerns, and based on the fact that the new allocation algorithm > > can > > > > be > > > > >>> configured fairly simply now, this is a proposal to go with 4 as > > the > > > > new > > > > >>> default and the allocate_tokens_for_local_replication_factor set > to > > > 3. > > > > >>> That gives a good experience out of the box for people and is the > > > most > > > > >>> conservative. It does assume that racks and DCs have been > > configured > > > > >>> correctly. We would, of course, go into some detail in the > > NEWS.txt. > > > > >>>> > > > > >>>> Joey Lynch and Josh Snyder did an extensive analysis of > > availability > > > > >>> concerns with high num_tokens/virtual nodes in their paper < > > > > >>> > > > > >> > > > > > > > > > > http://mail-archives.apache.org/mod_mbox/cassandra-dev/201804.mbox/%3CCALShVHcz5PixXFO_4bZZZNnKcrpph-=5QmCyb0M=w-mhdyl...@mail.gmail.com%3E > > > > >>> . > > > > >>> This worsens as clusters grow larger. I won't quote the paper > here > > > > but > > > > >> in > > > > >>> order to have a conservative default and with the accompanying > new > > > > >>> allocation algorithm, I think it makes sense as a default. > > > > >>>> > > > > >>>> The difficulties have always been that virtual nodes have been > > > > >>> beneficial for operations but that 256 is too high for the > purposes > > > of > > > > >>> repair and as Joey and Josh cover, for availability. Going lower > > > with > > > > >> the > > > > >>> original allocation algorithm has produced skew in allocation in > > its > > > > >> naive > > > > >>> distribution. Enter CASSANDRA-7032 < > > > > >>> https://issues.apache.org/jira/browse/CASSANDRA-7032> and the > new > > > > token > > > > >>> allocation algorithm. CASSANDRA-15260 < > > > > >>> https://issues.apache.org/jira/browse/CASSANDRA-15260> makes the > > new > > > > >>> algorithm operationally simpler. > > > > >>>> > > > > >>>> One other item of note - since Joey and Josh's analysis, there > > have > > > > >> been > > > > >>> improvements in streaming and other considerations that can > reduce > > > the > > > > >>> probability of more than one node representing some token range > > being > > > > >>> unavailable, but it would still be good to be conservative. > > > > >>>> > > > > >>>> Please chime in with any concerns with having num_tokens=4 and > > > > >>> allocate_tokens_for_local_replication_factor=3 and the > accompanying > > > > >>> rationale so we can improve the experience for all users. > > > > >>>> > > > > >>>> Other resources: > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > https://docs.datastax.com/en/dse/6.7/dse-admin/datastax_enterprise/config/configVnodes.html > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > https://www.datastax.com/blog/2016/01/new-token-allocation-algorithm-cassandra-30 > > > > >>>> > > > > >>> > > > > >>> > > > > >>> > > --------------------------------------------------------------------- > > > > >>> To unsubscribe, e-mail: mailto: > > dev-unsubscr...@cassandra.apache.org > > > > >>> For additional commands, e-mail: mailto: > > > dev-h...@cassandra.apache.org > > > > >>> > > > > >>> > > > > >> > > > > >> -- > > > > >> > > > > >> Ben Bromhead > > > > >> > > > > >> Instaclustr | www.instaclustr.com | @instaclustr > > > > >> <http://twitter.com/instaclustr> | (650) 284 9692 > > > > >> > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: mailto: > dev-unsubscr...@cassandra.apache.org > > > > > For additional commands, e-mail: mailto: > > dev-h...@cassandra.apache.org > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: mailto:dev-unsubscr...@cassandra.apache.org > > > > For additional commands, e-mail: mailto: > dev-h...@cassandra.apache.org > > > > > >
