I wouldn't recommend Shenandoah or ZGC, period. They're not designed for the kind of workload you'll typically see running a database (high throughput of objects that don't tenure) and both will fall over in interesting ways under high allocation rate. GenShen is intended to combine the generational goodness of G1 with the concurrent collection of Shenandoah, and will likely perform better than G1 in terms of pause time and better than Shenandoah for allocation rate and heap utilization. GenShen, however, is experimental. Right now I would say G1 is the best collector generally available.
In terms of providing data (beyond anecdotes), do we even agree on what the baseline load test looks like? Are we going off of something that's in dtest, or do we have a defined benchmarking suite somewhere? Cheers, Derek On Thu, Nov 17, 2022 at 1:47 PM J. D. Jordan <jeremiah.jor...@gmail.com> wrote: > > -1 on providing a bunch of choices and forcing users to pick one. We should > have a default and it should be “good enough” for most people. The people who > want to dig in and try other gc settings can still do it, and we could > provide them some profiles to start from, but there needs to be a default. > We need to be asking new operators less questions on install, not more. > > Re:experience with Shenandoah under high load, I have in the past seen the > exact same thing for both Shenandoah and ZGC. Both of them have issues at > high loads while performing great at moderate loads. I have not seen G1 ever > have such issues. So I would not be fine with a switch to Shenandoah or ZGC > as the default without extensive testing on current JVM versions that have > hopefully improved the behavior under load. > > > On Nov 17, 2022, at 9:39 AM, Joseph Lynch <joe.e.ly...@gmail.com> wrote: > > It seems like this is a choice most users might not know how to make? > > > > On Thu, Nov 17, 2022 at 7:06 AM Josh McKenzie <jmcken...@apache.org> wrote: > >> > >> Have we ever discussed including multiple profiles that are simple to swap > >> between and documented for their tested / intended use cases? > >> > >> Then the burden of having a “sane” default for the wild variance of > >> workloads people use it for would be somewhat mitigated. Sure, there’s > >> always going to be folks that run the default and never think to change it > >> but the UX could be as simple as a one line config change to swap between > >> GC profiles and we could add and deprecate / remove over time. > >> > >> Concretely, having config files such as: > >> > >> jvm11-CMS-write.options > >> jvm11-CMS-mixed.options > >> jvm11-CMS-read.options > >> jvm11-G1.options > >> jvm11-ZGC.options > >> jvm11-Shen.options > >> > >> > >> Arguably we could take it a step further and not actually allow a C* node > >> to startup without pointing to one of the config files from your primary > >> config, and provide a clean mechanism to integrate that selection on > >> headless installs. > >> > >> Notably, this could be a terrible idea. But it does seem like we keep > >> butting up against the complexity and mixed pressures of having the One > >> True Way to GC via the default config and the lift to change that. > >> > >> On Wed, Nov 16, 2022, at 9:49 PM, Derek Chen-Becker wrote: > >> > >> I'm fine with not including G1 in 4.1, but would we consider inclusion > >> for 4.1.X down the road once validation has been done? > >> > >> Derek > >> > >> > >> On Wed, Nov 16, 2022 at 4:39 PM David Capwell <dcapw...@apple.com> wrote: > >>> Getting poked in Slack to be more explicit in this thread… > >>> Switching to G1 on trunk, +1 > >>> Switching to G1 on 4.1, -1. 4.1 is about to be released and this isn’t a > >>> bug fix but a perf improvement ticket and as such should go through > >>> validation that the perf improvements are seen, there is not enough time > >>> left for that added performance work burden so strongly feel it should be > >>> pushed to 4.2/5.0 where it has plenty of time to be validated against. > >>> The ticket even asks to avoid validating the claims; saying 'Hoping we > >>> can skip due diligence on this ticket because the data is "in the past” > >>> already”'. Others have attempted both shenandoah and ZGC and found mixed > >>> results, so nothing leads me to believe that won’t be true here either. > >>>> On Nov 16, 2022, at 9:15 AM, J. D. Jordan <jeremiah.jor...@gmail.com> > >>>> wrote: > >>>> Heap - > >>>> +1 for G1 in trunk > >>>> +0 for G1 in 4.1 - I think it’s worthwhile and fairly well tested but I > >>>> understand pushback against changing this so late in the game. > >>>> Memtable - > >>>> -1 for off heap in 4.1. I think this needs more testing and isn’t > >>>> something to change at the last minute. > >>>> +1 for running performance/fuzz tests against the alternate memtable > >>>> choices in trunk and switching if they don’t show regressions. > >>>>> On Nov 16, 2022, at 10:48 AM, Josh McKenzie <jmcken...@apache.org> > >>>>> wrote: > >>>>> > >>>>> To clarify: -0 here on G1 as default for 4.1 as well; I'd like us to > >>>>> prioritize digging into G1's behavior on small heaps vs. CMS w/our > >>>>> default tuning sooner rather than later. With that info I'd likely be a > >>>>> strong +1 on the shift. > >>>>> -1 on switching to offheap_objects for 4.1 RC; again, think this is > >>>>> just a small step away from being a +1 w/some more rigor around seeing > >>>>> the current state of the technology's intersections. > >>>>> On Wed, Nov 16, 2022, at 7:47 AM, Aleksey Yeshchenko wrote: > >>>>>> All right. I’ll clarify then. > >>>>>> -0 on switching the default to G1 *this late* just before RC1. > >>>>>> -1 on switching the default offheap_objects *for 4.1 RC1*, but all for > >>>>>> it in principle, for 4.2, after we run some more test and resolve the > >>>>>> concerns raised by Jeff. > >>>>>> Let’s please try to avoid this kind of super late defaults switch > >>>>>> going forward? > >>>>>> — > >>>>>> AY > >>>>>>> On 16 Nov 2022, at 03:27, Derek Chen-Becker <de...@chen-becker.org> > >>>>>>> wrote: > >>>>>>> For the record, I'm +100 on G1. Take it with whatever sized grain of > >>>>>>> salt you think appropriate for a relative newcomer to the list, but > >>>>>>> I've spent my last 7-8 years dealing with the intersection of > >>>>>>> high-throughput, low latency systems and their interaction with GC and > >>>>>>> in my personal experience G1 outperforms CMS in all cases and with > >>>>>>> significantly less work (zero work, in many cases). The only things > >>>>>>> I've seen perform better *with a similar heap footprint* are GenShen > >>>>>>> (currently experimental) and Rust (beyond the scope of this topic). > >>>>>>> Derek > >>>>>>> On Tue, Nov 15, 2022 at 4:51 PM Jon Haddad > >>>>>>> <rustyrazorbl...@apache.org> wrote: > >>>>>>>> I'm curious what it would take for folks to be OK with merging this > >>>>>>>> into 4.1? How much additional time would you want to feel > >>>>>>>> comfortable? > >>>>>>>> I should probably have been a little more vigorous in my +1 of > >>>>>>>> Mick's PR. For a little background - I worked on several hundred > >>>>>>>> clusters while at TLP, mostly dealing with stability and performance > >>>>>>>> issues. A lot of them stemmed partially or wholly from the GC > >>>>>>>> settings we ship in the project. Par New with CMS and small new gen > >>>>>>>> results in a lot of premature promotion leading to high pause times > >>>>>>>> into the hundreds of ms which pushes p99 latency through the roof. > >>>>>>>> I'm a big +1 in favor of G1 because it's not just better for most > >>>>>>>> people but it's better for _every_ new Cassandra user. The first > >>>>>>>> experience that people have with the project is important, and our > >>>>>>>> current GC settings are quite bad - so bad they lead to problems > >>>>>>>> with stability in production. The G1 settings are mostly hands off, > >>>>>>>> result in shorter pause times and are a big improvement over the > >>>>>>>> status quo. > >>>>>>>> Most folks don't do GC tuning, they use what we supply, and what we > >>>>>>>> currently supply leads to a poor initial experience with the > >>>>>>>> database. I think we owe the community our best effort even if it > >>>>>>>> means pushing the release back little bit. > >>>>>>>> Just for some additional context, we're (Netflix) running 25K nodes > >>>>>>>> on G1 across a variety of hardware in AWS with wildly varying > >>>>>>>> workloads, and I haven't seen G1 be the root cause of a problem even > >>>>>>>> once. The settings that Mick is proposing are almost identical to > >>>>>>>> what we use (we use half of heap up to 30GB). > >>>>>>>> I'd really appreciate it if we took a second to consider the > >>>>>>>> community effect of another release that ships settings that cause > >>>>>>>> significant pain for our users. > >>>>>>>> Jon > >>>>>>>> On 2022/11/10 21:49:36 Mick Semb Wever wrote: > >>>>>>>>>> In case of GC, reasonably extensive performance testing should be > >>>>>>>>>> the > >>>>>>>>>> expectations. Potentially revisiting some of the G1 params for the > >>>>>>>>>> 4.1 > >>>>>>>>>> reality - quite a lot has changed since those optional defaults > >>>>>>>>>> where > >>>>>>>>>> picked. > >>>>>>>>> I've put our battle-tested g1 opts (from consultants at TLP and > >>>>>>>>> DataStax) > >>>>>>>>> in the patch for CASSANDRA-18027 > >>>>>>>>> In reality it is really not much of a change, g1 does make it > >>>>>>>>> simple. > >>>>>>>>> Picking the correct ParallelGCThreads and ConcGCThreads and the > >>>>>>>>> floor to > >>>>>>>>> the new heap (XX:NewSize) is still required, though we could do a > >>>>>>>>> much > >>>>>>>>> better job of dynamic defaults to them. > >>>>>>>>> Alex Dejanovski's blog is a starting point: > >>>>>>>>> https://thelastpickle.com/blog/2020/06/29/cassandra_4-0_garbage_collectors_performance_benchmarks.html > >>>>>>>>> where this gc opt set was used (though it doesn't prove why those > >>>>>>>>> options > >>>>>>>>> are chosen) > >>>>>>>>> The bar for objection to sneaking these into 4.1 was intended to be > >>>>>>>>> low, > >>>>>>>>> and I stand by those that raise concerns. > >>>>>>> -- > >>>>>>> +---------------------------------------------------------------+ > >>>>>>> | Derek Chen-Becker | > >>>>>>> | GPG Key available at https://keybase.io/dchenbecker and | > >>>>>>> | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org | > >>>>>>> | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7 7F42 AFC5 AFEE 96E4 6ACC | > >>>>>>> +---------------------------------------------------------------+ > >> > >> > >> -- > >> +---------------------------------------------------------------+ > >> | Derek Chen-Becker | > >> | GPG Key available at https://keybase.io/dchenbecker and | > >> | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org | > >> | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7 7F42 AFC5 AFEE 96E4 6ACC | > >> +---------------------------------------------------------------+ -- +---------------------------------------------------------------+ | Derek Chen-Becker | | GPG Key available at https://keybase.io/dchenbecker and | | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org | | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7 7F42 AFC5 AFEE 96E4 6ACC | +---------------------------------------------------------------+
