Although I like in-jvm DTests for many scenarios, I can see that they do not test the production code as it is. They use a separate implementation of instance initialization and thus they test the test server rather than the real node. Other than that, it can be problematic to test upgrades when the starting version must run with a different Java version than the end release. One more thing I've been observing sometimes is high consumption of metaspace, which does not seem to be cleaned after individual test cases. Given each started instance uses a dedicated class loader there is some amount of trash left and when there are a couple of multi-node test cases in a single test class, it sometimes happens that the test fail with out of memory in metaspace error.
Thanks, Jacek On Mon, Mar 28, 2022 at 10:06 PM David Capwell <dcapw...@apple.com> wrote: > I am back and the work for trunk to support vnode is at the last stage of > review; I had not planned to backport the changes to other branches (aka, > older branches would only support single token), so if someone would like > to pick up this work it is rather LHF after 17332 goes in (see trunk patch GH > PR: trunk <https://github.com/apache/cassandra/pull/1432>). > > I am in favor of deprecating python dtests, and agree we should figure out > what the gaps are (once vnode support is merged) so we can either shrink > them or special case to unfreeze (such as startup changes being allowed). > > On Mar 14, 2022, at 6:13 AM, Josh McKenzie <jmcken...@apache.org> wrote: > > vnode support for in-jvm dtests is in flight and fairly straightforward: > > https://issues.apache.org/jira/browse/CASSANDRA-17332 > > David's OOO right now but I suspect we can get this in in April some time. > > On Mon, Mar 14, 2022, at 8:36 AM, bened...@apache.org wrote: > > This is the limitation I mentioned. I think this is solely a question of > supplying an initial config that uses vnodes, i.e. that specifies multiple > tokens for each node. It is not really a limitation – I believe a dtest > could be written today using vnodes, by overriding the config’s tokens. It > does look like the token handling has been refactored since the initial > implementation to make this a little uglier than should be necessary. > > We should make this trivial, anyway, and perhaps offer a way to run all of > the dtests with vnodes (and suitably annotating those that cannot be run > with vnodes). This should be quite easy. > > > > *From: *Andrés de la Peña <adelap...@apache.org> > *Date: *Monday, 14 March 2022 at 12:28 > *To: *dev@cassandra.apache.org <dev@cassandra.apache.org> > *Subject: *Re: [DISCUSS] Should we deprecate / freeze python dtests > Last time I checked there wasn't support for vnodes on in-jvm dtests, > which seems an important limitation. > > On Mon, 14 Mar 2022 at 12:24, bened...@apache.org <bened...@apache.org> > wrote: > > I am strongly in favour of deprecating python dtests in all cases where > they are currently superseded by in-jvm dtests. They are environmentally > more challenging to work with, causing many problems on local and remote > machines. They are harder to debug, slower, flakier, and mostly less > sophisticated. > > > all focus on getting the in-jvm framework robust enough to cover > edge-cases > > Would be great to collect gaps. I think it’s just vnodes, which is by no > means a fundamental limitation? There may also be some stuff to do > startup/shutdown and environmental scripts, that may be a niche we retain > something like python dtests for. > > > people aren’t familiar > > I would be interested to hear from these folk to understand their concerns > or problems using in-jvm dtests, if there is a cohort holding off for this > reason > > > This is going to require documentation work from some of the original > authors > > I think a collection of template-like tests we can point people to would > be a cheap initial effort. Cutting and pasting an existing test with the > required functionality, then editing to suit, should get most people off to > a quick start who aren’t familiar. > > > Labor and process around revving new releases of the in-jvm dtest API > > I think we need to revisit how we do this, as it is currently broken. We > should consider either using ASF snapshots until we cut new releases of C* > itself, or else using git subprojects. This will also become a problem for > Accord’s integration over time, and perhaps other subprojects in future, so > it is worth better solving this. > > I think this has been made worse than necessary by moving too many > implementation details to the shared API project – some should be retained > within the C* tree, with the API primarily serving as the shared API itself > to ensure cross-version compatibility. However, this is far from a complete > explanation of (or solution to) the problem. > > > > > *From: *Josh McKenzie <jmcken...@apache.org> > *Date: *Monday, 14 March 2022 at 12:11 > *To: *dev@cassandra.apache.org <dev@cassandra.apache.org> > *Subject: *[DISCUSS] Should we deprecate / freeze python dtests > I've been wrestling with the python dtests recently and that led to some > discussions with other contributors about whether we as a project should be > writing new tests in the python dtest framework or the in-jvm framework. > This discussion has come up tangentially on some other topics, including > the lack of documentation / expertise on the in-jvm framework > dis-incentivizing some folks from authoring new tests there vs. the > difficulty debugging and maintaining timer-based, sleep-based > non-deterministic python dtests, etc. > > I don't know of a place where we've formally discussed this and made a > project-wide call on where we expect new distributed tests to be written; > if I've missed an email about this someone please link on the thread here > (and stop reading! ;)) > > At this time we don't specify a preference for where you write new > multi-node distributed tests on our "development/testing" portion of the > site and documentation: > https://cassandra.apache.org/_/development/testing.html > > The primary tradeoffs as I understand them for moving from python-based > multi-node testing to jdk-based are: > Pros: > > 1. Better debugging functionality (breakpoints, IDE integration, etc) > 2. Integration with simulator > 3. More deterministic runtime (anecdotally; python dtests _should_ be > deterministic but in practice they prove to be very prone to environmental > disruption) > 4. Test time visibility to internals of cassandra > > Cons: > > 1. The framework is not as mature as the python dtest framework (some > functionality missing) > 2. Labor and process around revving new releases of the in-jvm dtest > API > 3. People aren't familiar with it yet and there's a learning curve > > > So my bid here: I personally think we as a project should freeze writing > new tests in the python dtest framework and all focus on getting the in-jvm > framework robust enough to cover edge-cases that might still be causing new > tests to be written in the python framework. This is going to require > documentation work from some of the original authors of the in-jvm > framework as well as folks currently familiar with it and effort from those > of us not yet intimately familiar with the API to get to know it, however I > believe the long-term benefits to the project will be well worth it. > > We could institute a pre-commit check that warns on a commit increasing > our raw count of python dtests to help provide process-based visibility to > this change in direction for the project's testing. > > So: what do we think? > > >
