How those burn tests then compare to the fuzz tests? (the new ones) czw., 30 lis 2023, 20:22 użytkownik Benedict <bened...@apache.org> napisał:
> By “could run indefinitely” I don’t mean by default they run forever. > There will be parameters that change how much work is done for a given run, > but just running repeatedly (each time with a different generated seeds) is > the expected usage. Until you run out of compute or patience. > > I agree they are only of value pre-commit to check they haven’t been > broken in some way by changes. > > > > On 30 Nov 2023, at 18:36, Josh McKenzie <jmcken...@apache.org> wrote: > > > > that may be long-running and that could be run indefinitely > > Perfect. That was the distinction I wasn't aware of. Also means having the > burn target as part of regular CI runs is probably a mistake, yes? i.e. if > someone adds a burn tests that runs indefinitely, are there any guardrails > or built-in checks or timeouts to keep it from running right up to job > timeout and then failing? > > On Thu, Nov 30, 2023, at 1:11 PM, Benedict wrote: > > > A burn test is a randomised test targeting broad coverage of a single > system, subsystem or utility, that may be long-running and that could be > run indefinitely, each run providing incrementally more assurance of > quality of the system. > > A long test is a unit test that sometimes takes a long time to run, no > more no less. I’m not sure any of these offer all that much value anymore, > and perhaps we could look to deprecate them. > > On 30 Nov 2023, at 17:20, Josh McKenzie <jmcken...@apache.org> wrote: > > > Strongly agree. I started working on a declarative refactor out of our CI > configuration so circle, ASFCI, and other systems could inherit from it > (for instance, see pre-commit pipeline declaration here > <https://github.com/apache/cassandra/pull/2554/files#diff-a4c4d1d91048841f76d124386858bda9944644cfef8ccb4ab84319cedaf5b3feR71-R89>); > I had to set that down while I finished up implementing an internal CI > system since the code in neither the ASF CI structure nor circle structure > (.sh embedded in .yml /cry) was re-usable in their current form. > > Having a jvm.options and cassandra.yaml file per suite and referencing > them from a declarative job definition > <https://github.com/apache/cassandra/pull/2554/files#diff-a4c4d1d91048841f76d124386858bda9944644cfef8ccb4ab84319cedaf5b3feR237-R267> > would make things a lot easier to wrap our heads around and maintain I > think. > > As for what qualifies as burn vs. long... /shrug couldn't tell you. Would > have to go down the git blame + dev ML + JIRA rabbit hole. :) Maybe someone > else on-list knows. > > On Thu, Nov 30, 2023, at 4:25 AM, Jacek Lewandowski wrote: > > Hi, > > I'm getting a bit lost - what are the exact differences between those > test scenarios? What are the criteria for qualifying a test to be part of a > certain scenario? > > I'm working a little bit with tests and build scripts and the number of > different configurations for which we have a separate target in the build > starts to be problematic, I cannot imagine how problematic it is for a new > contributor. > > It is not urgent, but we should at least have a plan on how to > simplify and unify things. > > I'm in favour of reducing the number of test targets to the minimum - for > different configurations I think we should provide a parameter pointing to > jvm options file and maybe to cassandra.yaml. I know that we currently do > some super hacky things with cassandra yaml for different configs - like > concatenting parts of it. I presume it is not necessary - we can have a > default test config yaml and a directory with overriding yamls; while > building we could have a tool which is able to load the default > configuration, apply the override and save the resulting yaml somewhere in > the build/test/configs for example. That would allows us to easily use > those yamls in IDE as well - currently it is impossible. > > What do you think? > > Thank you and my apologize for bothering about lower priority stuff while > we have a 5.0 release headache... > > Jacek > > > >
