Alex’s suggestion was that we meta randomise, ie we randomise the config parameters to gain better rather than lesser coverage overall. This means we cover these specific configs and more - just not necessarily on any single commit.
I strongly endorse this approach over the status quo. > On 8 Dec 2023, at 13:26, Mick Semb Wever <m...@apache.org> wrote: > > > > > >> >>> I think everyone agrees here, but…. these variations are still catching >>> failures, and until we have an improvement or replacement we do rely on >>> them. I'm not in favour of removing them until we have proof /confidence >>> that any replacement is catching the same failures. Especially oa, tries, >>> vnodes. (Not tries and offheap is being replaced with "latest", which will >>> be valuable simplification.) >> >> What kind of proof do you expect? I cannot imagine how we could prove that >> because the ability of detecting failures results from the randomness of >> those tests. That's why when such a test fail you usually cannot reproduce >> that easily. > > > Unit tests that fail consistently but only on one configuration, should not > be removed/replaced until the replacement also catches the failure. > > >> We could extrapolate that to - why we only have those configurations? why >> don't test trie / oa + compression, or CDC, or system memtable? > > > Because, along the way, people have decided a certain configuration deserves > additional testing and it has been done this way in lieu of any other more > efficient approach. >
