> > I think everyone agrees here, but…. these variations are still catching >> failures, and until we have an improvement or replacement we do rely on >> them. I'm not in favour of removing them until we have proof /confidence >> that any replacement is catching the same failures. Especially oa, tries, >> vnodes. (Not tries and offheap is being replaced with "latest", which >> will be valuable simplification.) > > > What kind of proof do you expect? I cannot imagine how we could prove that > because the ability of detecting failures results from the randomness of > those tests. That's why when such a test fail you usually cannot reproduce > that easily. >
Unit tests that fail consistently but only on one configuration, should not be removed/replaced until the replacement also catches the failure. > We could extrapolate that to - why we only have those configurations? why > don't test trie / oa + compression, or CDC, or system memtable? > Because, along the way, people have decided a certain configuration deserves additional testing and it has been done this way in lieu of any other more efficient approach.
