Nagato-Yuzuru commented on PR #27326: URL: https://github.com/apache/datafusion/pull/27326#issuecomment-7047631823
I did not apply the second suggestion from #27325 here. I tried it on a local branch (new run seed, one sampled combination per case), and it found a problem in the test oracle. The complex check asserts !expected when ordering_satisfy rejects the conclusive prefix. This is a completeness check, but it uses one sample as the ground truth, and a sample can be sorted by chance. Example: orderings [b NULLS FIRST, c] and [a], requirement [a, b NULLS LAST, c]. The rejection is correct, but in each a group, b is all NULL or all non-NULL, so the sample is sorted. With 50,000 cases on main, this failed in 2 of 2 runs. The fixed seed hides this, because none of the 300 tables has this case. Any change to new tables per run (sampled or exhaustive) makes main flaky. Other checks with the same pattern can have this problem too (not verified yet). With the #24276 fix reverted, the sampled suite found the #24208 bug in 19 of 25 runs with 10,000 cases, and in 10 of 10 runs with 50,000 cases. Suggested order: 1. Merge this PR as a move only. 2. Fix the oracle in a follow-up PR. 3. Switch to new seeds with sampled cases. Is this OK? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
