edmondop commented on issue #26865: URL: https://github.com/apache/datafusion/issues/26865#issuecomment-6482177509
I think it's a fair concern and the answer should probably depend on our assessment of the situation. Given what you said about the project risks, I think a systematic solution would be more effective. A "one-off" effort could improve the situation in the short term, but if we believe there are forces that lead to this situation that aren't changing, it won't be sufficient ## Arguments in favor of enabling mutation testing: how do we tests our tests? If we believe these bugs are the consequence of a decrease in the effectiveness of testing, because the system has become complex and/or because AI generates a part of the code, in this case I think it is a good idea to enable mutation testing. It will fail like so (taken from #14590) ``` MISSED datafusion/physical-optimizer/src/join_selection.rs:227:9: replace || with && in try_collect_left in 77.1s build + 0.4s test ``` this is telling us that the physical optimizer tests pass when changing a comparison function from OR to AND, and therefore we should improve these tests. We can run this on a schedule, and mutation testing will help us have better tests, and identify and fix bugs before they are observed at runtime. ## Arguments against enabling mutation testing: some defects are inevitable, the defect ratio is not changing, this is just a "part of the game" If we believe these bugs are the inevitable defects that gets generated through software development, we don't think it is becoming harder to have write comprehensive and robust tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
