2010YOUY01 commented on PR #27480: URL: https://github.com/apache/datafusion/pull/27480#issuecomment-6949959498
Thank you for working on this! I'd like to spend some time experimenting with it locally before deciding whether we should integrate it into our CI infrastructure. Here are my concerns: I now think addressing mutation testing failures can introduce significant maintenance overhead. Here's a concrete example I just ran into: https://github.com/apache/datafusion/pull/27399#discussion_r4586178706 TL;DR: An AI-assisted review found that removing two lines from the execution code wouldn't cause any existing tests to fail, which is a typical mutation testing finding. However, addressing such findings isn't necessarily straightforward. It could encourage adding tests that overfit specific mutations rather than testing meaningful behavior. Such tests tend to be verbose and hard to maintain, which is already a common weakness of AI-generated code. I think a guideline like 'only try with e2e test, don't overfit mutation test violations with verbose UT' should work, but it can be challenging to enforce it 🤔 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
