edmondop commented on PR #27480:
URL: https://github.com/apache/datafusion/pull/27480#issuecomment-6951698489

   > Thank you for working on this! I'd like to spend some time experimenting 
with it locally before deciding whether we should integrate it into our CI 
infrastructure. Here are my concerns:
   > 
   > I now think addressing mutation testing failures can introduce significant 
maintenance overhead. Here's a concrete example I just ran into: [#27399 
(comment)](https://github.com/apache/datafusion/pull/27399#discussion_r4586178706)
   > 
   > TL;DR: An AI-assisted review found that removing two lines from the 
execution code wouldn't cause any existing tests to fail, which is a typical 
mutation testing finding.
   > 
   > However, addressing such findings isn't necessarily straightforward. It 
could encourage adding tests that overfit specific mutations rather than 
testing meaningful behavior. Such tests tend to be verbose and hard to 
maintain, which is already a common weakness of AI-generated code.
   > 
   > I think a guideline like 'only try with e2e test, don't overfit mutation 
test violations with verbose UT' should work, but it can be challenging to 
enforce it 🤔
   
   I agree about the maintenance overhead, totally, but I also think 
somehow.... that is the point.
   
   I have been thinking about it, and I am not sure there is a way to contain 
the defect rate in presence of AI generated code that doesn't involve 
increasing overhead and friction. This is a non-blocking check for now, so 
merging it would only help contributors write better tests, if they want. If 
they don't, they can ignore it :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to