shbhar commented on issue #16937: URL: https://github.com/apache/lucene/issues/16937#issuecomment-6531366738
@CH-Abhinav, your skepticism is both warranted and welcome. Extraordinary claims must be supported by extraordinary evidence after all! Having said that, this exercise is making much more narrow claims than you might be thinking of. I should probably change the wording from "bug" to "finding" if it can help avoid confusion. Starting with what it is NOT claiming: 1. Not claiming that all these findings are correct and there are 0 hallucinations Hence the manual confirmation and understanding. Though in about 20 findings I have spot-checked they all at least repro. Newer models are far better at not hallucinating, especially when using multiple "skeptical" agents to confirm from differnet angles (like agents trying to prove a finding wrong) and being forced to prove the "bug" is real with a minimal failing unit test 2. Not claiming all the findings are code logic bugs Even if the finding repros as stated, it might just be expected behavior or a simple doc improvement (like Uwe's mention above). Many findings just end up surfacing tradeoffs or inconsistencies that might not have been made explicitly but accidentally and an explicit consideration may be warranted and found to be enough. 3. Not claiming even the confirmed code logic bugs need to be fixed Even if findings are theoretically real bugs they might be so rare to be almost impossible to cause anyone an issue (not every bug results in an error) and therefore might not need a fix. Mature software thats complex/large enough probably can never be fully bug free and may not even have to be for its intended use. One example of such a contrived bug might be: https://shbhar.github.io/lucene-ai-bug-hunt/bugs/LFV-196.html. The AI itself classifies nearly all of these findings as low/medium priority (for new findings: 4 high, 126 medium and 637 low as of this writing) As with all things AI, there is a mix of both signal and noise. I'm personally confident that there is enough signal:noise ratio to make an effort to do *something* for these findings and this exercise is my stab at it! My reason to have some confidence is also because I'm seeing success using the same technique of "AI assisted bug hunting" yielding actually useful bugs in other places too, for example in Postgres (https://news.ycombinator.com/item?id=50119991) The issues linked here or the discussion in the mailing list show initial evidence of this signal Im talking about (still early though). The goal is not to overwhelm the community and we need some meaningful process to make these findings actually useful. Other perspectives or suggestions always welcome! The details, presentation or techniques use to create or compile the findings are not final either, happy to change them as needed based on discussion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
