Hi all, I work in Amazon advertising where we use Lucene for a lot of our ad candidate retrieval services. We also develop custom codecs and other customizations for cases where the stock Lucene options aren't enough. While developing one of them, I asked an AI agent to find bugs in my POC implementation, and it ended up finding an unreported bug in Lucene itself.
I kept going with this approach and it kept finding more unreported bugs, so I've gone deeper into AI-assisted bug finding with formal verification and other techniques. As of today the agents have found 592 bugs in Lucene (and counting) that appear to be unreported in GitHub or the mailing list. It also found (or already knew from its training data) nearly 200 bugs that have already been reported, which I'm not counting here. To avoid noise in these findings, the approach has each potential bug independently and adversarially verified by a second model (Claude vs GPT) with a minimal unit test repro and making sure the behavior is not intended from past discussions in mailing lists, tests, GitHub discussion & commit/PR history. Many of them seem to be minor or low impact (392 low, 196 medium, 4 high classified by AI), but a few look important. For example: - After a flush where the only changes are doc-values updates, openIfChanged returns null and isCurrent() stays true so the update can be missed: https://github.com/apache/lucene/issues/16900 - HNSW merge-time graph reuse can leave the entry node with no neighbours, which drops kNN recall to zero in 10.4.0 and later: https://shbhar.github.io/lucene-ai-bug-hunt/bugs/LFV-005.html I've posted the full running list with some categorization here: https://shbhar.github.io/lucene-ai-bug-hunt/. I've excluded a few that might be security sensitive according to AI, will send them privately to security@ soon. I'm working on understanding, triaging and verifying them myself, then filing issues on GitHub in my own words in line with Lucene's AI policy (so far I've filed only one). The agents help by preparing minimal repro tests, diagrams and short explainer videos (not on site yet) for surrounding context, but understanding and verifying each bug properly still takes a lot of time, and the community's help would make a big difference. Creating this thread to start a discussion on how to validate and make these findings actually useful, in particular: - Could folks help in spot-checking some bugs for areas they might be familiar with? - Is it beneficial to track this work in an umbrella github issue? Thanks, Shitij
