Hello, We have a search application built around Lucene 8. Motivated by the list of performance enhancements and optimizations in the change notes we upgraded from 8.1 to 8.11.2. We track the performance of different activities within our application and can clearly see an improvement in our facet count queries (about 15% on our p50 to 5% on our p95 execution times).
But when looking at the calls to retrieve matching documents, the parts of our application where we allow the retrieval of many top hits (up to 10k) has really suffered. For instance when the number of documents matching our query exceeds 5k I see the performance of the doc matching degrade 50% on the p50 and 35% on the p95. The facet collection still shows improvements of 1-5%. The counter point is that when a query will match 500 documents then we see across the board improvements in both document matching and facet count generation. Has anyone else seen this sort of performance change in applications that allow such a high number of top docs to be returned from IndexSearcher.search()? Is there some tuning or flag setting that I should consider when allowing such a large number of top documents to be returned? Thank you, Marc Davenport
