venkateshwaracholan commented on issue #16071: URL: https://github.com/apache/lucene/issues/16071#issuecomment-4697877265
@iverase I took a closer look at the current implementation and ran a couple of local experiments on main. A few observations that might be relevant: I traced the dense primary-sort path from SortedSkipperScorerSupplier through LRUQueryCache.cacheImpl() and confirmed that once cost * 100 >= maxDoc, the cached representation becomes a FixedBitSet(maxDoc). On a 1M-doc segment, a filter matching ~10% of documents and one matching ~100% of documents both consume roughly the same cache memory (~122KB), since both are represented as a segment-wide FixedBitSet. In a small cache-pressure experiment (maxCacheEntries=1), I observed a dense primary-sort filter displacing a previously cached PrefixQuery, which seems consistent with the cache-cost concerns raised earlier in the thread. One thing that surprised me while benchmarking is that caching may not happen at all unless maxRamBytesUsed exceeds roughly 5 * (maxDoc / 8) due to the worst-case RAM guard in LRUQueryCache. That might be relevant to the inconclusive benchmark results you mentioned. I haven't run luceneutil yet, so I don't have enough data to draw conclusions about overall performance impact. I mainly wanted to share the cache behavior I observed while investigating. Happy to run additional experiments if there's a particular scenario that would be useful to measure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
