Hello everyone, I did not get a response on this, but wanted to give an update and ask a few more questions. Our profiling shows that 80% of the time Lucene spends on reading FSTs for these queries, so the regression seems to be related to the change in Lucene 8.6 (LUCENE-9257) where the FSTs were moved off-heap. Our understanding is that with this change, reading FSTs became less efficient with NIOFSDirectory because Lucene spends more time on bringing data on heap which also significantly increases the number of system calls / kernel CPU.
Currently we are trying to avoid switching to MMAP because there is another process running on the same host and extensively utilizes the FS cache. We did try to use FileSwitchDirectory and MMAP only a minimal amount of files - Term Dictionary and Term Index. That helped, but only in some use cases. Is there anything else we are missing, maybe some other Lucene data structures are critical to MMAP with FSTs off-heap? Is there anything else we could try? There is probably an option to bring FSTs back on heap, but we are trying to avoid it since there is no configuration for this so it requires a lot of code changes. Thank you, Alex On Tue, Aug 9, 2022 at 6:34 AM Alexander Lukyanchikov < alexanderlukyanchi...@gmail.com> wrote: > Hello everyone, > We have a use-case which shows about 10 times higher latency for boolean > queries after migrating to Lucene 9.2. Each query contains 3 filter clauses > and up to a thousand single-term should clauses. They usually return less > than 5 documents with a single stored field and used to execute in about > 500 ms, but now take more than 5 seconds. Throughput remains roughly the > same, about 60 qps. > > A compounding factor is that our disk utilization is quite high and > usually in the range of 70% to 100%. That seems like an obvious issue, but > it did not affect these queries too much before. After the upgrade, we can > also see about twice as high kernel CPU utilization. We are currently using > NIOFSDirectory. > > Please let me know if these symptoms mean something to you, it would be > great to know if we are doing something wrong and if there is a way to fix > the queries without upgrading hardware. > > Thank you, > Alex >
