[
https://issues.apache.org/jira/browse/SOLR-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127863#comment-18127863
]
Jason Gerlowski commented on SOLR-18551:
----------------------------------------
It's probably worth auditing our usage of Lucene's IOContext throughout our
code. Even if this leak didn't exist, Lucene can operate more efficiently when
we're able to give it the correct hints about how a file-stream needs to be
read.
We also arguably shouldn't be using MMapDirectory for recovery/replication at
all, even if it's the "main" directory for searching/indexing. The
[recommendation from Lucene folks like
Uwe|https://github.com/apache/lucene/issues/15054#issuecomment-3184410243]
seems to be that NIOFSDirectory is more appropriate for replication use-cases.
But both of those are probably better as separate tickets...
> IndexFetcher can leak "mmaps" on TLOG/PULL non-leaders
> ------------------------------------------------------
>
> Key: SOLR-18551
> URL: https://issues.apache.org/jira/browse/SOLR-18551
> Project: Solr
> Issue Type: Bug
> Affects Versions: 10.0, main(11.0), 9.10.1
> Reporter: Jason Gerlowski
> Priority: Major
>
> MMapDirectory works by mapping index files into a process's virtual address
> space, but deleting these maps later on can an expensive operation for the
> JVM. Lucene, starting in 9.12, has dealt with this expense by lumping
> mappings together in "arenas" which can then be unmapped in bulk later. This
> changes the mmap lifecycle a bit: individual mmaps are added to the arena on
> creation and now persist until the entire arena is closed at some point later
> (typically when the DirectoryReader itself is closed).
> When DirectoryReaders and SegmentReaders (and the arena's they own) are
> long-lived, these mmaps can accumulate and eventually exhaust the OS mmap
> count or even the virtual-memory for a Solr process. This doesn't happen
> normally because IndexWriter, IndexReader, etc. are conscientious in the
> files that they open, and because SegmentReaders don't tend to be open all
> that long.
> But this has caused leaks in Lucene
> ([example|https://github.com/apache/lucene/issues/15068]) and I've found a
> leak scenario in Solr 9.10.1 now too.
> *Solr Leak Scenario*
> The particular path I've seen that triggers this leak in Solr is around the
> periodic polling done by TLOG and PULL replicas.
> Each time a TLOG or PULL replica polls the leader and there are new changes,
> the leader will provide a list of its index files and the follower will call
> "[IndexFetcher.slowFileExists|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L1337-L1344]";
> to see which of the files are already present locally. "slowFileExists"
> calls {{Directory.openInput(fileName, IOContext.DEFAULT)}} as a quick way to
> see if the file is present. But when MMapDirectory is in use, this tiny
> existence check creates a (possibly large) mmap that may stick around for
> quite awhile - often until the segment is merged away and the
> SegmentReader/arena are closed.
> This existence-check leak happens for each leader index file that exists on
> the follower and repeats every polling cycle. I've seen this hit the
> processes mmap limit. I've also seen it exhaust the process's virtual memory
> space (*128TB!*) with a comparatively small index of ~100gb.
> The issue tends to be worse on large indices that receive a trickle of
> updates: both because they have more index files and because they have many
> larger index files that are infrequently involved in merges.
> *Solution*
> The fix here is actually pretty simple. IndexFetcher.slowFileExists needs to
> use IOContext.READONCE which sidesteps Lucene's "arena pooling" codepath
> altogether. Given that slowFileExists doesn't care about the file content at
> all it seems wasteful to even create a mmap for it in the first place and
> there's probably a more efficient way to check its existence.
> Lucene's arena-pooling also gets much better in 10.x, as they tweak some
> defaults related to this leak behavior. We should consider adopting those
> changes in 9.x by setting
> {{-Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits}} to 64 (the
> Lucene default in 10.x due to all the problems this pooling behavior raised
> in their 9.12.x line). See
> [here|https://github.com/apache/lucene/issues/15068] for more context on that
> sysprop.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
