Jason Gerlowski created SOLR-18551:
--------------------------------------
Summary: IndexFetcher can leak "mmaps" on TLOG/PULL non-leaders
Key: SOLR-18551
URL: https://issues.apache.org/jira/browse/SOLR-18551
Project: Solr
Issue Type: Bug
Affects Versions: 9.10.1, 10.0, main(11.0)
Reporter: Jason Gerlowski
MMapDirectory works by mapping index files into a process's virtual address
space, but deleting these maps later on can an expensive operation for the JVM.
Lucene, starting in 9.12, has dealt with this expense by lumping mappings
together in "arenas" which can then be unmapped in bulk later. This changes
the mmap lifecycle a bit: individual mmaps are added to the arena on creation
and now persist until the entire arena is closed at some point later (typically
when the DirectoryReader itself is closed).
When DirectoryReaders and SegmentReaders (and the arena's they own) are
long-lived, these mmaps can accumulate and eventually exhaust the OS mmap count
or even the virtual-memory for a Solr process. This doesn't happen normally
because IndexWriter, IndexReader, etc. are conscientious in the files that they
open, and because SegmentReaders don't tend to be open all that long.
But this has caused leaks in Lucene
([example|https://github.com/apache/lucene/issues/15068]) and I've found a leak
scenario in Solr 9.10.1 now too.
*Solr Leak Scenario*
The particular path I've seen that triggers this leak in Solr is around the
periodic polling done by TLOG and PULL replicas.
Each time a TLOG or PULL replica polls the leader and there are new changes,
the leader will provide a list of its index files and the follower will call
"IndexFetcher.slowFileExists" to see which of the files are already present
locally. "slowFileExists" calls {{Directory.openInput(fileName,
IOContext.DEFAULT)}} as a quick way to see if the file is present. But when
MMapDirectory is in use, this tiny existence check creates a (possibly large)
mmap that may stick around for quite awhile - often until the segment is merged
away and the SegmentReader/arena are closed.
This existence-check leak happens for each leader index file that exists on the
follower and repeats every polling cycle. I've seen this hit the processes
mmap limit. I've also seen it exhaust the process's virtual memory space
(128TB!) with a comparatively small index of ~100gb.
The issue tends to be worse on large indices that receive a trickle of updates:
both because they have more index files and because they have many larger index
files that are infrequently involved in merges.
*Solution*
The fix here is actually pretty simple. IndexFetcher.slowFileExists needs to
use IOContext.READONCE which sidesteps Lucene's "arena pooling" codepath
altogether. Given that slowFileExists doesn't care about the file content at
all it seems wasteful to even create a mmap for it in the first place and
there's probably a more efficient way to check its existence.
Lucene's arena-pooling also gets much better in 10.x, as they tweak some
defaults related to this leak behavior. We should consider adopting those
changes in 9.x by setting
{{-Dorg.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits}} to 64 (the
Lucene default in 10.x due to all the problems this pooling behavior raised in
their 9.12.x line). See [here|https://github.com/apache/lucene/issues/15068]
for more context on that sysprop.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
