What I misses to add: Files opened with READ_ONCE are the way to go
anyways, because READ_ONCE files are designed to be kept open for short
time and only read with the same thread opening them (like the pattern
here). This reduces Java safepoints and deoptimization dramatically
because the arenas used are not "shared" and need no thread
synchronization and safepoints on closing.
So my recommenation is: In all Lucene versions user READ_ONCE for all
those shortly open files polled on many times to reduce contention and
JVM safepoints. In addition in Lucene 9 reduce the sysprop
org.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits to very low
values, but can also be tuned for Lucene 10 if you have many indexes on
same machine. Lower values increase cost of closing files but allow more
indexes in parallel.
Uwe
Am 20.02.2026 um 12:41 schrieb Uwe Schindler via dev:
Hi,
yes this looks a big problem. The functionality and heuristics for
grouping arenas in MMapDirectory ist made for the usual index file
names. Those filenames do not conform to the standard pattern of
Lucene and therefore cannot be grouped by segment number and it also
is not marked as metadata file. Basically what happens is that files
in Lucene are opened once and to spare system resources and additional
overhead while closing arenas is to group those together. The
inappropiate named files get a new group on each open and they are not
released in time and consume resources. MMapDir cant handle the files
correctly.
The problem with Lucene 9 is that it reserves slots for 1024 groups by
default per index and reopening the same file over and over keeps
those groups for much longer than expected. This can be mitigated by
another system property (see MMapDirectory javadocs) to limit the
number of groups to 64 or even lower (this was done in Lucene 10). The
limit 1024 was way too large as every index can have up to 1024 arenas
open at same time. The number of mapping allowed by the kernel (unless
tuned) is only 65536 arenas / maps. So once you have 64 indexes and
reopen the file over and over while keeping previous mmaps, you run
out of mappings in the kernel.
if you see this error for other files this is just a side effect of
the open same file in a new group, at some point you run out of kernel
resources after 64 indexes.
Quick workaround:
https://lucene.apache.org/core/9_12_0/core/org/apache/lucene/store/MMapDirectory.html#NO_GROUPING
This restores behaviour of around Lucene 9.10 where all files are
closed and all their mappings not cached.
Or set following sysprop (defaults 1024) to much lower values (I'd
recommend 16 if you have many indexes on same instance):
org.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits
(with the expense of slowdown on closing IndexInputs).
Uwe
Am 19.02.2026 um 18:21 schrieb Matthew Biscocho (BLOOMBERG/ 120 PARK):
Thanks Uwe for the info.
We ran into this error in a bunch of different ways from Solr. I did
some digging and we ran into this with other places that should
probably have this change as well for example the
replication.properties file:
https://github.com/apache/solr/blob/a36c3b511dbd56e929d93f2b2ed7cf94adb436e9/solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java#L1189
Ran into it here that also include actual segments that probably
should be READONCE probably:
https://github.com/apache/solr/blob/a36c3b511dbd56e929d93f2b2ed7cf94adb436e9/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L1367
But this issue still occurred with segment files that did do a READONCE
java.io.IOException: Map failed:
MemorySegmentIndexInput(path=\"/var/solrIndex/foobar/foobar_shard25_replica_t49/data/index.20260215100909238/_55i.nvd\")
[this may be caused by lack of enough unfragmented virtual address
space or too restrictive virtual memory limits enforced by the
operating system, preventing us to map a chunk of 2162259 bytes
It throws these memory issues from here:
https://github.com/apache/solr/blob/a36c3b511dbd56e929d93f2b2ed7cf94adb436e9/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L1303
Looking at Lucene, one difference that Lucene 9.12 didn't have is
this change relating to ReadOnceHint? Not sure if this makes a
difference.https://github.com/apache/lucene/blame/main/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L328
From:[email protected] At: 02/19/26 12:02:57
UTC-5:00To:[email protected]
Cc:[email protected]
Subject: Re: FYI for those trying Java 21
Fantastic! PR welcome ;-)
If you don't get to it, someone will.
While what you suggest looks totally reasonable, do you really think
this
is likely to fix the matter? We see similar errors for other files
like a
".tim" file.
On Thu, Feb 19, 2026 at 8:48 AM Uwe Schindler via
dev<[email protected]>
wrote:
i think the issue is here:
https://github.com/apache/solr/blob/b71c872c35b1a12551e3d9b832c1ef9c40a1ea0a/sol
r/core/src/java/org/apache/solr/core/SolrCore.java#L443
this should use IOContext.READ_ONCE
uwe
Am 19.02.2026 um 14:09 schrieb Uwe Schindler via dev:
Oh I see, it is "index.properties". This is not a Lucene file. I think
when opening that file it should pass READ_ONCE as IOContext to get
rid of that issue.
Uwe
Am 19.02.2026 um 14:05 schrieb Uwe Schindler via dev:
Hi,
I think the problem here is that not all fixes around that use-case
were applied to Lucene 9. The problem here is that in Lucene 9 we do
not open all files in "read-once" mode, especially the segment infos
files. The error message about 78 bytes not beeing able to be mapped
looks really like the segment infos file, we don't have any other
files so small. This is no longer an issue in Lucene 10.
Uwe
Am 18.02.2026 um 17:40 schrieb David Smiley:
The same boolean setting was useful as a mitigation for the
following
error, totally different business / search cluster -- Solr 9.10,
Java 21,
did not happen with Java 17 with identical search cluster:
Caused by: java.io.IOException: Map failed:
MemorySegmentIndexInput(path="/var/solr/REDACTED_PATH/REDACTED_COLLECTION_shard39_replica_t77/data/index.properties")
[this may be caused by lack of enough unfragmented virtual address
space or
too restrictive virtual memory limits enforced by the operating
system,
preventing us to map a chunk of 78 bytes. Please review 'ulimit
-v',
'ulimit -m' (both should return 'unlimited'), and 'sysctl
vm.max_map_count'
Reminder: the setting doesn't exist in Solr 10; a proper solution
must be
found for a Solr 10 upgrade, assuming the problem re-appears. I'm
somewhat
optimistic a further Java upgrade (to say 25) would be helpful,
albeit I
haven't investigated that yet.
On Wed, Aug 21, 2024 at 1:23 PM David Smiley<[email protected]>
wrote:
For those trying Java 21 with Solr, if you find the performance
to be
bad then try setting
-Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
This made a huge difference for me on our Solr nodes at work.
I filed the following ticket to Solr that ought to avoid the JDK
performance regression if done:
https://issues.apache.org/jira/browse/SOLR-17375
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail:[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail:[email protected]
For additional commands, e-mail:[email protected]
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail:[email protected]
