Hi,
yes this looks a big problem. The functionality and heuristics for
grouping arenas in MMapDirectory ist made for the usual index file
names. Those filenames do not conform to the standard pattern of Lucene
and therefore cannot be grouped by segment number and it also is not
marked as metadata file. Basically what happens is that files in Lucene
are opened once and to spare system resources and additional overhead
while closing arenas is to group those together. The inappropiate named
files get a new group on each open and they are not released in time and
consume resources. MMapDir cant handle the files correctly.
The problem with Lucene 9 is that it reserves slots for 1024 groups by
default per index and reopening the same file over and over keeps those
groups for much longer than expected. This can be mitigated by another
system property (see MMapDirectory javadocs) to limit the number of
groups to 64 or even lower (this was done in Lucene 10). The limit 1024
was way too large as every index can have up to 1024 arenas open at same
time. The number of mapping allowed by the kernel (unless tuned) is only
65536 arenas / maps. So once you have 64 indexes and reopen the file
over and over while keeping previous mmaps, you run out of mappings in
the kernel.
if you see this error for other files this is just a side effect of the
open same file in a new group, at some point you run out of kernel
resources after 64 indexes.
Quick workaround:
https://lucene.apache.org/core/9_12_0/core/org/apache/lucene/store/MMapDirectory.html#NO_GROUPING
This restores behaviour of around Lucene 9.10 where all files are closed
and all their mappings not cached.
Or set following sysprop (defaults 1024) to much lower values (I'd
recommend 16 if you have many indexes on same instance):
org.apache.lucene.store.MMapDirectory.sharedArenaMaxPermits
(with the expense of slowdown on closing IndexInputs).
Uwe
Am 19.02.2026 um 18:21 schrieb Matthew Biscocho (BLOOMBERG/ 120 PARK):
Thanks Uwe for the info.
We ran into this error in a bunch of different ways from Solr. I did some
digging and we ran into this with other places that should probably have this
change as well for example the replication.properties file:
https://github.com/apache/solr/blob/a36c3b511dbd56e929d93f2b2ed7cf94adb436e9/solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java#L1189
Ran into it here that also include actual segments that probably should be
READONCE probably:
https://github.com/apache/solr/blob/a36c3b511dbd56e929d93f2b2ed7cf94adb436e9/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L1367
But this issue still occurred with segment files that did do a READONCE
java.io.IOException: Map failed:
MemorySegmentIndexInput(path=\"/var/solrIndex/foobar/foobar_shard25_replica_t49/data/index.20260215100909238/_55i.nvd\")
[this may be caused by lack of enough unfragmented virtual address space or too
restrictive virtual memory limits enforced by the operating system, preventing us to map
a chunk of 2162259 bytes
It throws these memory issues from here:
https://github.com/apache/solr/blob/a36c3b511dbd56e929d93f2b2ed7cf94adb436e9/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L1303
Looking at Lucene, one difference that Lucene 9.12 didn't have is this change
relating to ReadOnceHint? Not sure if this makes a
difference.https://github.com/apache/lucene/blame/main/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java#L328
From:[email protected] At: 02/19/26 12:02:57 UTC-5:00To:[email protected]
Cc:[email protected]
Subject: Re: FYI for those trying Java 21
Fantastic! PR welcome ;-)
If you don't get to it, someone will.
While what you suggest looks totally reasonable, do you really think this
is likely to fix the matter? We see similar errors for other files like a
".tim" file.
On Thu, Feb 19, 2026 at 8:48 AM Uwe Schindler via dev<[email protected]>
wrote:
i think the issue is here:
https://github.com/apache/solr/blob/b71c872c35b1a12551e3d9b832c1ef9c40a1ea0a/sol
r/core/src/java/org/apache/solr/core/SolrCore.java#L443
this should use IOContext.READ_ONCE
uwe
Am 19.02.2026 um 14:09 schrieb Uwe Schindler via dev:
Oh I see, it is "index.properties". This is not a Lucene file. I think
when opening that file it should pass READ_ONCE as IOContext to get
rid of that issue.
Uwe
Am 19.02.2026 um 14:05 schrieb Uwe Schindler via dev:
Hi,
I think the problem here is that not all fixes around that use-case
were applied to Lucene 9. The problem here is that in Lucene 9 we do
not open all files in "read-once" mode, especially the segment infos
files. The error message about 78 bytes not beeing able to be mapped
looks really like the segment infos file, we don't have any other
files so small. This is no longer an issue in Lucene 10.
Uwe
Am 18.02.2026 um 17:40 schrieb David Smiley:
The same boolean setting was useful as a mitigation for the following
error, totally different business / search cluster -- Solr 9.10,
Java 21,
did not happen with Java 17 with identical search cluster:
Caused by: java.io.IOException: Map failed:
MemorySegmentIndexInput(path="/var/solr/REDACTED_PATH/REDACTED_COLLECTION_shard39_replica_t77/data/index.properties")
[this may be caused by lack of enough unfragmented virtual address
space or
too restrictive virtual memory limits enforced by the operating
system,
preventing us to map a chunk of 78 bytes. Please review 'ulimit -v',
'ulimit -m' (both should return 'unlimited'), and 'sysctl
vm.max_map_count'
Reminder: the setting doesn't exist in Solr 10; a proper solution
must be
found for a Solr 10 upgrade, assuming the problem re-appears. I'm
somewhat
optimistic a further Java upgrade (to say 25) would be helpful,
albeit I
haven't investigated that yet.
On Wed, Aug 21, 2024 at 1:23 PM David Smiley<[email protected]>
wrote:
For those trying Java 21 with Solr, if you find the performance to be
bad then try setting
-Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
This made a huge difference for me on our Solr nodes at work.
I filed the following ticket to Solr that ought to avoid the JDK
performance regression if done:
https://issues.apache.org/jira/browse/SOLR-17375
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail:[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail:[email protected]
For additional commands, e-mail:[email protected]
--
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail:[email protected]
