Hi all,
After upgrading our OpenSearch cluster from 2.16.0 to 2.19.1 (moving from
Lucene 9.10 to Lucene 9.12), our largest clusters started crashing with the
following error:
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 2097152 bytes. Error
detail: AllocateHeap
We narrowed down the issue to the vm max map count (262144) being reached.
Prior to server crash, we see map count (measured by `cat /proc/{pid}/maps
| wc -l`) approach the 262144 limit we set. Looking at one of the outputs
of `cat /proc/{pid}/maps`, we observed that 246K of the 252K maps are for
deleted doc values (.dvd) files.
Is this expected? If so, were there any changes in the Lucene codebase
between those two versions that could have caused this? Any suggestions on
debugging?
Thanks in advance and sorry if this is a better question for the OS
community or the Lucene developer list.
Justin Borromeo
