Hi,
Dawid Weiss and I are both involved in the Apache Lucene project and
we know the problems with MappedByteBuffer and unmapping. Dawid
already responded with a source code link to our impl (which needs to
use the hacky cleaner() approach; also look at the heavy documentation
in this class):
https://github.com/apache/lucene-solr/blob/trunk/lucene/core/src/java/org/apache/lucene/store/MMapDirectory.java
So we would be very happy to get this issue resolved! The cleaner()
hack is enabled by default in Lucene if the JVM supports it (so we
won't break if JIGSAW prevents this, but our *large* users would
heavily complain).
This is fundamentally about *integrity* of the runtime. It follows
there
are security implications, but it’s still fundamentally an integrity
issue
and guarding an unsafe operation with a Security Manager is
unfortunately an insufficient solution.
Right, and just to add that there has been many attempts over the years
to find solutions to this issue. I think the closest was atomimcally
remapping but that wasn't feasible on all platforms and also didn't free
up the address space in a timely manner.
So we should really find a solution here. I was talking with several
people on various conferences (Rory O'Donnel or Mark Reinhold) and we
had some ideas how to solve this. My idea how to solve this is
explained below (I am not a JVM internals or Hotspot guy, so excuse
some obviously "wrong" assumptions):
Actually there are 2 issues, not only one. The first issue is, as
mentioned before: you cannot unmap via API. This is needed for many
apps, including Apache Lucene, for a reason which comes more from
"another" bug, and this is my issue #2 (see below).
First, unmapping for Lucene is very important at the moment, because
we operate on the Lucene indexes purely using mmap (see [1]), which
may be several hundreds of Gigabytes easily. On highly dynamic
systems, Lucene often maps new files (also very largeones ) and relies
on the fact, that older, deleted files are unmapped in time (this does
not need to be ASAP, just "in time"). So we have those 2 "bugs", which
force us to unmap:
(1) disk space issues / delete after last close (POSIX) vs. No delete
at all (Windows)
- disk space: we have seen customers running out of disk space on
Lucene, because unmapping wasn’t done in time and therefore POSIX with
delete on last close cannot free the disk space, although the file was
already deleted. The problem you are seeing on Windows that you cannot
delete, is therefore worse on Linux, because it is hidden to the user
- you cannot free the disk space of the deleted file! Lucene creates
and deletes files all the time while indexing realtime data (e.g.
think of Github's very dynamic code search index, which is backed by
Lucene/Elasticsearch).
- virtual memory: If you map huge files (several hundreds of
Gigabytes) and they are not unmapped in time, you may run out of
virtual address space. This especially affects Windows, because it
does not use the full 46 bits (or like that) of addresses. So
effectively you can only map like 4 Terabytes on Windows. If you have
fragmentation of address space this gets worse (In Lucene, we map in
chunks of 1 GiB because of the signed 32 bit integer limit of
ByteBuffer, so fragmentation is not our biggest issue).
(2) It takes veeeeeeeeeeeeeeeery long time until the unmapping
actually occurs!
This is the real bug! If the garbage collector would clean up the
buffers asap, we would not need to unmap from user code. In Lucene we
just delay the file delete on Windows, so we are not really affected
by the file deletion inability (but that would be nice if it could be
fixed).
If you look at the usage pattern of those huge, mapped files, you will
see why they are in most cases *never ever* unmapped automatically:
Lucene maps very large files and uses them for longer time. So the
MappedByteBuffer object gets migrated to older generations on the
heap. Garbage collection there happens, of course, very delayed. That
would not be the most problematic part, but there is a second issue:
The MappedByteBuffer object is just a very small object (in heap size
measurement: just an object header and a few pointers), so the garbage
collector does not see it as heavy! It's just a very small like 30
bytes object instance. Why should the Garbage collector clean it up?
And in fact it will almost never do this! The garbage collector cannot
see that our 30 bytes object instance "sits" on something like 300
Gigabytes of virtual memory and disk space!
One proposal to fix this would be to add something like an internal
OpenJDK Java Annotation or similar where you can "mark" heavy objects,
so Garbage collector could free them by preference (similar to
sun.misc.Contended).
For the Apache Lucene team,
Uwe
[1]
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
-----
Uwe Schindler
uschind...@apache.org
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/
