James Taylor created PHOENIX-2405:
-------------------------------------
Summary: Use Java heap memory instead of memory mapped files
during ORDER BY
Key: PHOENIX-2405
URL: https://issues.apache.org/jira/browse/PHOENIX-2405
Project: Phoenix
Issue Type: Bug
Reporter: James Taylor
We currently use memory mapped files to buffer data as it's being sorted in an
ORDER BY (see MappedByteBufferQueue). The following types of exceptions have
been seen to occur:
{code}
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:904)
{code}
[~apurtell] has read that memory mapped files are not cleaned up after very
well in Java:
{quote}
"Map failed" means the JVM ran out of virtual address space. If you search
around stack overflow for suggestions on what to do when your app (in this case
Phoenix) encounters this issue when using mapped buffers, the answers tend
toward manually cleaning up the mapped buffers or explicitly triggering a full
GC. See
http://stackoverflow.com/questions/8553158/prevent-outofmemory-when-using-java-nio-mappedbytebuffer
for example. There are apparently long standing JVM/JRE problems with
reclamation of mapped buffers. I think we may want to explore in Phoenix a
different way to achieve what the current code is doing.
{quote}
Instead of using memory mapped files, we could use heap memory, or perhaps
there are other mechanisms too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
