[
https://issues.apache.org/jira/browse/PHOENIX-2685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alok Singh updated PHOENIX-2685:
--------------------------------
Description:
During one of out load tests, in which we execute 5-10 concurrent requests over
a few hours, we have found that eventually the jvm process runs out of file
handles due to 1000s of tmp files (/tmp/ResultSpoolerxxxxxx.bin).
Upon further investigation, it looks like that under high concurrent load, with
the default 'phoenix.query.spoolThresholdBytes' value, SpoolingResultIterator
will almost always run into this issue.
{code}
final MemoryChunk chunk = mm.allocate(0, thresholdBytes);
long waitTime = System.currentTimeMillis() - startTime;
GLOBAL_MEMORY_WAIT_TIME.update(waitTime);
memoryMetrics.getMemoryWaitTimeMetric().change(waitTime);
DeferredFileOutputStream spoolTo = null;
try {
// Can't be bigger than int, since it's the max of the above allocation
int size = (int)chunk.getSize();
spoolTo = new DeferredFileOutputStream(size, "ResultSpooler",".bin", new
File(spoolDirectory)) {
@Override
protected void thresholdReached() throws IOException {
try {
super.thresholdReached();
} finally {
chunk.close();
}
}
};
{code}
The memory chunks successfully created here will not be released until a GC
cycle invokes the MemoryChunk finalizer. Since the default behavoir is to
allocate the full "spoolThresholdBytes" and only resize the chunk after
iterating over the scan results, if there is sufficient concurrent load the
GlobablMemoryManager will run out of freememory and cause the SpoolingIterators
to always spool to disk.
{code}
int size = (int)chunk.getSize();
spoolTo = new DeferredFileOutputStream(size, "ResultSpooler",".bin", new
File(spoolDirectory))
{code}
The "size" ends up being 0, causing the "thresholdReached()" method to be
invoked when an attempt is made to write to the DeferredFileouputStream.
One way to mitigate this is to reduce the size of
"phoenix.query.spoolThresholdBytes". We ended up reducing it to 10Mb.
Another would be to use a "Memory Usage" aware outputstream that has a
reference to the memory manager and switches to a spooling only when allocation
request to the memory manager fail. A patch with a simple MemoryManager based
outputstream/SpoolingResultIterator is attached.
was:
During one of out load tests, in which we execute 5-10 concurrent requests over
a few hours, we have found that eventually the jvm process runs out of file
handles due to 1000s of tmp files (/tmp/ResultSpoolerxxxxxx.bin).
Upon further investigation, it looks like that under high concurrent load, with
the default 'phoenix.query.spoolThresholdBytes' value, SpoolingResultIterator
will almost always run into this issue.
{code}
final MemoryChunk chunk = mm.allocate(0, thresholdBytes);
long waitTime = System.currentTimeMillis() - startTime;
GLOBAL_MEMORY_WAIT_TIME.update(waitTime);
memoryMetrics.getMemoryWaitTimeMetric().change(waitTime);
DeferredFileOutputStream spoolTo = null;
try {
// Can't be bigger than int, since it's the max of the above allocation
int size = (int)chunk.getSize();
spoolTo = new DeferredFileOutputStream(size, "ResultSpooler",".bin", new
File(spoolDirectory)) {
@Override
protected void thresholdReached() throws IOException {
try {
super.thresholdReached();
} finally {
chunk.close();
}
}
};
{code}
The memory chunks successfully created here will not be released until a GC
cycle invokes the MemoryChunk finalizer. Since the default behavoir is to
allocate the full "spoolThresholdBytes" and only resize the chunk after
iterating over the scan results, if there is sufficient concurrent load the
GlobablMemoryManager will run out of freememory and cause the SpoolingIterators
to always spool to disk.
{code}
int size = (int)chunk.getSize();
spoolTo = new DeferredFileOutputStream(size, "ResultSpooler",".bin", new
File(spoolDirectory))
{code}
The "size" ends up being 0, causing the "thresholdReached()" method to be
invoked when an attempt is made to write to the SpoolingIterator.
One way to mitigate this is to reduce the size of
"phoenix.query.spoolThresholdBytes". We ended up reducing it to 10Mb.
Another would be to use a "Memory Usage" aware outputstream that has a
reference to the memory manager and switches to a spooling only when allocation
request to the memory manager fail. A patch with a simple MemoryManager based
outputstream/SpoolingResultIterator is attached.
> SpoolingResultIterator opening a large number of files, hitting process max
> open file limit
> -------------------------------------------------------------------------------------------
>
> Key: PHOENIX-2685
> URL: https://issues.apache.org/jira/browse/PHOENIX-2685
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.5.1, 4.6.0, 4.7.0
> Reporter: Alok Singh
> Labels: patch
> Attachments: memory_manager_aware_outputstream.patch
>
>
> During one of out load tests, in which we execute 5-10 concurrent requests
> over a few hours, we have found that eventually the jvm process runs out of
> file handles due to 1000s of tmp files (/tmp/ResultSpoolerxxxxxx.bin).
> Upon further investigation, it looks like that under high concurrent load,
> with the default 'phoenix.query.spoolThresholdBytes' value,
> SpoolingResultIterator will almost always run into this issue.
> {code}
> final MemoryChunk chunk = mm.allocate(0, thresholdBytes);
> long waitTime = System.currentTimeMillis() - startTime;
> GLOBAL_MEMORY_WAIT_TIME.update(waitTime);
> memoryMetrics.getMemoryWaitTimeMetric().change(waitTime);
> DeferredFileOutputStream spoolTo = null;
> try {
> // Can't be bigger than int, since it's the max of the above allocation
> int size = (int)chunk.getSize();
> spoolTo = new DeferredFileOutputStream(size, "ResultSpooler",".bin", new
> File(spoolDirectory)) {
> @Override
> protected void thresholdReached() throws IOException {
> try {
> super.thresholdReached();
> } finally {
> chunk.close();
> }
> }
> };
> {code}
> The memory chunks successfully created here will not be released until a GC
> cycle invokes the MemoryChunk finalizer. Since the default behavoir is to
> allocate the full "spoolThresholdBytes" and only resize the chunk after
> iterating over the scan results, if there is sufficient concurrent load the
> GlobablMemoryManager will run out of freememory and cause the
> SpoolingIterators to always spool to disk.
> {code}
> int size = (int)chunk.getSize();
> spoolTo = new DeferredFileOutputStream(size, "ResultSpooler",".bin", new
> File(spoolDirectory))
> {code}
> The "size" ends up being 0, causing the "thresholdReached()" method to be
> invoked when an attempt is made to write to the DeferredFileouputStream.
> One way to mitigate this is to reduce the size of
> "phoenix.query.spoolThresholdBytes". We ended up reducing it to 10Mb.
> Another would be to use a "Memory Usage" aware outputstream that has a
> reference to the memory manager and switches to a spooling only when
> allocation request to the memory manager fail. A patch with a simple
> MemoryManager based outputstream/SpoolingResultIterator is attached.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
