[
https://issues.apache.org/jira/browse/PHOENIX-2685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alok Singh updated PHOENIX-2685:
--------------------------------
Attachment: memory_manager_aware_outputstream.patch
> SpoolingResultIterator opening a large number of files, hitting process max
> open file limit
> -------------------------------------------------------------------------------------------
>
> Key: PHOENIX-2685
> URL: https://issues.apache.org/jira/browse/PHOENIX-2685
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.5.1, 4.6.0, 4.7.0
> Reporter: Alok Singh
> Labels: patch
> Attachments: memory_manager_aware_outputstream.patch
>
>
> During one of out load tests, in which we execute 5-10 concurrent requests
> over a few hours, we have found that eventually the jvm process runs out of
> file handles due to 1000s of tmp files (/tmp/ResultSpoolerxxxxxx.bin).
> Upon further investigation, it looks like that under high concurrent load,
> with the default 'phoenix.query.spoolThresholdBytes' value,
> SpoolingResultIterator will almost always run into this issue.
> {code}
> final MemoryChunk chunk = mm.allocate(0, thresholdBytes);
> long waitTime = System.currentTimeMillis() - startTime;
> GLOBAL_MEMORY_WAIT_TIME.update(waitTime);
> memoryMetrics.getMemoryWaitTimeMetric().change(waitTime);
> DeferredFileOutputStream spoolTo = null;
> try {
> // Can't be bigger than int, since it's the max of the above allocation
> int size = (int)chunk.getSize();
> spoolTo = new DeferredFileOutputStream(size, "ResultSpooler",".bin", new
> File(spoolDirectory)) {
> @Override
> protected void thresholdReached() throws IOException {
> try {
> super.thresholdReached();
> } finally {
> chunk.close();
> }
> }
> };
> {code}
> The memory chunks successfully created here will not be released until a GC
> cycle invokes the MemoryChunk finalizer. Since the default behavoir is to
> allocate the full "spoolThresholdBytes" and only resize the chunk after
> iterating over the scan results, if there is sufficient concurrent load the
> GlobablMemoryManager will run out of freememory and cause the
> SpoolingIterators to always spool to disk.
> {code}
> int size = (int)chunk.getSize();
> spoolTo = new DeferredFileOutputStream(size, "ResultSpooler",".bin", new
> File(spoolDirectory))
> {code}
> The "size" ends up being 0, causing the "thresholdReached()" method to be
> invoked when an attempt is made to write to the SpoolingIterator.
> One way to mitigate this is to reduce the size of
> "phoenix.query.spoolThresholdBytes". We ended up reducing it to 10Mb.
> Another would be to use a "Memory Usage" aware outputstream that has a
> reference to the memory manager and switches to a spooling only when
> allocation request to the memory manager fail. A patch with a simple
> MemoryManager based outputstream/SpoolingResultIterator is attached.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
