[
https://issues.apache.org/jira/browse/DRILL-5273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15871363#comment-15871363
]
Paul Rogers commented on DRILL-5273:
------------------------------------
As it turns out, the solution is simple once we recognize the problem. The
{{CompliantTextRecordReader}} simply allocates normal, non-managed buffers,
then releases them in its {{close()}} call. With this fix, the 5000 file query
runs to completion as expected with no OOM.
Note that an even better solution would be to use normal heap buffers for
working buffers. There is really no gain in using a single 1 MB direct buffer
per scan: working with the buffer is expensive, and so is allocating and
managing it.
> CompliantTextReader exhausts 4 GB memory when reading 5000 small files
> ----------------------------------------------------------------------
>
> Key: DRILL-5273
> URL: https://issues.apache.org/jira/browse/DRILL-5273
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.10
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Fix For: 1.10
>
>
> A test case was created that consists of 5000 text files, each with a single
> line with the file number: 1 to 5001. Each file has a single record, and at
> most 4 characters per record.
> Run the following query:
> {code}
> SELECT * FROM `dfs.data`.`5000files/text
> {code}
> The query will fail with an OOM in the scan batch on around record 3700 on a
> Mac with 4GB of direct memory.
> The code to read records in {ScanBatch} is complex. The following appears to
> occur:
> * Iterate over the record readers for each file.
> * For each, call setup
> The setup code is:
> {code}
> public void setup(OperatorContext context, OutputMutator outputMutator)
> throws ExecutionSetupException {
> oContext = context;
> readBuffer = context.getManagedBuffer(READ_BUFFER);
> whitespaceBuffer = context.getManagedBuffer(WHITE_SPACE_BUFFER);
> {code}
> The two buffers are in direct memory. There is no code that releases the
> buffers.
> The sizes are:
> {code}
> private static final int READ_BUFFER = 1024*1024;
> private static final int WHITE_SPACE_BUFFER = 64*1024;
> = 1,048,576 + 65536 = 1,114,112
> {code}
> This is exactly the amount of memory that accumulates per call to
> {{ScanBatch.next()}}
> {code}
> Ctor: 0 -- Initial memory in constructor
> Init setup: 1114112 -- After call to first record reader setup
> Entry Memory: 1114112 -- first next() call, returns one record
> Entry Memory: 1114112 -- second next(), eof and start second reader
> Entry Memory: 2228224 -- third next(), second reader returns EOF
> ...
> {code}
> If we leak 1 MB per file, with 5000 files we would leak 5 GB of memory, which
> would explain the OOM when given only 4 GB.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
