[
https://issues.apache.org/jira/browse/DRILL-5273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880018#comment-15880018
]
ASF GitHub Bot commented on DRILL-5273:
---------------------------------------
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/750#discussion_r102650409
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/text/compliant/CompliantTextRecordReader.java
---
@@ -118,12 +118,21 @@ public boolean apply(@Nullable SchemaPath path) {
* @param outputMutator Used to create the schema in the output record
batch
* @throws ExecutionSetupException
*/
+ @SuppressWarnings("resource")
@Override
public void setup(OperatorContext context, OutputMutator outputMutator)
throws ExecutionSetupException {
oContext = context;
- readBuffer = context.getManagedBuffer(READ_BUFFER);
- whitespaceBuffer = context.getManagedBuffer(WHITE_SPACE_BUFFER);
+ // Note: DO NOT use managed buffers here. They remain in existence
+ // until the fragment is shut down. The buffers here are large.
--- End diff --
The reason is a bit different. The original call allocates a managed
buffer: it is freed only when the fragment context shuts down at the end of
query execution. But, if we read many files (5000 in one test case), then we
leave 5000 buffers in existence for the whole query.
Instead, we want to take control over buffer lifetime. We allocate a
regular (not managed) buffer ourselves, and then release it when this reader
closes.
That way, instead of accumulating 5000 buffers of 1 MB each, we have only
one 1 MB buffer in existence at any one time.
Of course, a further refinement would be to allocate the buffer on the
ScanBatch and have all 5000 readers sequentially share that same buffer. But, I
was not sure that any performance benefit was worth the cost in extra code
complexity...
> CompliantTextReader exhausts 4 GB memory when reading 5000 small files
> ----------------------------------------------------------------------
>
> Key: DRILL-5273
> URL: https://issues.apache.org/jira/browse/DRILL-5273
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.10
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Fix For: 1.10.0
>
>
> A test case was created that consists of 5000 text files, each with a single
> line with the file number: 1 to 5001. Each file has a single record, and at
> most 4 characters per record.
> Run the following query:
> {code}
> SELECT * FROM `dfs.data`.`5000files/text
> {code}
> The query will fail with an OOM in the scan batch on around record 3700 on a
> Mac with 4GB of direct memory.
> The code to read records in {ScanBatch} is complex. The following appears to
> occur:
> * Iterate over the record readers for each file.
> * For each, call setup
> The setup code is:
> {code}
> public void setup(OperatorContext context, OutputMutator outputMutator)
> throws ExecutionSetupException {
> oContext = context;
> readBuffer = context.getManagedBuffer(READ_BUFFER);
> whitespaceBuffer = context.getManagedBuffer(WHITE_SPACE_BUFFER);
> {code}
> The two buffers are in direct memory. There is no code that releases the
> buffers.
> The sizes are:
> {code}
> private static final int READ_BUFFER = 1024*1024;
> private static final int WHITE_SPACE_BUFFER = 64*1024;
> = 1,048,576 + 65536 = 1,114,112
> {code}
> This is exactly the amount of memory that accumulates per call to
> {{ScanBatch.next()}}
> {code}
> Ctor: 0 -- Initial memory in constructor
> Init setup: 1114112 -- After call to first record reader setup
> Entry Memory: 1114112 -- first next() call, returns one record
> Entry Memory: 1114112 -- second next(), eof and start second reader
> Entry Memory: 2228224 -- third next(), second reader returns EOF
> ...
> {code}
> If we leak 1 MB per file, with 5000 files we would leak 5 GB of memory, which
> would explain the OOM when given only 4 GB.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
