[
https://issues.apache.org/jira/browse/DRILL-5512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017741#comment-16017741
]
ASF GitHub Bot commented on DRILL-5512:
---------------------------------------
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/838#discussion_r117536637
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java
---
@@ -173,9 +174,8 @@ public IterOutcome next() {
currentReader.allocate(mutator.fieldVectorMap());
} catch (OutOfMemoryException e) {
- logger.debug("Caught Out of Memory Exception", e);
clearFieldVectorMap();
- return IterOutcome.OUT_OF_MEMORY;
+ throw UserException.memoryError(e).build(logger);
--- End diff --
As it turns out, the idea of the OUT_OF_MEMORY return code works better in
theory than in practice. No reader correctly handles this case. Let's say we
have three columns (a, b, c). Let say that column c needs to double its vector,
but hits OOM. No reader has the internal state needed to hold onto the value
for c, unwind the call stack, then on the next next() call, rewind back to the
point of writing c into the in-flight row.
Moving forward, we want to take a broader approach to memory: budget
sufficient memory that readers can work. Modify the mutators so that they
enforce batch size limits so that the reader operates within its budget.
As we move to that approach, the OUT_OF_MEMORY status will be retired.
The JIRA mentions another JIRA that holds a spec for all this stuff;
something we discussed six months ago, but did not have time to implement then.
This all merits a complete discussion; maybe we can discuss the overall
approach in that other JIRA.
> Standardize error handling in ScanBatch
> ---------------------------------------
>
> Key: DRILL-5512
> URL: https://issues.apache.org/jira/browse/DRILL-5512
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.10.0
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Priority: Minor
> Labels: ready-to-commit
> Fix For: 1.10.0
>
>
> ScanBatch is the Drill operator executor that handles most readers. Like most
> Drill operators, it uses an ad-hoc set of error detection and reporting
> methods that evolved over Drill development.
> This ticket asks to standardize on error handling as outlined in DRILL-5083.
> This basically means reporting all errors as a {{UserException}} rather than
> using the {{IterOutcome.STOP}} return status or using the
> {{FragmentContext.fail()}} method.
> This work requires the new error codes introduced in DRILL-5511, and is a
> step toward making readers aware of vector size limits.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
