GitHub user zbdzzg opened a pull request:
https://github.com/apache/drill/pull/584
DRILL-4884: Fix bug that drill sometimes produced IOB exception while
querying data of 65536 limitation
Drill produces IOB while using a non batched scanner and limiting SQL by
65536.
SQL:
```
select id from isearch.tmall_auction_cluster limit 1 offset 65535
```
Result:
```
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534)
~[classes/:na]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:324)
[classes/:na]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184)
[classes/:na]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290)
[classes/:na]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[classes/:na]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
Caused by: java.lang.IndexOutOfBoundsException: index: 131072, length: 2
(expected: range(0, 131072))
at io.netty.buffer.DrillBuf.checkIndexD(DrillBuf.java:175)
~[classes/:4.0.27.Final]
at io.netty.buffer.DrillBuf.chk(DrillBuf.java:197)
~[classes/:4.0.27.Final]
at io.netty.buffer.DrillBuf.setChar(DrillBuf.java:517)
~[classes/:4.0.27.Final]
at
org.apache.drill.exec.record.selection.SelectionVector2.setIndex(SelectionVector2.java:79)
~[classes/:na]
at
org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.limitWithNoSV(LimitRecordBatch.java:167)
~[classes/:na]
at
org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.doWork(LimitRecordBatch.java:145)
~[classes/:na]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93)
~[classes/:na]
at
org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.innerNext(LimitRecordBatch.java:115)
~[classes/:na]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
~[classes/:na]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
~[classes/:na]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[classes/:na]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[classes/:na]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[classes/:na]
at
org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:94)
~[classes/:na]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
~[classes/:na]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
~[classes/:na]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
~[classes/:na]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
~[classes/:na]
at
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
~[classes/:na]
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:132)
~[classes/:na]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
~[classes/:na]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215)
~[classes/:na]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
~[classes/:na]
at
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81)
~[classes/:na]
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94)
~[classes/:na]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:256)
~[classes/:na]
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:250)
~[classes/:na]
at java.security.AccessController.doPrivileged(Native Method)
~[na:1.8.0_101]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_101]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
~[hadoop-common-2.7.1.jar:na]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:250)
[classes/:na]
... 4 common frames omitted
```
Code from IteratorValidatorBatchIterator.java said that it is OK incoming
returning 65536 records:
```
if (incoming.getRecordCount() > MAX_BATCH_SIZE) { // MAX_BATCH_SIZE == 65536
throw new IllegalStateException(
String.format(
"Incoming batch [#%d, %s] has size %d, which is beyond
the"
+ " limit of %d",
instNum, batchTypeName, incoming.getRecordCount(),
MAX_BATCH_SIZE
));
}
```
Code from LimitRecordBatch.java said that a loop will not break as expected
when the incoming returns 65536 records:
```
private void limitWithNoSV(int recordCount) {
final int offset = Math.max(0, Math.min(recordCount - 1,
recordsToSkip));
recordsToSkip -= offset;
int fetch;
if(noEndLimit) {
fetch = recordCount;
} else {
fetch = Math.min(recordCount, offset + recordsLeft);
recordsLeft -= Math.max(0, fetch - offset);
}
int svIndex = 0;
for(char i = (char) offset; i < fetch; svIndex++, i++) { // since
fetch==recordCount==65536, param i can be increased from 65535 to 65536, then
be casted to char type of value 0, the loop abnormally continues.
outgoingSv.setIndex(svIndex, i);
}
outgoingSv.setRecordCount(svIndex);
}
```
The IllegalStateException should be thrown when incoming returns 65535
records rather than 65536.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zbdzzg/drill master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/584.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #584
----
commit c01f326860fe4d9bed21c3807153d94dc75a7794
Author: hongze.zhz <[email protected]>
Date: 2016-09-09T08:19:16Z
DRILL-4884: Fix bug that drill sometimes produced IOB exception while
querying data of 65536 limitation
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
