Github user zbdzzg commented on a diff in the pull request:
https://github.com/apache/drill/pull/584#discussion_r84828174
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/validate/IteratorValidatorBatchIterator.java
---
@@ -301,7 +301,7 @@ public IterOutcome next() {
"Incoming batch [#%d, %s] has an empty schema. This is
not allowed.",
instNum, batchTypeName));
}
- if (incoming.getRecordCount() > MAX_BATCH_SIZE) {
+ if (incoming.getRecordCount() >= MAX_BATCH_SIZE) {
--- End diff --
Thanks for reply.
I tried turn assertion off and rerun, the IOB is produced again. The
IllegalStateException is thrown when turning it on.
The built-in storage plug-ins always have smaller batch limitation (smaller
than 65536) in reader implementations, so this problem is supposed to be
avoided by design. The scene to us running into this problem is that we have a
customized storage plug-in with a large batch-size set in the reader part. I
personally guess that the assertion is in technically enabled in testing /
debugging for a new written storage plug-in, so the IllegalStateException
should be thrown in which the assertion is on, then we know that our batch-size
is too large to execute all stuffs. As a comparison, a wired IOB exception
makes us confused.
If we consider the scene disabling assertion, another bound check should be
inserted to LimitRecordBatch.java, but what I am seeing is that most check
logics is written at IteratorValidator, I suppose that author wants storage
developers make all things work in the early debugging, the assertion can be
off after the exceptions like this IOB are resolved.
So I don t think that making any changes to LimitRecordBatch.java is
reasonable rather than Doing same thing to validator part. This change benefits
the plugin development. Do you agree?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
