Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/319#discussion_r49781604
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java
---
@@ -286,7 +286,7 @@ public void flush(boolean schemaChanged) throws
IOException {
// sender has acknowledged the terminate request. After sending
the last batch, all further batches are
// dropped.
// 3. Partitioner thread is interrupted due to cancellation of
fragment.
- final boolean isLastBatch = isLast || terminated ||
Thread.currentThread().isInterrupted();
+ final boolean isLastBatch = isLast ||
Thread.currentThread().isInterrupted();
--- End diff --
That is true. We have 2 possible solutions to fix this problem:
- either we change all receivers so they no longer wait for the last batch
when it's an early termination
- or we make sure the partition sender sends the last batch as soon as
possible to avoid the case where it's sent too late.
Like we discussed it, it's not easy to enforce the 1st solution as even the
senders don't respect this , e.g. a single sender that receives an early
termination message will close the fragment without letting it's receivers wait
for the last batch.
I will update the PR to implement the 2nd solution
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
