[
https://issues.apache.org/jira/browse/TEZ-3701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15987602#comment-15987602
]
Jason Lowe commented on TEZ-3701:
---------------------------------
The code doesn't explicitly wait on the executor (e.g.: shutdown() and
awaitTermination()) but it does implement its own form of waiting via
{{pendingSpillCount}}. The code in {{close()}} looks like it should properly
wait for spills to complete before performing the final merge. Could you
elaborate on how it fails to wait properly?
{code}
public List<Event> close() throws IOException, InterruptedException {
List<Event> eventList = Lists.newLinkedList();
isShutdown.set(true);
spillLock.lock();
try {
LOG.info(destNameTrimmed + ": " + "Waiting for all spills to complete :
Pending : " + pendingSpillCount.get());
while (pendingSpillCount.get() != 0 && spillException == null) {
spillInProgress.await();
}
} finally {
spillLock.unlock();
}
{code}
> UnorderedPartitionedKVWriter does not wait for submitted tasks to finish
> before finalMerge.
> -------------------------------------------------------------------------------------------
>
> Key: TEZ-3701
> URL: https://issues.apache.org/jira/browse/TEZ-3701
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Harish Jaiprakash
> Assignee: Harish Jaiprakash
> Priority: Blocker
>
> UnorderedPartitionedKVWriter add task to the executor, but does not wait for
> them to finish before starting the final merge. This can cause finalMerge to
> fail or write incorrect data.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
