[
https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14486543#comment-14486543
]
Jeff Zhang commented on TEZ-714:
--------------------------------
[~bikassaha] Upload new patch to address the review comments.
bq. The test is not checking/verifying that one of the commits was indeed not
scheduled on the threadpool and cancelled before that. Maybe set a flag when
the callable is scheduled and check that the flag was not set. Also perhaps set
a flag when interrupted exception is caught when the scheduled event got
canceled.
java built-in ThreadPoolExecutor can't guarantee the other commit is waiting in
the threadpool when it it canceled, e.g. the following sequence may happen.
* commit 1 is running and commit 2 is waiting on the thread pool
* commit 1 is failed and then commit 2 is canceled.
* But before commit 2 is canceled, it may already been scheduled because commit
1 is failed and there's an idle thread for commit 2 to run.
So I create a customized ThreadPoolExecutor to wait before the commit is
started.
{code}
// mean the commits have been submitted to ThreadPool
Assert.assertEquals(2, dag.commitFutures.size());
// mean the commits have been canceled
Assert.assertTrue(dag.commitFutures.isEmpty());
{code}
> OutputCommitters should not run in the main AM dispatcher thread
> ----------------------------------------------------------------
>
> Key: TEZ-714
> URL: https://issues.apache.org/jira/browse/TEZ-714
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Assignee: Jeff Zhang
> Priority: Critical
> Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-10.patch,
> TEZ-714-11.patch, TEZ-714-2.patch, TEZ-714-3.patch, TEZ-714-4.patch,
> TEZ-714-5.patch, TEZ-714-6.patch, TEZ-714-7.patch, TEZ-714-8.patch,
> TEZ-714-9.patch, Vertex_2.pdf
>
>
> Follow up jira from TEZ-41.
> 1) If there's multiple OutputCommitters on a Vertex, they can be run in
> parallel.
> 2) Running an OutputCommitter in the main thread blocks all other event
> handling, w.r.t the DAG, and causes the event queue to back up.
> 3) This should also cover shared commits that happen in the DAG.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
