[
https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14377105#comment-14377105
]
Bikas Saha commented on TEZ-714:
--------------------------------
Not seen the patch yet because it may change if you agree with these comments
bq. Regarding the issue of "not sure why group-commit and non-group commit need
to be differentiated in different transitions.
Can this be fixed by having the events for both be different? But still handled
in the same transition. The transition can check if its a group commit event vs
normal commit event (based on event type) - and then log for group commit.
Maybe group commit event can derive from normal commit event.
Is this recovery log written relevant only in the non-commit-at-end case where
group commits can happen before the DAG finishes?
bq. Unit test is still not perfect. Because currently in the DAGImpl/VertexImpl
we run the shared thread pool in the AsynDispatcher
For these tests we could choose to use the normal thread pool by overriding the
setup. Since this is a new test, it can try to not depend on ordering like the
existing tests do. If so, then it should be fine to use the real threadpool
instead of the fake thread pool that delegates to the dispatcher. Maybe you can
create a new TestCommit that starts from scratch without the hacks in
TestVertexImpl.
bq. For the some existing transition, like (RUNNING to ERROR due to INTERNAL
ERROR)
Is this for VertexImpl or DAGImpl? That sounds like a bug. Is that relevant to
the commit operation though?
> OutputCommitters should not run in the main AM dispatcher thread
> ----------------------------------------------------------------
>
> Key: TEZ-714
> URL: https://issues.apache.org/jira/browse/TEZ-714
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Assignee: Jeff Zhang
> Priority: Critical
> Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-2.patch, Vertex_2.pdf
>
>
> Follow up jira from TEZ-41.
> 1) If there's multiple OutputCommitters on a Vertex, they can be run in
> parallel.
> 2) Running an OutputCommitter in the main thread blocks all other event
> handling, w.r.t the DAG, and causes the event queue to back up.
> 3) This should also cover shared commits that happen in the DAG.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
