[
https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14359827#comment-14359827
]
Bikas Saha commented on TEZ-714:
--------------------------------
DAGImpl.finished() and VertexImpl.finished() are intended to run common code
upon completion. Not sure why the code around that area has been refactored.
Not sure why there are edits in the non-commit methods of DAGImpl and
VertexImpl. E.g. could the changes be restricted to the existing commit logic
methods changing to submit async commit operations instead of calling sync
commit (along with maintaining num operations so that we know when all async
operations have completed). And a committedHandleTransition that does the
counting of pending commit operations. When count ==0 and state == committing
then change state to succeeded. If state is not succeeded then its already
changed to some other terminal state. State changes to committing from running
only when there are outstanding commit operations?
Not sure why a group commit completion is different from a non-group commit
completion? Could both be simply considered x+y commit operations that we need
to wait for?
The intention was to run each commit independently so that they could run in
parallel (at the end of a vertex or at the end of a DAG). This would help in
job completion time. Unless I am missing something, the patch does not seem to
be doing that.
> OutputCommitters should not run in the main AM dispatcher thread
> ----------------------------------------------------------------
>
> Key: TEZ-714
> URL: https://issues.apache.org/jira/browse/TEZ-714
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Assignee: Jeff Zhang
> Priority: Critical
> Attachments: DAG_2.pdf, TEZ-714-1.patch, Vertex_2.pdf
>
>
> Follow up jira from TEZ-41.
> 1) If there's multiple OutputCommitters on a Vertex, they can be run in
> parallel.
> 2) Running an OutputCommitter in the main thread blocks all other event
> handling, w.r.t the DAG, and causes the event queue to back up.
> 3) This should also cover shared commits that happen in the DAG.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
