[
https://issues.apache.org/jira/browse/TEZ-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14377209#comment-14377209
]
Jeff Zhang commented on TEZ-714:
--------------------------------
bq. Can this be fixed by having the events for both be different? But still
handled in the same transition.
It could, but this may make the transition complicated. Currently we need to
differentiate these 2 kinds of commits, besides there's 2 possible states
(RUNNING, COMMITTING) when the commit happens and we also need check handle 2
different cases (commit succeeded & failure), so there would be totally 8
different cases in one transition which may be difficult to read.
bq. Is this recovery log written relevant only in the non-commit-at-end case
where group commits can happen before the DAG finishes?
Yes
bq. Maybe you can create a new TestCommit that starts from scratch without the
hacks in TestVertexImpl.
Yes, this is I plan to do.
bq. Is this for VertexImpl or DAGImpl? That sounds like a bug. Is that relevant
to the commit operation though?
It is relevant to the abort. Currently in DAG's InternalErrorTransition (no
matter what state it is ), dag would abort directly and go to ERROR state
without waiting for vertex to finish.
> OutputCommitters should not run in the main AM dispatcher thread
> ----------------------------------------------------------------
>
> Key: TEZ-714
> URL: https://issues.apache.org/jira/browse/TEZ-714
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Assignee: Jeff Zhang
> Priority: Critical
> Attachments: DAG_2.pdf, TEZ-714-1.patch, TEZ-714-2.patch, Vertex_2.pdf
>
>
> Follow up jira from TEZ-41.
> 1) If there's multiple OutputCommitters on a Vertex, they can be run in
> parallel.
> 2) Running an OutputCommitter in the main thread blocks all other event
> handling, w.r.t the DAG, and causes the event queue to back up.
> 3) This should also cover shared commits that happen in the DAG.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
