[
https://issues.apache.org/jira/browse/TEZ-728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13874208#comment-13874208
]
Siddharth Seth commented on TEZ-728:
------------------------------------
Comments
- OutptuCommitter.abortOutput is documented to be called on only non-successful
Outputs. We either need to change the Javadoc or add another method which will
be used to abort an already committed output.
The MR OutputCommitter abort will likely not know what to do for already
committed outputs.
- abortOutput may end up getting called twice - once in VertexImpl and once in
the DAG (abortAllOutputsOnFailure || vertex.getState() != VertexState.SUCCEEDED)
- I don't think this is possible, but verifying - calling abort on a committer
which hasn't yet been initialized.
- Does the Vertex state need to be changed to FAILED - in the cases where a
SUCCESSFUL vertex is being aborted. Or do we need a new state which indicates
SUCCESS_PENDING_COMMIT. From a monitoring point of view - we end up reporting
SUCCESS (before commit) and then potentially change it later. Returning SUCCESS
if the Vertex commit is pending, to me at least is misleading.
- (!successfulOutputsAlreadyCommitted && !committedOutputs.contains(committer))
- Didn't quite understand this bit. If I'm reading this correctly, for a
failed / partially committed DAG - if abortAllOutputsOnFailure - then try
committing outputs which haven't been committed ?
- There's a jira open to move OutputCommitter operations into a separate thread
to keep them away from blocking the main dispatcher. Hoping this will be easy
to refactor into that. It may not be required if this at the end of a DAG
execution.
> Semantics of output commit
> --------------------------
>
> Key: TEZ-728
> URL: https://issues.apache.org/jira/browse/TEZ-728
> Project: Apache Tez
> Issue Type: Task
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: TEZ-728.1.patch
>
>
> Currently, vertices commit outputs when they succeed. However, if the job
> fails then these outputs are not aborted.
> After speaking to Pig and Hive folks, both allow optional partial visibility
> semantics. So if there are 2 vertices writing output and one of them (A)
> passes and the other fails. Based on a user flag, Pig and Hive allow the
> partial output of vertex A to be visible or not. So we need to support
> 1) DAG fails - no output is visible
> 2) DAG fails - partial output is visible
> In order to support this, we could move output commit to DAG completion. If
> the DAG succeeds, commit will be called on all output committers. If the DAG
> fails, then abort will be called on all output committers. Optionally, if the
> DAG fails then commit will be called on all successful vertices and abort
> will be called on all failed vertices.
> This will also help the case when multiple vertices are writing to the same
> output (union store). The DAG can call commit once on that output and ensure
> correct commit semantics according to the commit API.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
