[
https://issues.apache.org/jira/browse/TEZ-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13977734#comment-13977734
]
Mohammad Kamrul Islam edited comment on TEZ-1062 at 4/23/14 2:44 AM:
---------------------------------------------------------------------
Thanks again [~bikassaha] for the review.
Need some clarifications:
bq. Secondly, we should call getContext().canCommit() only once ...
Struggling to find the correct ordering of two checks. How about this?
{noformat}
+ while (!getContext().canCommit()) {
+ Thread.sleep(100);
+ }
+ for (LogicalOutput output : outputs.values()) {
+ if ((output instanceof MROutput) && (((MROutput)
output).isCommitRequired())) {
+ ((MROutput) output).commit();
+ }
+ }
{noformat}
Related question:
bq. If any output fails to commit then we should abort all the outputs that
needed commit.
I think already committed outputs can't be aborted.
How to determine some output fails? when output.commit() throw exception?
By abort, do you mean to break the loop.
EDITED:
bq. I think it makes sense to move this code into the postOp of SimpleProcessor.
MROutput is not accessible from Tez-runitme-library. If I include the
tez-mapreduce (where MROutput resides) in the pom.xml of tez-runtime-library
module, it creates a circular dependency . any thoughts?
was (Author: kamrul):
Thanks again [~bikassaha] for the review.
Need some clarifications:
bq. Secondly, we should call getContext().canCommit() only once ...
Struggling to find the correct ordering of two checks. How about this?
{noformat}
+ while (!getContext().canCommit()) {
+ Thread.sleep(100);
+ }
+ for (LogicalOutput output : outputs.values()) {
+ if ((output instanceof MROutput) && (((MROutput)
output).isCommitRequired())) {
+ ((MROutput) output).commit();
+ }
+ }
{noformat}
Related question:
bq. If any output fails to commit then we should abort all the outputs that
needed commit.
I think already committed outputs can't be aborted.
How to determine some output fails? when output.commit() throw exception?
By abort, do you mean to break the loop.
> Create SimpleProcessor for processors that only need to implement the run
> method
> --------------------------------------------------------------------------------
>
> Key: TEZ-1062
> URL: https://issues.apache.org/jira/browse/TEZ-1062
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Bikas Saha
> Assignee: Mohammad Kamrul Islam
> Attachments: TEZ-1062.1.patch
>
>
> The SimpleProcessor could take care of all things like starting input,
> committing outputs. It would handle no events, since simple processors dont
> need to handle inputs. Thus the user would only need to implement their
> custom task logic in a new execute() method.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
