[
https://issues.apache.org/jira/browse/TEZ-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610181#comment-15610181
]
Hitesh Shah commented on TEZ-3271:
----------------------------------
bq. This function throws TezException and IOException. Let me what the right
thing to do in this particular situation.
The function could do explicit throws for both of the exceptions. On the
VertexImpl, a generic Exception catch would make sense to ensure that we dont
hit a dispatcher error.
{code}
Iterator<TezTaskAttemptID> attempts =
task.getAttempts().keySet().iterator();
2150 while (attempts.hasNext()) {
2151 attempt = attempts.next();
2152 }
{code}
- Relies on TaskImpl's impl of addAndScheduleAttempt() but you could just use
size of the map to decipher the last attempt.
TestVertexImpl changes:
- createInvalidDAGPlan not needed?
- "Assert.assertEquals(2, v6.numSuccessSourceAttemptCompletions);" - shoudnt
this get 4 completions given that v6 expects task attempt completions for both
tasks in v4 and v5?
- Likewise for
"testFailuresMaxPercentExceededSourceTaskAttemptCompletionEvents" - I would
assume this will get 2 for tasks of v5?
Sorry for the last minute review comment on testVertexFailuresMaxPercent -
should we be using a 2-vertex DAG with the first vertex having a threshold and
verifying that the second vertex ( using a shuffle edge ) completes
successfully ?
> Provide mapreduce failures.maxpercent equivalent
> ------------------------------------------------
>
> Key: TEZ-3271
> URL: https://issues.apache.org/jira/browse/TEZ-3271
> Project: Apache Tez
> Issue Type: New Feature
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Attachments: Succeeded with Failures.png, TEZ-3271.1.patch,
> TEZ-3271.2.patch, TEZ-3271.3.patch, TEZ-3271.4.patch, TEZ-3271.5.patch,
> TEZ-3271.6.patch, TEZ-3271.7.patch, TEZ-3271.8.patch, TEZ-3271.9.patch
>
>
> There is a certain category of work that need not have 100% of tasks succeed
> to cause the work to be considered a success. To meet that end, I propose we
> provide a tez equivalent of mapreduce.map.failures.maxpercent and
> mapreduce.reduce.failures.maxpercent. In this way a vertex will be considered
> a success if the number of failures is below a configured threshold.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
