[ 
https://issues.apache.org/jira/browse/TEZ-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612432#comment-15612432
 ] 

Jonathan Eagles commented on TEZ-3271:
--------------------------------------

bq. The function could do explicit throws for both of the exceptions. On the 
VertexImpl, a generic Exception catch would make sense to ensure that we dont 
hit a dispatcher error.
The above it true, but one important this I didn't mention above is that 
edgeManager.getNumDestinationConsumerTasks throws of type Exception. Throwing 
TezException, IOException, and Exception seems redundant. I will catch the 
Exception and rethrow TezException to explicitly give a throws TezException, 
IOException signature.
bq. Relies on TaskImpl's impl of addAndScheduleAttempt() but you could just use 
size of the map to decipher the last attempt.
Changed the logic to always ensure the highest attempt is chosen.

bq. "Assert.assertEquals(2, v6.numSuccessSourceAttemptCompletions);" - shoudnt 
this get 4 completions given that v6 expects task attempt completions for both 
tasks in v4 and v5?
Likewise for "testFailuresMaxPercentExceededSourceTaskAttemptCompletionEvents" 
- I would assume this will get 2 for tasks of v5?
Will address this
bq. Sorry for the last minute review comment on testVertexFailuresMaxPercent - 
should we be using a 2-vertex DAG with the first vertex having a threshold and 
verifying that the second vertex ( using a shuffle edge ) completes 
successfully ?
Think the main thrust is that the shuffle is missing between these vertices and 
should be added. Will do that.

> Provide mapreduce failures.maxpercent equivalent
> ------------------------------------------------
>
>                 Key: TEZ-3271
>                 URL: https://issues.apache.org/jira/browse/TEZ-3271
>             Project: Apache Tez
>          Issue Type: New Feature
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>         Attachments: Succeeded with Failures.png, TEZ-3271.1.patch, 
> TEZ-3271.2.patch, TEZ-3271.3.patch, TEZ-3271.4.patch, TEZ-3271.5.patch, 
> TEZ-3271.6.patch, TEZ-3271.7.patch, TEZ-3271.8.patch, TEZ-3271.9.patch
>
>
> There is a certain category of work that need not have 100% of tasks succeed 
> to cause the work to be considered a success. To meet that end, I propose we 
> provide a tez equivalent of mapreduce.map.failures.maxpercent and 
> mapreduce.reduce.failures.maxpercent. In this way a vertex will be considered 
> a success if the number of failures is below a configured threshold.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to