[
https://issues.apache.org/jira/browse/TEZ-808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957564#comment-14957564
]
Rohini Palaniswamy commented on TEZ-808:
----------------------------------------
Thought needs to be given on the APIs as well and how to have progress
reported. Progress reporting on mapreduce tasks is also very useful for users
to see how much data each of the tasks have processed and gives an idea on when
they will finish. One of the users asked a while ago where is that on Tez UI.
The current one available in Tez is ProcessorContext.setProgress(float).
Mapreduce had progress() and getProgress(). The getProgress() was implemented
by RecordReader's and fetched from there and only they had an idea of how much
of the input had been processed. All other places - Processor code
(Mapper/Reducer function) and framework code called the Reporter.progress() to
indicate progress is being made. So currently in Pig we can call setProgress,
but we can only keep calling it with 0 when processing is going on and then 1
at the end. It can probably do some kind of aggregation of the different input
progress to report the processor progress, but only MRInput has getProgress()
API and none of the others have. It is not part of LogicalInput.
> Handle task attempts that are not making progress
> -------------------------------------------------
>
> Key: TEZ-808
> URL: https://issues.apache.org/jira/browse/TEZ-808
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Bikas Saha
>
> If a task attempt is not making progress then it may cause the job to hang.
> We may want to kill and restart the attempt. With speculation support and
> free resources we may want to run another version in parallel.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
