[
https://issues.apache.org/jira/browse/PIG-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988278#action_12988278
]
Santhosh Srinivasan commented on PIG-1829:
------------------------------------------
+1 for both - investigate further to check for options to retrieve the job from
history and push for standardized access mechanisms.
For now, can we document this behavior - the only concern I have is around the
duration for which JT retains the job in memory. For e.g., if more than half
the time we are unable to retrieve statistics, it will be a bit frustrating for
the users.
> "0" value seen in PigStat's map/reduce runtime, even when the job is
> successful
> -------------------------------------------------------------------------------
>
> Key: PIG-1829
> URL: https://issues.apache.org/jira/browse/PIG-1829
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.8.0
> Reporter: Thejas M Nair
> Fix For: 0.9.0
>
>
> Pig runtime calls JobClient.getMapTaskReports(jobId) and
> JobClient.getReduceTaskReports(jobId) to get statistics about numbers of
> maps/reducers, as well as max/min/avg time of these tasks. But from time to
> time, these calls return empty lists. When that happens pig is reports 0
> values for the stats.
> The jobtracker keeps the stats information only for a limited duration based
> on the configuration parameters mapred.jobtracker.completeuserjobs.maximum
> and mapred.job.tracker.retiredjobs.cache.size. Since pig collects the stats
> after jobs have finished running, it is possible that the stats for the
> initial jobs are no longer available. To have better chances of getting the
> stats, it should be collected as soon as the job is over.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
