Null Pointer Exceptions in the mappers leading to lot of retries
----------------------------------------------------------------
Key: PIG-445
URL: https://issues.apache.org/jira/browse/PIG-445
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: types_branch
Reporter: Shravan Matthur Narayanamurthy
Assignee: Shravan Matthur Narayanamurthy
Even with successfully completed jobs, usually with a large data set, we see
that there are NPE produced in the mappers which lead to task failure. However,
this problem goes away on retries. The problem occurs at places where we access
the reporter to report progress.
>From the analysis, this should happen with jobs that use combiner. The
>combiner is called whenever the mapper outputs a buffer full of data. So the
>combiner is called multiple times in between a map task. In the Combiner.close
>method we currently set the reporter to null as it was assumed that combiner
>is called only after the entire output of map is produced.
The fix is to not set the reporter to null in the Combiner.close() method
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
