[
https://issues.apache.org/jira/browse/PIG-2152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089226#comment-13089226
]
Josh Wills commented on PIG-2152:
---------------------------------
Hey, I hit this issue too, using CDH3u1. I'm wondering if you're hitting it
when a) you're running over lots of data and b) using combiners. It looks to me
that the issue is the setRep(null) call in the cleanup() method in PigCombiner,
which is impacting the Map tasks b/c PhysicalOperator only uses a single global
PigReporter and the Map tasks and the Combiner tasks run in the same JVM.
Could we reproduce this by messing with the settings to force spills more often
than just once?
> Null pointer exception while reporting progress
> -----------------------------------------------
>
> Key: PIG-2152
> URL: https://issues.apache.org/jira/browse/PIG-2152
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Olga Natkovich
> Fix For: 0.9.1
>
> Attachments: null_pointer_traces (copy)
>
>
> We have observed the following issues with code built from Pig 0.9 branch. We
> have not seen this with earlier versions; however, since this happens once in
> a while and is not reproducible at will it is not clear whether the issue is
> specific to 0.9 or not.
> Here is the stack:
> java.lang.NullPointerException at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.ProgressableReporter.progress(ProgressableReporter.java:37)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:399)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:284)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:290)
> at
> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNext(POLocalRearrange.java:256)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:261)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:256)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:58)
> at
> org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at
> org.apache.hadoop.mapred.Child$4.run(Child.java:261) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:396) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at
> org.apache.hadoop.mapred.Child.main(Child.java:255)
> Note that the code in progress function looks as follows:
> public void progress() {
> if(rep!=null)
> rep.progress();
> }
> This points to some sort of synchronization issue
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
