Hi Hitesh ObjectRegistry is used for broadcastShuffle. If a vertex has more than one task. Pig uses ObjectRegistry to save objects that can be used for all tasks in a vertex. The ObjectRegistryImpl's annotation is singleton. I suspect it is a race condition between constructor and caller.
Regards! Chen On Thu, Jun 5, 2014 at 8:49 PM, Hitesh Shah <hit...@apache.org> wrote: > @Chen, I am not sure asking for the user code to change is an option. We > should address the underlying issue which is fixing ObjectRegistry to work > in local mode. > > thanks > -- Hitesh > > > On Thu, Jun 5, 2014 at 1:34 PM, Chen He <airb...@gmail.com> wrote: > > > Hi Hitesh > > > > In my code, I did not touch any ObjectRegistry related code. I suspect > the > > instantiation process of ObjectRegistry has problem so that it failed to > > create an instance when it is called. > > > > I didn't know why PigProcessor collectSample vertex. But looks like it is > > not a must for TezLocalMode. > > > > Chen > > > > > > On Thu, Jun 5, 2014 at 2:59 PM, Chen He <airb...@gmail.com> wrote: > > > > > Hi Hitesh > > > > > > Yes, objectRegistry returns NPE. I saw it is a inject static variable. > > > > > > Regards! > > > > > > Chen > > > > > > > > > On Thu, Jun 5, 2014 at 9:15 AM, Hitesh Shah <hit...@apache.org> wrote: > > > > > >> @Chen, this might be a bug in the tez local execution code. Can you > > >> confirm > > >> that ObjectRegistryFactory::getObjectRegistry returns a non-null > value? > > >> > > >> thanks > > >> — Hitesh > > >> > > >> > > >> On Wed, Jun 4, 2014 at 4:21 PM, Chen He <airb...@gmail.com> wrote: > > >> > > >> > I am using pig -x tez_local to test tez local mode and met following > > >> > problems (use tutorial.jar) > > >> > > > >> > 2014-06-03 11:06:35,964 [pool-1-thread-1] INFO > > >> > org.apache.hadoop.mapred.YarnTezDagChild - Running task, > > >> > taskAttemptId=attempt_1401811588093_0000_1_05_000000_0 > > >> > 2014-06-03 11:06:35,965 [pool-1-thread-1] ERROR > > >> > org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor - > > >> > Encountered exception while processing: > > >> > java.lang.NullPointerException > > >> > at > > >> > > > >> > > > >> > > > org.apache.pig.backend.hadoop.executionengine.tez.ObjectCache.retrieve(ObjectCache.java:47) > > >> > at > > >> > > > >> > > > >> > > > org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.collectSample(PigProcessor.java:307) > > >> > at > > >> > > > >> > > > >> > > > org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.initializeInputs(PigProcessor.java:218) > > >> > at > > >> > > > >> > > > >> > > > org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.run(PigProcessor.java:162) > > >> > at > > >> > > > >> > > > >> > > > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:317) > > >> > at > > >> > > > org.apache.hadoop.mapred.YarnTezDagChild$3.run(YarnTezDagChild.java:602) > > >> > at java.security.AccessController.doPrivileged(Native > Method) > > >> > at javax.security.auth.Subject.doAs(Subject.java:415) > > >> > at > > >> > > > >> > > > >> > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) > > >> > at > > >> > > > >> > > > org.apache.hadoop.mapred.YarnTezDagChild.pollTask(YarnTezDagChild.java:591) > > >> > at > > >> > > > >> > > > org.apache.hadoop.mapred.YarnTezDagChild.runTask(YarnTezDagChild.java:332) > > >> > at > > >> > > > >> > > > >> > > > org.apache.tez.dag.app.launcher.LocalContainerLauncher$1.run(LocalContainerLauncher.java:250) > > >> > at java.lang.Thread.run(Thread.java:744) > > >> > at > > >> > > > >> > > > >> > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > >> > at > > >> > > > >> > > > >> > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > >> > at java.lang.Thread.run(Thread.java:744) > > >> > > > >> > code checking implies that the ObjectCache is trying to get > something > > >> from > > >> > a Map but it is not there. > > >> > But what is the purpose of using ObjectCache in PIG on TEZ? > > >> > Why pig needs sampleVertex in the PigProcessor? > > >> > > > >> > Any reply will be appreciated. > > >> > > > >> > Regards! > > >> > > > >> > Chen > > >> > > > >> > > > > > > > > >