Hi Daniel Thank you for the update. I will comment that line when I run it.
Regards! Chen On Fri, Jun 6, 2014 at 1:19 PM, Daniel Dai <da...@hortonworks.com> wrote: > There are several fix for Order by in PIG-3846. I will try to commit > it soon and hope it solves the problem. Can your focus on > non-orderby/skewedjoin queries for now? > > Thanks, > Daniel > > On Fri, Jun 6, 2014 at 7:12 AM, Chen He <airb...@gmail.com> wrote: > > Hi Daniel > > > > I saw there is a line using "ordered by" in the script1-local.pig file. > > > > "ordered_uniq_frequency = ORDER filtered_uniq_frequency BY hour, score;" > > > > Regards! > > > > Chen > > > > > > On Fri, Jun 6, 2014 at 12:17 AM, Chen He <airb...@gmail.com> wrote: > > > >> Thank you for the reply, Daniel. I will check tomorrow. Actually, i just > >> use tutorial script1-local. > >> > >> Regards! > >> > >> Chen > >> On Jun 5, 2014 11:42 PM, "Daniel Dai" <da...@hortonworks.com> wrote: > >> > >>> Hi, Chen, > >>> Are you seeing this issue only when running a query containing "order > >>> by"? It seems a Pig bug on this side, I can take a look tomorrow. But > >>> in worst case, we can disable ObjectRegistry in local mode. > >>> > >>> Thanks, > >>> Daniel > >>> > >>> On Thu, Jun 5, 2014 at 9:14 PM, Chen He <airb...@gmail.com> wrote: > >>> > Hi Hitesh > >>> > > >>> > ObjectRegistry is used for broadcastShuffle. If a vertex has more > than > >>> one > >>> > task. Pig uses ObjectRegistry to save objects that can be used for > all > >>> > tasks in a vertex. The ObjectRegistryImpl's annotation is > singleton. I > >>> > suspect it is a race condition between constructor and caller. > >>> > > >>> > Regards! > >>> > > >>> > Chen > >>> > > >>> > > >>> > On Thu, Jun 5, 2014 at 8:49 PM, Hitesh Shah <hit...@apache.org> > wrote: > >>> > > >>> >> @Chen, I am not sure asking for the user code to change is an > option. > >>> We > >>> >> should address the underlying issue which is fixing ObjectRegistry > to > >>> work > >>> >> in local mode. > >>> >> > >>> >> thanks > >>> >> -- Hitesh > >>> >> > >>> >> > >>> >> On Thu, Jun 5, 2014 at 1:34 PM, Chen He <airb...@gmail.com> wrote: > >>> >> > >>> >> > Hi Hitesh > >>> >> > > >>> >> > In my code, I did not touch any ObjectRegistry related code. I > >>> suspect > >>> >> the > >>> >> > instantiation process of ObjectRegistry has problem so that it > >>> failed to > >>> >> > create an instance when it is called. > >>> >> > > >>> >> > I didn't know why PigProcessor collectSample vertex. But looks > like > >>> it is > >>> >> > not a must for TezLocalMode. > >>> >> > > >>> >> > Chen > >>> >> > > >>> >> > > >>> >> > On Thu, Jun 5, 2014 at 2:59 PM, Chen He <airb...@gmail.com> > wrote: > >>> >> > > >>> >> > > Hi Hitesh > >>> >> > > > >>> >> > > Yes, objectRegistry returns NPE. I saw it is a inject static > >>> variable. > >>> >> > > > >>> >> > > Regards! > >>> >> > > > >>> >> > > Chen > >>> >> > > > >>> >> > > > >>> >> > > On Thu, Jun 5, 2014 at 9:15 AM, Hitesh Shah <hit...@apache.org> > >>> wrote: > >>> >> > > > >>> >> > >> @Chen, this might be a bug in the tez local execution code. Can > >>> you > >>> >> > >> confirm > >>> >> > >> that ObjectRegistryFactory::getObjectRegistry returns a > non-null > >>> >> value? > >>> >> > >> > >>> >> > >> thanks > >>> >> > >> — Hitesh > >>> >> > >> > >>> >> > >> > >>> >> > >> On Wed, Jun 4, 2014 at 4:21 PM, Chen He <airb...@gmail.com> > >>> wrote: > >>> >> > >> > >>> >> > >> > I am using pig -x tez_local to test tez local mode and met > >>> following > >>> >> > >> > problems (use tutorial.jar) > >>> >> > >> > > >>> >> > >> > 2014-06-03 11:06:35,964 [pool-1-thread-1] INFO > >>> >> > >> > org.apache.hadoop.mapred.YarnTezDagChild - Running task, > >>> >> > >> > taskAttemptId=attempt_1401811588093_0000_1_05_000000_0 > >>> >> > >> > 2014-06-03 11:06:35,965 [pool-1-thread-1] ERROR > >>> >> > >> > > org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor - > >>> >> > >> > Encountered exception while processing: > >>> >> > >> > java.lang.NullPointerException > >>> >> > >> > at > >>> >> > >> > > >>> >> > >> > > >>> >> > >> > >>> >> > > >>> >> > >>> > org.apache.pig.backend.hadoop.executionengine.tez.ObjectCache.retrieve(ObjectCache.java:47) > >>> >> > >> > at > >>> >> > >> > > >>> >> > >> > > >>> >> > >> > >>> >> > > >>> >> > >>> > org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.collectSample(PigProcessor.java:307) > >>> >> > >> > at > >>> >> > >> > > >>> >> > >> > > >>> >> > >> > >>> >> > > >>> >> > >>> > org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.initializeInputs(PigProcessor.java:218) > >>> >> > >> > at > >>> >> > >> > > >>> >> > >> > > >>> >> > >> > >>> >> > > >>> >> > >>> > org.apache.pig.backend.hadoop.executionengine.tez.PigProcessor.run(PigProcessor.java:162) > >>> >> > >> > at > >>> >> > >> > > >>> >> > >> > > >>> >> > >> > >>> >> > > >>> >> > >>> > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:317) > >>> >> > >> > at > >>> >> > >> > > >>> >> > > >>> > org.apache.hadoop.mapred.YarnTezDagChild$3.run(YarnTezDagChild.java:602) > >>> >> > >> > at java.security.AccessController.doPrivileged(Native > >>> >> Method) > >>> >> > >> > at javax.security.auth.Subject.doAs(Subject.java:415) > >>> >> > >> > at > >>> >> > >> > > >>> >> > >> > > >>> >> > >> > >>> >> > > >>> >> > >>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) > >>> >> > >> > at > >>> >> > >> > > >>> >> > >> > >>> >> > > >>> >> > >>> > org.apache.hadoop.mapred.YarnTezDagChild.pollTask(YarnTezDagChild.java:591) > >>> >> > >> > at > >>> >> > >> > > >>> >> > >> > >>> >> > > >>> >> > >>> > org.apache.hadoop.mapred.YarnTezDagChild.runTask(YarnTezDagChild.java:332) > >>> >> > >> > at > >>> >> > >> > > >>> >> > >> > > >>> >> > >> > >>> >> > > >>> >> > >>> > org.apache.tez.dag.app.launcher.LocalContainerLauncher$1.run(LocalContainerLauncher.java:250) > >>> >> > >> > at java.lang.Thread.run(Thread.java:744) > >>> >> > >> > at > >>> >> > >> > > >>> >> > >> > > >>> >> > >> > >>> >> > > >>> >> > >>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > >>> >> > >> > at > >>> >> > >> > > >>> >> > >> > > >>> >> > >> > >>> >> > > >>> >> > >>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > >>> >> > >> > at java.lang.Thread.run(Thread.java:744) > >>> >> > >> > > >>> >> > >> > code checking implies that the ObjectCache is trying to get > >>> >> something > >>> >> > >> from > >>> >> > >> > a Map but it is not there. > >>> >> > >> > But what is the purpose of using ObjectCache in PIG on TEZ? > >>> >> > >> > Why pig needs sampleVertex in the PigProcessor? > >>> >> > >> > > >>> >> > >> > Any reply will be appreciated. > >>> >> > >> > > >>> >> > >> > Regards! > >>> >> > >> > > >>> >> > >> > Chen > >>> >> > >> > > >>> >> > >> > >>> >> > > > >>> >> > > > >>> >> > > >>> >> > >>> > >>> -- > >>> CONFIDENTIALITY NOTICE > >>> NOTICE: This message is intended for the use of the individual or > entity > >>> to > >>> which it is addressed and may contain information that is confidential, > >>> privileged and exempt from disclosure under applicable law. If the > reader > >>> of this message is not the intended recipient, you are hereby notified > >>> that > >>> any printing, copying, dissemination, distribution, disclosure or > >>> forwarding of this communication is strictly prohibited. If you have > >>> received this communication in error, please contact the sender > >>> immediately > >>> and delete it from your system. Thank You. > >>> > >> > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >