zjffdu
Tue, 03 Nov 2009 17:54:53 -0800
Yes, Pig is not thread safe now. There's a jira item for this issue. https://issues.apache.org/jira/browse/PIG-240
I attach a patch for this item. I am not sure whether this patch resolve this issue completely. But at least it works for me. I have ready used this patch for several months. Maybe you can try this patch. Jeff Zhang -----Original Message----- From: Ashutosh Chauhan [mailto:ashutosh.chau...@gmail.com] Sent: 2009年11月3日 10:05 To: pig-user@hadoop.apache.org Subject: Re: Unable to find clone for op Const 16-169 Hi Vincent, AFAIK PigServer is not thread-safe. So, it can't support queries running concurrently in multiple threads. As a result, you may end up in race conditions as the one Bennie encountered. I guess, there is a jira open to make PigServer thread-safe. You may want to comment there with your use-case. Thanks, Ashutosh On Tue, Nov 3, 2009 at 12:57, Vincent Barat <vincent.ba...@ubikod.com>wrote: > I have the exact same problem, and yes, I run my queries from Java > concurrently. The issue is easier to reproduce in local mode than in MR mode > (I never saw it in MR mode actually). > > I've tried to add calls to pigServer.shutdown() to see if this can help, > but the issue remains. > > Ashutosh Chauhan a écrit : > > Hi Bennie, >> >> Are you using Pig Java API to run your queries? If so, are you trying to >> run >> queries concurrently in multiple threads ? >> >> Ashutosh >> >> On Tue, Nov 3, 2009 at 11:00, Bennie Schut <bsc...@ebuddy.com> wrote: >> >> From time to time I receive this error: >>> >>> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable >>> to store alias uqusers11 >>> at org.apache.pig.PigServer.store(PigServer.java:536) >>> at org.apache.pig.PigServer.store(PigServer.java:493) >>> at >>> >>> >>> com.ebuddy.dwhmapreduce.pig.chatsessions.UniqueUsers.run(UniqueUsers.java:76 ) >>> at >>> >>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja va:886) >>> at >>> >>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 08) >>> at java.lang.Thread.run(Thread.java:619) >>> Caused by: java.lang.RuntimeException: Unable to find clone for op Const >>> 16-169( 1 ) >>> at >>> org.apache.pig.impl.logicalLayer.LogicalPlan.clone(LogicalPlan.java:139) >>> at >>> >>> >>> org.apache.pig.impl.logicalLayer.LogicalPlanCloneHelper.<init>(LogicalPlanCl oneHelper.java:63) >>> at >>> >>> >>> org.apache.pig.impl.logicalLayer.LogicalPlanCloner.getClonedPlan(LogicalPlan Cloner.java:45) >>> at >>> >>> >>> org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParse r.java:3044) >>> at >>> >>> >>> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.jav a:1328) >>> at >>> >>> >>> org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:90 7) >>> at >>> >>> >>> org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:6 96) >>> at >>> >>> >>> org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder .java:63) >>> at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1034) >>> at org.apache.pig.PigServer$Graph.clone(PigServer.java:1077) >>> at org.apache.pig.PigServer.clonePlan(PigServer.java:389) >>> at org.apache.pig.PigServer.compileLp(PigServer.java:804) >>> at org.apache.pig.PigServer.compileLp(PigServer.java:791) >>> at org.apache.pig.PigServer.store(PigServer.java:509) >>> ... 5 more >>> >>> However running it a 2nd time it runs just fine. It's a little hard to >>> reproduce. I received this on a line like this: >>> uqusers11 = FOREACH uqusers10 GENERATE user_id, protocol, logincldr_id, >>> logintime_id; >>> >>> but also sometimes on this: >>> uqusers6 = FOREACH uqusers5 GENERATE flatten($0), MIN(uqusers4.login) as >>> mindate; >>> >>> so for now it mostly seems to happen on foreach statements (I'm >>> currently using pig trunk and hadoop 20.1) >>> >>> Can anyone point me in the right direction on what to look at when >>> looking at errors like this. >>> Thanks, >>> Bennie. >>> >>> >>