Thank you very much for the patch Dmitry. This seems to fix the issue when we run PIG from the command line, we no longer have to add this jar in the PIG_CLASSPATH, if we only register it it will work as expected.
However, after pointing to the right patched PIG jar, we're still having issues using it from PigServer, basically the same issue and same stack trace. Do you know of any other workarounds to use dynamic invokers with PigServer, or we should refrain from using dynamic invokers with PigServer for now? Thanks, Charles On Thu, Nov 17, 2011 at 12:17 AM, Dmitriy Ryaboy <[email protected]> wrote: > Can you try applying the patch below to your version of pig? > > diff --git src/org/apache/pig/builtin/Invoker.java > src/org/apache/pig/builtin/Invoker.java > index adfc04a..8150165 100644 > --- src/org/apache/pig/builtin/Invoker.java > +++ src/org/apache/pig/builtin/Invoker.java > @@ -31,6 +31,7 @@ import org.apache.commons.logging.LogFactory; > import org.apache.pig.backend.executionengine.ExecException; > import org.apache.pig.data.DataBag; > import org.apache.pig.data.Tuple; > +import org.apache.pig.impl.PigContext; > import org.apache.pig.impl.logicalLayer.FrontendException; > > import com.google.common.collect.Lists; > @@ -67,7 +68,14 @@ public class Invoker<T> { > throws ClassNotFoundException, FrontendException, > SecurityException, NoSuchMethodException { > String className = fullName.substring(0, > fullName.lastIndexOf('.')); > String methodName = > fullName.substring(fullName.lastIndexOf('.')+1); > - Class<?> klazz = Class.forName(className); > + Class<?> klazz; > + try { > + klazz = PigContext.resolveClassName(className); > + } catch (IOException e) { > + // the amusing part is that PigContext throws this to wrap > one of > + // the exceptions we declare! > + throw new FrontendException(e); > + } > String[] paramSpecs = "".equals(paramSpecsStr) ? new > String[0] : paramSpecsStr.split(" "); > isStatic_ = "static".equalsIgnoreCase(isStatic) || > "true".equals(isStatic); > paramClasses_ = new Class<?>[paramSpecs.length]; > > On Wed, Nov 16, 2011 at 8:21 PM, Stan Rosenberg > <[email protected]> wrote: > > Hi Dimitriy, > > > > The script does run if invoked from command line but only if we set > > PIG_CLASSPATH to point at the jar. > > > > stan > > On Nov 16, 2011 11:18 PM, "Dmitriy Ryaboy" <[email protected]> wrote: > > > >> Does the script run if you launch it from the pig command line instead > >> of via PigServer? > >> > >> On Wed, Nov 16, 2011 at 3:01 PM, Charles Menguy > >> <[email protected]> wrote: > >> > Hi, > >> > > >> > We're trying to run a PIG script using the PigServer API in Java, but > >> we're > >> > having a couple issues. > >> > > >> > It seems to work well in most cases, but in our case we need to use > Pig's > >> > dynamic invokers. Basically this looks something like the following: > >> > DEFINE isEmail InvokeForInt('com.company.pig.udf.Utils.isEmail', > >> 'String'); > >> > > >> > We register the script using PigServer like below: > >> > pigServer.registerScript(pigScript, params); > >> > > >> > The problem is that PigServer doesn't seem to like these dynamic > >> invokers. > >> > We get an error during the parsing of the script by PigServer. We've > >> > checked and the classpath is fine, and the UDF is defined correctly > with > >> > the right signature. Any help or advice would be very much > appreciated on > >> > this. I have attached the stack trace below for more details. > >> > > >> > Thanks, > >> > > >> > Charles > >> > > >> > ======================== > >> > > >> > Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR > >> 1000: > >> > Error during parsing. could not instantiate 'InvokeForInt' with > arguments > >> > '[com.company.pig.udf.Utils.isEmail, String]' > >> > at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1618) > >> > at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1562) > >> > at org.apache.pig.PigServer.registerQuery(PigServer.java:534) > >> > at > >> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:868) > >> > at > >> > > >> > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:388) > >> > at > >> > > >> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168) > >> > at org.apache.pig.PigServer.registerScript(PigServer.java:619) > >> > at org.apache.pig.PigServer.registerScript(PigServer.java:577) > >> > at > com.proclivitysystems.etl.job.PIGJobRunner.run(PIGJobRunner.java:96) > >> > ... 2 more > >> > Caused by: org.apache.pig.impl.logicalLayer.parser.ParseException: > could > >> > not instantiate 'InvokeForInt' with arguments > >> > '[com.company.pig.udf.Utils.isEmail, String]' > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.EvalFuncSpec(QueryParser.java:5465) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseEvalSpec(QueryParser.java:5291) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.UnaryExpr(QueryParser.java:5187) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.CastExpr(QueryParser.java:5133) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.MultiplicativeExpr(QueryParser.java:5042) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.AdditiveExpr(QueryParser.java:4968) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.InfixExpr(QueryParser.java:4934) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.PUnaryCond(QueryParser.java:2160) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.PAndCond(QueryParser.java:2096) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.POrCond(QueryParser.java:2040) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.PCond(QueryParser.java:2006) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.BinCond(QueryParser.java:5406) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseEvalSpec(QueryParser.java:5300) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.UnaryExpr(QueryParser.java:5187) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.CastExpr(QueryParser.java:5133) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.MultiplicativeExpr(QueryParser.java:5042) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.AdditiveExpr(QueryParser.java:4968) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.InfixExpr(QueryParser.java:4934) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedCommand(QueryParser.java:4117) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedBlock(QueryParser.java:4045) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:3433) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1464) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:1013) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:800) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63) > >> > at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1612) > >> > ... 10 more > >> > Caused by: java.lang.RuntimeException: could not instantiate > >> 'InvokeForInt' > >> > with arguments '[com.company.pig.udf.Utils.isEmail, String]' > >> > at > >> > > >> > org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:502) > >> > at > >> > > >> > org.apache.pig.impl.PigContext.instantiateFuncFromAlias(PigContext.java:529) > >> > at > >> > > >> > org.apache.pig.impl.logicalLayer.parser.QueryParser.EvalFuncSpec(QueryParser.java:5462) > >> > ... 35 more > >> > Caused by: java.lang.reflect.InvocationTargetException > >> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > >> > at > >> > > >> > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > >> > at > >> > > >> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > >> > at java.lang.reflect.Constructor.newInstance(Constructor.java:532) > >> > at > >> > > >> > org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:470) > >> > ... 37 more > >> > > >> > > >
