Great, good to know about the command-line issue, I'll make a jira and apply that patch.
There shouldn't be any issue using dynamic invokers from PigServer, I suspect an environment issue (but I can't even posit what that issue would be at the moment). Any chance you can provide a reproducible test case and file a Jira? I'm going to be offline for the next couple of days, but I'll try to fix this asap if we can figure out what the issue is. D On Thu, Nov 17, 2011 at 11:03 AM, Charles Menguy <[email protected]> wrote: > Thank you very much for the patch Dmitry. > > This seems to fix the issue when we run PIG from the command line, we no > longer have to add this jar in the PIG_CLASSPATH, if we only register it it > will work as expected. > > However, after pointing to the right patched PIG jar, we're still having > issues using it from PigServer, basically the same issue and same stack > trace. > > Do you know of any other workarounds to use dynamic invokers with > PigServer, or we should refrain from using dynamic invokers with PigServer > for now? > > Thanks, > > Charles > > On Thu, Nov 17, 2011 at 12:17 AM, Dmitriy Ryaboy <[email protected]> wrote: > >> Can you try applying the patch below to your version of pig? >> >> diff --git src/org/apache/pig/builtin/Invoker.java >> src/org/apache/pig/builtin/Invoker.java >> index adfc04a..8150165 100644 >> --- src/org/apache/pig/builtin/Invoker.java >> +++ src/org/apache/pig/builtin/Invoker.java >> @@ -31,6 +31,7 @@ import org.apache.commons.logging.LogFactory; >> import org.apache.pig.backend.executionengine.ExecException; >> import org.apache.pig.data.DataBag; >> import org.apache.pig.data.Tuple; >> +import org.apache.pig.impl.PigContext; >> import org.apache.pig.impl.logicalLayer.FrontendException; >> >> import com.google.common.collect.Lists; >> @@ -67,7 +68,14 @@ public class Invoker<T> { >> throws ClassNotFoundException, FrontendException, >> SecurityException, NoSuchMethodException { >> String className = fullName.substring(0, >> fullName.lastIndexOf('.')); >> String methodName = >> fullName.substring(fullName.lastIndexOf('.')+1); >> - Class<?> klazz = Class.forName(className); >> + Class<?> klazz; >> + try { >> + klazz = PigContext.resolveClassName(className); >> + } catch (IOException e) { >> + // the amusing part is that PigContext throws this to wrap >> one of >> + // the exceptions we declare! >> + throw new FrontendException(e); >> + } >> String[] paramSpecs = "".equals(paramSpecsStr) ? new >> String[0] : paramSpecsStr.split(" "); >> isStatic_ = "static".equalsIgnoreCase(isStatic) || >> "true".equals(isStatic); >> paramClasses_ = new Class<?>[paramSpecs.length]; >> >> On Wed, Nov 16, 2011 at 8:21 PM, Stan Rosenberg >> <[email protected]> wrote: >> > Hi Dimitriy, >> > >> > The script does run if invoked from command line but only if we set >> > PIG_CLASSPATH to point at the jar. >> > >> > stan >> > On Nov 16, 2011 11:18 PM, "Dmitriy Ryaboy" <[email protected]> wrote: >> > >> >> Does the script run if you launch it from the pig command line instead >> >> of via PigServer? >> >> >> >> On Wed, Nov 16, 2011 at 3:01 PM, Charles Menguy >> >> <[email protected]> wrote: >> >> > Hi, >> >> > >> >> > We're trying to run a PIG script using the PigServer API in Java, but >> >> we're >> >> > having a couple issues. >> >> > >> >> > It seems to work well in most cases, but in our case we need to use >> Pig's >> >> > dynamic invokers. Basically this looks something like the following: >> >> > DEFINE isEmail InvokeForInt('com.company.pig.udf.Utils.isEmail', >> >> 'String'); >> >> > >> >> > We register the script using PigServer like below: >> >> > pigServer.registerScript(pigScript, params); >> >> > >> >> > The problem is that PigServer doesn't seem to like these dynamic >> >> invokers. >> >> > We get an error during the parsing of the script by PigServer. We've >> >> > checked and the classpath is fine, and the UDF is defined correctly >> with >> >> > the right signature. Any help or advice would be very much >> appreciated on >> >> > this. I have attached the stack trace below for more details. >> >> > >> >> > Thanks, >> >> > >> >> > Charles >> >> > >> >> > ======================== >> >> > >> >> > Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR >> >> 1000: >> >> > Error during parsing. could not instantiate 'InvokeForInt' with >> arguments >> >> > '[com.company.pig.udf.Utils.isEmail, String]' >> >> > at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1618) >> >> > at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1562) >> >> > at org.apache.pig.PigServer.registerQuery(PigServer.java:534) >> >> > at >> >> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:868) >> >> > at >> >> > >> >> >> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:388) >> >> > at >> >> > >> >> >> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168) >> >> > at org.apache.pig.PigServer.registerScript(PigServer.java:619) >> >> > at org.apache.pig.PigServer.registerScript(PigServer.java:577) >> >> > at >> com.proclivitysystems.etl.job.PIGJobRunner.run(PIGJobRunner.java:96) >> >> > ... 2 more >> >> > Caused by: org.apache.pig.impl.logicalLayer.parser.ParseException: >> could >> >> > not instantiate 'InvokeForInt' with arguments >> >> > '[com.company.pig.udf.Utils.isEmail, String]' >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.EvalFuncSpec(QueryParser.java:5465) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseEvalSpec(QueryParser.java:5291) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.UnaryExpr(QueryParser.java:5187) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.CastExpr(QueryParser.java:5133) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.MultiplicativeExpr(QueryParser.java:5042) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.AdditiveExpr(QueryParser.java:4968) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.InfixExpr(QueryParser.java:4934) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.PUnaryCond(QueryParser.java:2160) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.PAndCond(QueryParser.java:2096) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.POrCond(QueryParser.java:2040) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.PCond(QueryParser.java:2006) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.BinCond(QueryParser.java:5406) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseEvalSpec(QueryParser.java:5300) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.UnaryExpr(QueryParser.java:5187) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.CastExpr(QueryParser.java:5133) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.MultiplicativeExpr(QueryParser.java:5042) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.AdditiveExpr(QueryParser.java:4968) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.InfixExpr(QueryParser.java:4934) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedCommand(QueryParser.java:4117) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.NestedBlock(QueryParser.java:4045) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.ForEachClause(QueryParser.java:3433) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1464) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:1013) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:800) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63) >> >> > at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1612) >> >> > ... 10 more >> >> > Caused by: java.lang.RuntimeException: could not instantiate >> >> 'InvokeForInt' >> >> > with arguments '[com.company.pig.udf.Utils.isEmail, String]' >> >> > at >> >> > >> >> >> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:502) >> >> > at >> >> > >> >> >> org.apache.pig.impl.PigContext.instantiateFuncFromAlias(PigContext.java:529) >> >> > at >> >> > >> >> >> org.apache.pig.impl.logicalLayer.parser.QueryParser.EvalFuncSpec(QueryParser.java:5462) >> >> > ... 35 more >> >> > Caused by: java.lang.reflect.InvocationTargetException >> >> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >> Method) >> >> > at >> >> > >> >> >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >> >> > at >> >> > >> >> >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >> >> > at java.lang.reflect.Constructor.newInstance(Constructor.java:532) >> >> > at >> >> > >> >> >> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:470) >> >> > ... 37 more >> >> > >> >> >> > >> >
