Hi Marek, I wish I could offer a solution. I tried all sorts of permutations and to no avail. I retreated back to our base install of Pig 0.8 where Jython UDFs work for me. Here's a list of things I've tried:
1) Setting JYTHON_HOME to the Jython path 2) Setting PIG_CLASSPATH to <Jython path>/jython.jar 3) Copying jython.jar to <Pig path>/lib 4) Using Jython 2.5.0 and 2.5.2 I can't afford to burn any more time right now looking into this. I'm hoping the problem can be uncovered. Jython UDF support is critical for my productivity. Chris On Tue, Jun 19, 2012 at 2:40 PM, Marek Dudek <[email protected]>wrote: > Chris, > > I can confirm the problem; just run into it myself :( > > In my case I was able to run my code succesfully on MapR cluster (hadoop > 0.20.2), with Pig 0.9.1 and Jython 2.5.2. > Now, I am getting same "Deserialization error" after upgrading to Pig > 0.10.0 > (see log snippets below). > > Any ideas/luck so far? Is there anything I can try/test to help? > > Marek > > > > ----------------- > 2012-06-19 16:24:54,470 [main] INFO org.apache.pig.Main - Apache Pig > version 0.10.0 (r1328203) compiled Apr 20 2012, 05:54:07 > 2012-06-19 16:24:54,470 [main] INFO org.apache.pig.Main - Logging error > messages to: /transfer/marek/poc123-code/pig_1340141094466.log > (...) > 2012-06-19 16:26:27,484 [main] ERROR > org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed! > 2012-06-19 16:26:27,488 [main] INFO > org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: > > HadoopVersion PigVersion UserId StartedAt FinishedAt > Features > 0.20.2-dev 0.10.0 marek 2012-06-19 16:25:13 2012-06-19 > 16:26:27 FILTER > > Failed! > (...) > > 2012-06-19 16:26:27,488 [main] INFO > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLaunch > er - Failed! > 2012-06-19 16:26:27,544 [main] ERROR org.apache.pig.tools.grunt.GruntParser > - ERROR 2997: Unable to recreate exception from backed error: > java.io.IOException: Deserialization error: could not instantiate > 'org.apache.pig.scripting.jython.JythonFunction' with arguments > '[/transfer/marek/poc123-code/poc123-udf.py, parse_time_local]' > at > > org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java: > 55) > at > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBa > se.setup(PigGenericMapBase.java:177) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:654) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) > at org.apache.hadoop.mapred.Child$4.run(Child.java:270) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja > va:1109) > at org.apache.hadoop.mapred.Child.main(Child.java:264) > Caused by: java.lang.RuntimeException: could not instantiate > 'org.apache.pig.scripting.jython.JythonFunction' with arguments > '[/transfer/marek/poc123-code/poc123-udf.py, parse_time_local]' > at > org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:553) > at > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperat > ors.POUserFunc.instantiateFunc(POUserFunc.java:107) > at > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperat > ors.POUserFunc.readObject(POUserFunc.java:439) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39 > ) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl > .java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969) > at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) > at java.util.ArrayList.readObject(ArrayList.java:593) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl > .java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969) > at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) > at java.util.HashMap.readObject(HashMap.java:1030) > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl > .java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969) > at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946) > at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946) > at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) > at java.util.ArrayList.readObject(ArrayList.java:593) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl > .java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969) > at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946) > at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) > at java.util.HashMap.readObject(HashMap.java:1029) > at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl > .java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969) > at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946) > at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) > at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946) > at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870) > at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) > at > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) > at > > org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java: > 53) > ... 9 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces > sorImpl.java:39) > at > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc > torAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:521) > ... 77 more > Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: > Could not initialize: /transfer/marek/poc123-code/poc123-udf.py > at > > org.apache.pig.scripting.jython.JythonFunction.<init>(JythonFunction.java:87 > ) > ... 82 more > Caused by: java.lang.IllegalStateException: Could not initialize > interpreter > (from file system or classpath) with > /transfer/marek/poc123-code/poc123-udf.py > at > > org.apache.pig.scripting.ScriptEngine.getScriptAsStream(ScriptEngine.java:14 > 5) > at > > org.apache.pig.scripting.jython.JythonScriptEngine$Interpreter.init(JythonSc > riptEngine.java:153) > at > > org.apache.pig.scripting.jython.JythonScriptEngine.getFunction(JythonScriptE > ngine.java:328) > at > > org.apache.pig.scripting.jython.JythonFunction.<init>(JythonFunction.java:52 > ) > ... 82 more > >
