Hi Shawn, I think this is more of CDH packaging problem than Pig problem. I suspect this is related to Java versions of jython and other components.
You may look into https://docs.cloudera.com/download/attachments/8784980/CDH3b3_Installation_Guide.pdf?version=1&modificationDate=1300229469101 for more details. Thanks, Aniket On Fri, April 1, 2011 4:42 pm, Xiaomeng Wan wrote: > Hi Aniket, > > > Here is the stacktrace of the exception. > > > java.io.IOException: Deserialization error: could not instantiate > 'org.apache.pig.scripting.jython.JythonFunction' with arguments > '[/home/shawn/TESS/code/mypyudfs.py, isStopWord]' > at > org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.ja > va:55) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.s > etup(PigMapBase.java:151) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142) at > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:646) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322) at > org.apache.hadoop.mapred.Child$4.run(Child.java:251) > at java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation > .java:1115) > at org.apache.hadoop.mapred.Child.main(Child.java:245) Caused by: > java.lang.RuntimeException: could not instantiate > 'org.apache.pig.scripting.jython.JythonFunction' with arguments > '[/home/shawn/TESS/code/mypyudfs.py, isStopWord]' > at > org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:50 > 2) > at > org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOpe > rators.POUserFunc.instantiateFunc(POUserFunc.java:109) > at > org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOpe > rators.POUserFunc.readObject(POUserFunc.java:451) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java > :39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI > mpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849) at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at > java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) > at java.util.ArrayList.readObject(ArrayList.java:593) at > sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI > mpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849) at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at > java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) > at java.util.HashMap.readObject(HashMap.java:1030) at > sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI > mpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849) at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871) at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871) at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at > java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) > at java.util.ArrayList.readObject(ArrayList.java:593) at > sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI > mpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849) at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871) at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at > java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) > at java.util.ArrayList.readObject(ArrayList.java:593) at > sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI > mpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849) at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at > java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) > at java.util.HashMap.readObject(HashMap.java:1030) at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java > :39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI > mpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849) at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871) at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947) > at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871) at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at > java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) > at > org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.ja > va:53) > ... 9 more > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAc > cessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConst > ructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at > org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:47 > 0) > ... 87 more > Caused by: java.lang.IllegalStateException: Could not initialize: > /home/shawn/TESS/code/mypyudfs.py > at > org.apache.pig.scripting.jython.JythonFunction.<init>(JythonFunction.java > :86) > ... 92 more > 2011-04-01 14:31:40,977 INFO org.apache.hadoop.mapred.Task: Runnning > cleanup for the task > > Thanks! > > > Regards, > Shawn > > > On Fri, Apr 1, 2011 at 2:24 PM, Aniket Mokashi <[email protected]> > wrote: > >> Hi Shawn, >> >> >> Every time we throw an Exception with 'could not instantiate ..' error >> message, we also pass down the real exception instance, this might be >> able to point to the reason why we fail in this scenario. Can you provide >> details of your exception message from the log? >> >> The way this works is, when you register the myudf.py script we >> register all the function names inside script to pig and when we use >> these functions, we parse and construct them with JythonFunction >> constructor. >> >> Thanks, >> Aniket >> >> >> On Fri, April 1, 2011 12:06 pm, Xiaomeng Wan wrote: >> >>> Hi Aniket, >>> >>> >>> >>> We put both jython.jar and myudf.py in classpath and also register >>> jython.jar in our pig script. It worked well before the upgrading, >>> only failed after. >>> >>> Regards, >>> Shawn >>> >>> >>> >>> On Thu, Mar 31, 2011 at 4:38 PM, Aniket Mokashi >>> <[email protected]> >>> wrote: >>> >>> >>>> I think this might be because when you start in hadoop mode, your >>>> classpath configuration does not have jython.jar. Can you put that >>>> explicitly in classpath and check it out? >>>> >>>> Thanks, >>>> Aniket >>>> >>>> >>>> >>>> On Thu, March 31, 2011 6:07 pm, Xiaomeng Wan wrote: >>>> >>>> >>>>> Hi, >>>>> We recently updated our hadoop from CDH2 to CDH3b4, and had >>>>> problems using some old python udfs. Runing in local mode still >>>>> works, but in hadoop mode, it gives errors like "could not >>>>> instantiate 'org.apache.pig.scripting.jython.JythonFunction' with >>>>> arguments...". Anyone see similar error with python udf on this >>>>> hadoop distribution? We are using pig 0.8.0. Thanks! >>>>> >>>>> >>>>> >>>>> >>>>> Regards >>>>> Shawn >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>> >>> >> >> >> > >
