I think I saw this same behavior the other day trying to work the examples in the jython udf docs.
On Fri, Feb 10, 2012 at 4:13 PM, Chris Diehl <[email protected]> wrote: > Hi Everyone, > > Today I've been trying to pass a bag of tuples to and from a Jython UDF, so > far without success. After perusing the mailing list, I tried the following > approach > > Here's a snippet of my Pig script leading up to the use of the UDF. > > -- Group by object type and id > g_reduced_recs = GROUP reduced_recs BY (objType,objId); > > -- Shift timestamps for each object id > g_shifted_recs = FOREACH g_reduced_recs GENERATE group, > pyfu.normalize_activity_timestamps(reduced_recs); > > Describe g_shifted_recs yields... > > grunt> describe g_reduced_recs; > 2012-02-11 00:01:03,684 [main] WARN org.apache.pig.PigServer - Encountered > Warning IMPLICIT_CAST_TO_MAP 2 time(s). > g_reduced_recs: {group: (objType: chararray,objId: chararray),reduced_recs: > {objId: chararray,objType: chararray,timestamp: chararray}} > > For the UDF, I'm trying to keep it simple at the moment and just pass the > bag through the UDF unchanged. Again so far with no luck. > > > @outputSchema("b:bag{t:tuple(objId:chararray,objType:chararray,timestamp:chararray)}") > def normalize_activity_timestamps(recs): > return recs > > Diving into the failed task details I'm seeing: > > java.lang.RuntimeException: Problem while configuring reduce plan. at > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.setup(PigMapReduce.java:324) > at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174) at > org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:572) at > org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:414) at > org.apache.hadoop.mapred.Child$4.run(Child.java:270) at > java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:396) at > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) > at org.apache.hadoop.mapred.Child.main(Child.java:264) Caused by: > java.io.IOException: Deserialization error: could not instantiate > 'org.apache.pig.scripting.jython.JythonFunction' with arguments > > '[/Share/Home/chris.diehl/shared_storage/log_analysis_pig_python_scripts/pyfu.py, > normalize_activity_timestamps]' at > > org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:55) > at > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.setup(PigMapReduce.java:300) > ... 8 more Caused by: java.lang.RuntimeException: could not instantiate > 'org.apache.pig.scripting.jython.JythonFunction' with arguments > > '[/Share/Home/chris.diehl/shared_storage/log_analysis_pig_python_scripts/pyfu.py, > normalize_activity_timestamps]' at > org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:502) > at > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.instantiateFunc(POUserFunc.java:110) > at > > org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.readObject(POUserFunc.java:460) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974) at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848) at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) at > java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) at > java.util.ArrayList.readObject(ArrayList.java:593) at > sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974) at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848) at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) at > java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) at > java.util.HashMap.readObject(HashMap.java:1030) at > sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974) at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848) at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946) at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870) at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946) at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870) at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) at > java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) at > java.util.ArrayList.readObject(ArrayList.java:593) at > sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974) at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848) at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946) at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870) at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) at > java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) at > java.util.HashMap.readObject(HashMap.java:1030) at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) at > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974) at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848) at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) at > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946) at > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870) at > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328) at > java.io.ObjectInputStream.readObject(ObjectInputStream.java:350) at > > org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:53) > ... 9 more Caused by: java.lang.reflect.InvocationTargetException at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at > org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:470) > ... 74 more Caused by: java.lang.IllegalStateException: Could not > initialize: > > /Share/Home/chris.diehl/shared_storage/log_analysis_pig_python_scripts/pyfu.py > at > > org.apache.pig.scripting.jython.JythonFunction.<init>(JythonFunction.java:86) > ... 79 more > > Any thoughts on what might be going on? Or what I'm doing wrong? We're > running Pig 0.8.1. Any pointers would be most appreciated! > > Thanks, Chris > -- Russell Jurney twitter.com/rjurney [email protected] datasyndrome.com
