[ https://issues.apache.org/jira/browse/HIVE-7711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123099#comment-14123099 ]
Dr. Christian Betz commented on HIVE-7711: ------------------------------------------ Hi, sorry for answering so late, but I had some other issues to work on more urgently. I really appreciate you fast answer though :) # I did check sourcecode of hive and kryo (in addition to documentation) and found a solution/workaround: Just providing my own *"does nothing"-Serializer* and "attaching" it to my GenericUDF-Class using the *@DefaultSerializer* annotation (see https://github.com/EsotericSoftware/kryo#default-serializers) solved the problem. \\ I do not have my a code snippet right now to include/attach, but it basically is an implementation of *com.esotericsoftware.kryo.Serializer* where the write method does nothing and the read method returns a new instance of the class given in the third parameter. # I reviewed the patch you provided but did not build my own hive version from it. It is basically not a solution to the problem, but emits a warning for a missing serializer. As neither GenericUDF nor my implementation of GenericUDF implement serializable I'm pretty sure I wouldn't see a warning message though. However, the idea of generating a warning message is good. Maybe you could think about implementing the above mentioned "does-nothing"-serializer for GenericUDF (and UDF), as that mimics the behaviour of the orignal code before you moved to kryo. # I'm totally missing advanced documentation for all of this. I did not find anything about "GenericUDFs should be serializable" on the web or in the GenericUDF-source code. I think there should be something on that somewhere, but I leave it up to you to decide on a new issue / subtask for that. However, thanks in advance if you do it ;) For me, you could well close that issue as my code is working right now, however I would strongly advise to improve your codebase, your tests, and your documentation on this issue. > Error Serializing GenericUDF > ---------------------------- > > Key: HIVE-7711 > URL: https://issues.apache.org/jira/browse/HIVE-7711 > Project: Hive > Issue Type: Bug > Affects Versions: 0.13.0 > Reporter: Dr. Christian Betz > Attachments: HIVE-7711.1.patch.txt > > > I get an exception running a job with a GenericUDF in HIVE 0.13.0 (which was > ok in HIVE 0.12.0). > The org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc is serialized > using Kryo, trying to serialize stuff in my GenericUDF which is not > serializable (doesn't implement Serializable). > Switching to Kryo made the comment in ExprNodeGenericFuncDesc obsolte: > "/** > * In case genericUDF is Serializable, we will serialize the object. > * > * In case genericUDF does not implement Serializable, Java will remember > the > * class of genericUDF and creates a new instance when deserialized. This is > * exactly what we want. > */" > Find the stacktrace below, however, the description above should be clear. > Exception in thread "main" > org.apache.hive.com.esotericsoftware.kryo.KryoException: > java.lang.UnsupportedOperationException > Serialization trace: > value (java.util.concurrent.atomic.AtomicReference) > state (clojure.lang.Atom) > state (udfs.ArraySum) > genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) > childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator) > aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork) > mapWork (org.apache.hadoop.hive.ql.plan.MapredWork) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672) > at > org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:918) > at > org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:826) > at > org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:840) > at > org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:738) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > Caused by: java.lang.UnsupportedOperationException > at clojure.lang.APersistentMap.put(APersistentMap.java:287) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:140) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > ... 41 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)