[jira] [Commented] (HIVE-7711) Error Serializing GenericUDF
[
https://issues.apache.org/jira/browse/HIVE-7711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798855#comment-16798855
]
Aakash Nand commented on HIVE-7711:
---
[~cbbetz] I was facing this issue from a very long time and today I tried
implementing DoNothingSerializer and it works. Thank you for the workaround.
Following is the code snippet that I implemented to solve this issue.
{code:java}
import org.apache.hive.com.esotericsoftware.kryo.Kryo;
import org.apache.hive.com.esotericsoftware.kryo.Serializer;
import org.apache.hive.com.esotericsoftware.kryo.io.Input;
import org.apache.hive.com.esotericsoftware.kryo.io.Output;
public class DoNothingSerializer extends Serializer {
@Override
public App read(Kryo arg0, Input arg1, Class arg2) {
// TODO Auto-generated method stub
return new App();
}
@Override
public void write(Kryo arg0, Output arg1, App arg2) {
// TODO Auto-generated method stub
}
}{code}
[~cbbetz] it would be really helpful if you could explain the logic behind the
solution. As a end user of hive, I have no idea how this solution is solving
the problem.
> Error Serializing GenericUDF
>
>
> Key: HIVE-7711
> URL: https://issues.apache.org/jira/browse/HIVE-7711
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Dr. Christian Betz
>Priority: Major
> Attachments: HIVE-7711.1.patch.txt
>
>
> I get an exception running a job with a GenericUDF in HIVE 0.13.0 (which was
> ok in HIVE 0.12.0).
> The org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc is serialized
> using Kryo, trying to serialize stuff in my GenericUDF which is not
> serializable (doesn't implement Serializable).
> Switching to Kryo made the comment in ExprNodeGenericFuncDesc obsolte:
> "/**
>* In case genericUDF is Serializable, we will serialize the object.
>*
>* In case genericUDF does not implement Serializable, Java will remember
> the
>* class of genericUDF and creates a new instance when deserialized. This is
>* exactly what we want.
>*/"
> Find the stacktrace below, however, the description above should be clear.
> Exception in thread "main"
> org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.UnsupportedOperationException
> Serialization trace:
> value (java.util.concurrent.atomic.AtomicReference)
> state (clojure.lang.Atom)
> state (udfs.ArraySum)
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> mapWork (org.apache.hadoop.hive.ql.plan.MapredWork)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
[jira] [Commented] (HIVE-7711) Error Serializing GenericUDF
[ https://issues.apache.org/jira/browse/HIVE-7711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14509422#comment-14509422 ] ankush commented on HIVE-7711: -- Could you please let me know how i find the kryo version ? how i find the kryo version that i using ? Please help > Error Serializing GenericUDF > > > Key: HIVE-7711 > URL: https://issues.apache.org/jira/browse/HIVE-7711 > Project: Hive > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: Dr. Christian Betz > Attachments: HIVE-7711.1.patch.txt > > > I get an exception running a job with a GenericUDF in HIVE 0.13.0 (which was > ok in HIVE 0.12.0). > The org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc is serialized > using Kryo, trying to serialize stuff in my GenericUDF which is not > serializable (doesn't implement Serializable). > Switching to Kryo made the comment in ExprNodeGenericFuncDesc obsolte: > "/** >* In case genericUDF is Serializable, we will serialize the object. >* >* In case genericUDF does not implement Serializable, Java will remember > the >* class of genericUDF and creates a new instance when deserialized. This is >* exactly what we want. >*/" > Find the stacktrace below, however, the description above should be clear. > Exception in thread "main" > org.apache.hive.com.esotericsoftware.kryo.KryoException: > java.lang.UnsupportedOperationException > Serialization trace: > value (java.util.concurrent.atomic.AtomicReference) > state (clojure.lang.Atom) > state (udfs.ArraySum) > genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) > colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) > childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator) > aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork) > mapWork (org.apache.hadoop.hive.ql.plan.MapredWork) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17) > at > org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694) > at > org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106) > at > org.apache.hive.com.esotericsoftware.kryo.
