[jira] [Comment Edited] (HIVE-7711) Error Serializing GenericUDF
[
https://issues.apache.org/jira/browse/HIVE-7711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798855#comment-16798855
]
Aakash Nand edited comment on HIVE-7711 at 3/22/19 9:41 AM:
[~cbbetz], I was facing this issue from a very long time and today I tried
implementing DoNothingSerializer and it works. Thank you for the workaround.
Following is the code snippet that I implemented to solve this issue.
{code:java}
import org.apache.hive.com.esotericsoftware.kryo.Kryo;
import org.apache.hive.com.esotericsoftware.kryo.Serializer;
import org.apache.hive.com.esotericsoftware.kryo.io.Input;
import org.apache.hive.com.esotericsoftware.kryo.io.Output;
public class DoNothingSerializer extends Serializer {
@Override
public App read(Kryo arg0, Input arg1, Class arg2) {
// TODO Auto-generated method stub
return new App();
}
@Override
public void write(Kryo arg0, Output arg1, App arg2) {
// TODO Auto-generated method stub
}
}{code}
[~cbbetz], it would be really helpful if you could explain the logic behind the
solution. As an end user of the hive, it is difficult for me to understand how
serializer is solving this problem.
I also added the following annotation to App.java for DefaultSerializer
{code:java}
@DefaultSerializer(value = DoNothingSerializer.class)
public class App extends GenericUDF
{
// Some logic here
}{code}
was (Author: aakashnand):
[~cbbetz] I was facing this issue from a very long time and today I tried
implementing DoNothingSerializer and it works. Thank you for the workaround.
Following is the code snippet that I implemented to solve this issue.
{code:java}
import org.apache.hive.com.esotericsoftware.kryo.Kryo;
import org.apache.hive.com.esotericsoftware.kryo.Serializer;
import org.apache.hive.com.esotericsoftware.kryo.io.Input;
import org.apache.hive.com.esotericsoftware.kryo.io.Output;
public class DoNothingSerializer extends Serializer {
@Override
public App read(Kryo arg0, Input arg1, Class arg2) {
// TODO Auto-generated method stub
return new App();
}
@Override
public void write(Kryo arg0, Output arg1, App arg2) {
// TODO Auto-generated method stub
}
}{code}
[~cbbetz], it would be really helpful if you could explain the logic behind the
solution. As an end user of the hive, I have no idea how this solution is
solving the problem.
I also added the following annotation to App.java for DefaultSerializer
{code:java}
@DefaultSerializer(value = DoNothingSerializer.class)
public class App extends GenericUDF
{
// Some logic here
}{code}
> Error Serializing GenericUDF
>
>
> Key: HIVE-7711
> URL: https://issues.apache.org/jira/browse/HIVE-7711
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Dr. Christian Betz
>Priority: Major
> Attachments: HIVE-7711.1.patch.txt
>
>
> I get an exception running a job with a GenericUDF in HIVE 0.13.0 (which was
> ok in HIVE 0.12.0).
> The org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc is serialized
> using Kryo, trying to serialize stuff in my GenericUDF which is not
> serializable (doesn't implement Serializable).
> Switching to Kryo made the comment in ExprNodeGenericFuncDesc obsolte:
> "/**
>* In case genericUDF is Serializable, we will serialize the object.
>*
>* In case genericUDF does not implement Serializable, Java will remember
> the
>* class of genericUDF and creates a new instance when deserialized. This is
>* exactly what we want.
>*/"
> Find the stacktrace below, however, the description above should be clear.
> Exception in thread "main"
> org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.UnsupportedOperationException
> Serialization trace:
> value (java.util.concurrent.atomic.AtomicReference)
> state (clojure.lang.Atom)
> state (udfs.ArraySum)
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> mapWork (org.apache.hadoop.hive.ql.plan.MapredWork)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
>
[jira] [Comment Edited] (HIVE-7711) Error Serializing GenericUDF
[
https://issues.apache.org/jira/browse/HIVE-7711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798855#comment-16798855
]
Aakash Nand edited comment on HIVE-7711 at 3/22/19 9:40 AM:
[~cbbetz] I was facing this issue from a very long time and today I tried
implementing DoNothingSerializer and it works. Thank you for the workaround.
Following is the code snippet that I implemented to solve this issue.
{code:java}
import org.apache.hive.com.esotericsoftware.kryo.Kryo;
import org.apache.hive.com.esotericsoftware.kryo.Serializer;
import org.apache.hive.com.esotericsoftware.kryo.io.Input;
import org.apache.hive.com.esotericsoftware.kryo.io.Output;
public class DoNothingSerializer extends Serializer {
@Override
public App read(Kryo arg0, Input arg1, Class arg2) {
// TODO Auto-generated method stub
return new App();
}
@Override
public void write(Kryo arg0, Output arg1, App arg2) {
// TODO Auto-generated method stub
}
}{code}
[~cbbetz], it would be really helpful if you could explain the logic behind the
solution. As an end user of the hive, I have no idea how this solution is
solving the problem.
I also added the following annotation to App.java for DefaultSerializer
{code:java}
@DefaultSerializer(value = DoNothingSerializer.class)
public class App extends GenericUDF
{
// Some logic here
}{code}
was (Author: aakashnand):
[~cbbetz] I was facing this issue from a very long time and today I tried
implementing DoNothingSerializer and it works. Thank you for the workaround.
Following is the code snippet that I implemented to solve this issue.
{code:java}
import org.apache.hive.com.esotericsoftware.kryo.Kryo;
import org.apache.hive.com.esotericsoftware.kryo.Serializer;
import org.apache.hive.com.esotericsoftware.kryo.io.Input;
import org.apache.hive.com.esotericsoftware.kryo.io.Output;
public class DoNothingSerializer extends Serializer {
@Override
public App read(Kryo arg0, Input arg1, Class arg2) {
// TODO Auto-generated method stub
return new App();
}
@Override
public void write(Kryo arg0, Output arg1, App arg2) {
// TODO Auto-generated method stub
}
}{code}
[~cbbetz] it would be really helpful if you could explain the logic behind the
solution. As a end user of hive, I have no idea how this solution is solving
the problem.
> Error Serializing GenericUDF
>
>
> Key: HIVE-7711
> URL: https://issues.apache.org/jira/browse/HIVE-7711
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Dr. Christian Betz
>Priority: Major
> Attachments: HIVE-7711.1.patch.txt
>
>
> I get an exception running a job with a GenericUDF in HIVE 0.13.0 (which was
> ok in HIVE 0.12.0).
> The org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc is serialized
> using Kryo, trying to serialize stuff in my GenericUDF which is not
> serializable (doesn't implement Serializable).
> Switching to Kryo made the comment in ExprNodeGenericFuncDesc obsolte:
> "/**
>* In case genericUDF is Serializable, we will serialize the object.
>*
>* In case genericUDF does not implement Serializable, Java will remember
> the
>* class of genericUDF and creates a new instance when deserialized. This is
>* exactly what we want.
>*/"
> Find the stacktrace below, however, the description above should be clear.
> Exception in thread "main"
> org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.UnsupportedOperationException
> Serialization trace:
> value (java.util.concurrent.atomic.AtomicReference)
> state (clojure.lang.Atom)
> state (udfs.ArraySum)
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> mapWork (org.apache.hadoop.hive.ql.plan.MapredWork)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at
>
