[
https://issues.apache.org/jira/browse/HIVE-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200771#comment-15200771
]
Xuefu Zhang commented on HIVE-13277:
------------------------------------
[~lirui], thanks for for your investigation. While new Kryo may fix the
problem, do we know what's caused the issue? Will the problem go away if we
register the class in question? Thanks.
> Exception "Unable to create serializer
> 'org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer' "
> occurred during query execution on spark engine when vectorized execution is
> switched on
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-13277
> URL: https://issues.apache.org/jira/browse/HIVE-13277
> Project: Hive
> Issue Type: Bug
> Environment: Hive on Spark engine
> Hive Version: Apache Hive 2.0.0
> Spark Version: Apache Spark 1.6.0
> Reporter: Xin Hao
>
> Found when executing TPCx-BB query2 for Hive on Spark engine, and switch on :
> Found during TPCx-BB query2 execution on spark engine when vectorized
> execution is switched on:
> (1) set hive.vectorized.execution.enabled=true;
> (2) set hive.vectorized.execution.reduce.enabled=true; (default value for
> Apache Hive 2.0.0)
> It's OK for spark engine when hive.vectorized.execution.enabled is switched
> off:
> (1) set hive.vectorized.execution.enabled=false;
> (2) set hive.vectorized.execution.reduce.enabled=true;
> For MR engine, the query could pass and no exception occurred when vectorized
> execution is either switched on or switched off.
> Detail Error Message is below:
> {noformat}
> 2016-03-14T10:09:33,692 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - 16/03/14 10:09:33 INFO
> spark.MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 154
> bytes
> 2016-03-14T10:09:33,818 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - 16/03/14 10:09:33 WARN
> scheduler.TaskSetManager: Lost task 0.0 in stage 4.0 (TID 25, bhx3):
> java.lang.RuntimeException: Failed to load plan:
> hdfs://bhx3:8020/tmp/hive/root/40b90ebd-32d4-47bc-a5ab-12ff1c05d0d2/hive_2016-03-14_10-08-56_307_7692316402338632647-1/-mr-10002/ab0c0021-0c1a-496e-9703-87d5879353c8/reduce.xml:
> org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.IllegalArgumentException: Unable to create serializer
> "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
> class: org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator
> 2016-03-14T10:09:33,818 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - Serialization trace:
> 2016-03-14T10:09:33,818 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - childOperators
> (org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator)
> 2016-03-14T10:09:33,818 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - childOperators
> (org.apache.hadoop.hive.ql.exec.vector.VectorLimitOperator)
> 2016-03-14T10:09:33,818 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - childOperators
> (org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator)
> 2016-03-14T10:09:33,818 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - reducer
> (org.apache.hadoop.hive.ql.plan.ReduceWork)
> 2016-03-14T10:09:33,818 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:451)
> 2016-03-14T10:09:33,818 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hadoop.hive.ql.exec.Utilities.getReduceWork(Utilities.java:306)
> 2016-03-14T10:09:33,819 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.init(SparkReduceRecordHandler.java:117)
> 2016-03-14T10:09:33,819 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunction.call(HiveReduceFunction.java:46)
> 2016-03-14T10:09:33,819 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hadoop.hive.ql.exec.spark.HiveReduceFunction.call(HiveReduceFunction.java:28)
> 2016-03-14T10:09:33,819 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:192)
> 2016-03-14T10:09:33,819 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:192)
> 2016-03-14T10:09:33,819 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
> 2016-03-14T10:09:33,819 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
> 2016-03-14T10:09:33,819 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> 2016-03-14T10:09:33,819 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> 2016-03-14T10:09:33,819 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> 2016-03-14T10:09:33,819 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> 2016-03-14T10:09:33,819 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.spark.scheduler.Task.run(Task.scala:89)
> 2016-03-14T10:09:33,819 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
> 2016-03-14T10:09:33,819 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 2016-03-14T10:09:33,819 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 2016-03-14T10:09:33,820 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> java.lang.Thread.run(Thread.java:745)
> 2016-03-14T10:09:33,820 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - Caused by:
> org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.IllegalArgumentException: Unable to create serializer
> "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
> class: org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator
> 2016-03-14T10:09:33,820 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - Serialization trace:
> 2016-03-14T10:09:33,820 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - childOperators
> (org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator)
> 2016-03-14T10:09:33,820 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - childOperators
> (org.apache.hadoop.hive.ql.exec.vector.VectorLimitOperator)
> 2016-03-14T10:09:33,820 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - childOperators
> (org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator)
> 2016-03-14T10:09:33,820 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - reducer
> (org.apache.hadoop.hive.ql.plan.ReduceWork)
> 2016-03-14T10:09:33,820 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)
> 2016-03-14T10:09:33,820 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> 2016-03-14T10:09:33,820 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
> 2016-03-14T10:09:33,820 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> 2016-03-14T10:09:33,820 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> 2016-03-14T10:09:33,820 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
> 2016-03-14T10:09:33,820 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> 2016-03-14T10:09:33,820 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> 2016-03-14T10:09:33,820 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
> 2016-03-14T10:09:33,821 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> 2016-03-14T10:09:33,821 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> 2016-03-14T10:09:33,821 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
> 2016-03-14T10:09:33,821 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> 2016-03-14T10:09:33,821 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
> 2016-03-14T10:09:33,821 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:686)
> 2016-03-14T10:09:33,821 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializeObjectByKryo(SerializationUtilities.java:438)
> 2016-03-14T10:09:33,821 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializePlan(SerializationUtilities.java:347)
> 2016-03-14T10:09:33,821 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializePlan(SerializationUtilities.java:324)
> 2016-03-14T10:09:33,821 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:424)
> 2016-03-14T10:09:33,821 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - ... 17 more
> 2016-03-14T10:09:33,821 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - Caused by:
> java.lang.IllegalArgumentException: Unable to create serializer
> "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for
> class: org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator
> 2016-03-14T10:09:33,821 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:67)
> 2016-03-14T10:09:33,822 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:45)
> 2016-03-14T10:09:33,822 INFO [stderr-redir-1]: client.SparkClientImpl
> (SparkClientImpl.java:run(593)) - at
> org.apache.hive.com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:380)
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
