[
https://issues.apache.org/jira/browse/SPARK-21569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16514529#comment-16514529
]
YongGang Cao edited comment on SPARK-21569 at 6/15/18 11:47 PM:
----------------------------------------------------------------
seems this is not workaround-able, at least from java side. unless we turn off
the registration required which will harm the performance as documented.
tried to register both of following in SparkConf, no luck, still get the not
registered error message.
{code:java}
org.apache.spark.internal.io.FileCommitProtocol.TaskCommitMessage.class,
org.apache.spark.internal.io.FileCommitProtocol.TaskCommitMessage[].class{code}
was (Author: ygcao):
seems this is not workaround-able, at least from java side.
tried to register both of following in SparkConf, no luck, still get the not
registered error message.
{code:java}
org.apache.spark.internal.io.FileCommitProtocol.TaskCommitMessage.class,
org.apache.spark.internal.io.FileCommitProtocol.TaskCommitMessage[].class{code}
> Internal Spark class needs to be kryo-registered
> ------------------------------------------------
>
> Key: SPARK-21569
> URL: https://issues.apache.org/jira/browse/SPARK-21569
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.2.0
> Reporter: Ryan Williams
> Priority: Major
>
> [Full repro here|https://github.com/ryan-williams/spark-bugs/tree/hf]
> As of 2.2.0, {{saveAsNewAPIHadoopFile}} jobs fail (when
> {{spark.kryo.registrationRequired=true}}) with:
> {code}
> java.lang.IllegalArgumentException: Class is not registered:
> org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage
> Note: To register this class use:
> kryo.register(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage.class);
> at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:458)
> at
> com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
> at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:488)
> at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:593)
> at
> org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:315)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:383)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> This internal Spark class should be kryo-registered by Spark by default.
> This was not a problem in 2.1.1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
