[ https://issues.apache.org/jira/browse/SPARK-39140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534663#comment-17534663 ]
Gengliang Wang commented on SPARK-39140: ---------------------------------------- [~rednaxelafx] Thank you so much for the detailed explanation! This is interesting. > JavaSerializer doesn't serialize the fields of superclass > --------------------------------------------------------- > > Key: SPARK-39140 > URL: https://issues.apache.org/jira/browse/SPARK-39140 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 3.3.0 > Reporter: Gengliang Wang > Priority: Major > > To reproduce: > > {code:java} > abstract class AA { > val ts = System.nanoTime() > } > case class BB(x: Int) extends AA { > } > val input = BB(1) > println("original ts: " + input.ts) > val javaSerializer = new JavaSerializer(new SparkConf()) > val javaInstance = javaSerializer.newInstance() > val bytes1 = javaInstance.serialize[BB](input) > val obj1 = javaInstance.deserialize[BB](bytes1) > println("deserialization result from java: " + obj1.ts) > val kryoSerializer = new KryoSerializer(new SparkConf()) > val kryoInstance = kryoSerializer.newInstance() > val bytes2 = kryoInstance.serialize[BB](input) > val obj2 = kryoInstance.deserialize[BB](bytes2) > println("deserialization result from kryo: " + obj2.ts) {code} > > > The output is > > {code:java} > original ts: 115014173658666 > deserialization result from java: 115014306794333 > deserialization result from kryo: 115014173658666{code} > > We can see that the fields from the superclass AA are not serialized with > JavaSerializer. When switching to KryoSerializer, it works. > This caused bugs in the project SPARK-38615: TreeNode.origin with actual > information is not serialized to executors when a plan can't be executed with > whole-staged-codegen. > It could also lead to bugs in serializing the lambda function within RDD API > like > mapPartitions/mapPartitionsWithIndex/etc. > -- This message was sent by Atlassian Jira (v8.20.7#820007) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org