JingsongLi commented on PR #11076: URL: https://github.com/apache/paimon/pull/11076#issuecomment-6784075689
[P2] Keep the compressed Spark exchange independent of application Kryo registrations At [`NullBitmapShuffle.java:194`](https://github.com/apache/paimon/blob/5110959a3b51b2b6a8d4634354ca32f08056d445/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/globalindex/sorted/NullBitmapShuffle.java#L194), `partitionBy` uses the application's `spark.serializer`, whereas the original SQL range exchange and the new uncompressed path use `UnsafeRowSerializer`. With `spark.serializer=org.apache.spark.serializer.KryoSerializer` and `spark.kryo.registrationRequired=true`, compression introduces `GenericRowWithSchema` / `GenericRow` objects that Spark does not register by default, so an otherwise working index build fails during shuffle writing. I reproduced this on Java 8 / Spark 3.5.9 with 10,000 rows, 90% null keys and four output partitions: the original SQL exchange and the new uncompressed path both return 10,000 rows, but the compressed path fails in `BypassMergeSortShuffleWriter` with `Class is not registered: org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema`. Each output partition returns only a `Long` count, so this is not caused by collecting external `Row` results. Setting `registrationRequired=false` makes all three paths succeed. Please use an explicit transport serializer for this exchange, or fall back to the native path when the serializer configuration is unsupported, and add a regression test with strict Kryo registration enabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
