JingsongLi commented on PR #11076:
URL: https://github.com/apache/paimon/pull/11076#issuecomment-6784075689

   [P2] Keep the compressed Spark exchange independent of application Kryo 
registrations
   
   At 
[`NullBitmapShuffle.java:194`](https://github.com/apache/paimon/blob/5110959a3b51b2b6a8d4634354ca32f08056d445/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/globalindex/sorted/NullBitmapShuffle.java#L194),
 `partitionBy` uses the application's `spark.serializer`, whereas the original 
SQL range exchange and the new uncompressed path use `UnsafeRowSerializer`. 
With `spark.serializer=org.apache.spark.serializer.KryoSerializer` and 
`spark.kryo.registrationRequired=true`, compression introduces 
`GenericRowWithSchema` / `GenericRow` objects that Spark does not register by 
default, so an otherwise working index build fails during shuffle writing.
   
   I reproduced this on Java 8 / Spark 3.5.9 with 10,000 rows, 90% null keys 
and four output partitions: the original SQL exchange and the new uncompressed 
path both return 10,000 rows, but the compressed path fails in 
`BypassMergeSortShuffleWriter` with `Class is not registered: 
org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema`. Each output 
partition returns only a `Long` count, so this is not caused by collecting 
external `Row` results. Setting `registrationRequired=false` makes all three 
paths succeed.
   
   Please use an explicit transport serializer for this exchange, or fall back 
to the native path when the serializer configuration is unsupported, and add a 
regression test with strict Kryo registration enabled.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to