rahil-c opened a new issue, #20191: URL: https://github.com/apache/hudi/issues/20191
### Describe the problem `read_blob()` on OUT_OF_LINE blobs fails on executors when `spark.serializer` is `KryoSerializer` and Hudi's Kryo registrator (`HoodieSparkKryoRegistrar`) is not configured. `BatchedBlobReader.readBatched` and `BatchedBlobReadExec.doExecute` broadcast a `HadoopStorageConfiguration` directly. Its Hadoop `Configuration` is a `transient` field that only Java `readObject` restores. Plain Kryo skips it, so the executor gets a storage configuration whose configuration is null, and opening the external file fails with a `NullPointerException`. Hudi's registrator registers `HadoopStorageConfiguration` with a `JavaSerializer`, which hides the bug in setups that follow the quick-start configs. Engines and jobs that enable Kryo without it hit the failure on every out-of-line read. Local mode never serializes broadcasts to tasks, so existing tests do not see it. ### To reproduce 1. Start Spark with `spark.serializer=org.apache.spark.serializer.KryoSerializer` and no `spark.kryo.registrator`, on a real cluster (not local mode). 2. Read a Hudi table with an OUT_OF_LINE `BLOB` column and select `read_blob(col)`. 3. Tasks fail with a `NullPointerException` from the null Hadoop configuration. ### Expected behavior `read_blob()` works under any `spark.serializer`, as the rest of Hudi's read path does by broadcasting `org.apache.spark.util.SerializableConfiguration`. ### Environment - Hudi version: master (1.2.0-SNAPSHOT) - Spark version: 3.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
