Davis-Zhang-Onehouse opened a new pull request, #20688: URL: https://github.com/apache/hudi/pull/20688
### Describe the issue this Pull Request addresses issue: #20687 Part of #20683, RFC-118 (#20684). Stacked on #20686: review only the last commit. Until #20686 merges, this PR also shows its commit. Defects in the batched BLOB read path: - `read_blob()` over a join whose two sides have same-named BLOB columns resolves the column by name (`schema.fieldIndex`), so it can read the other side's bytes. - Managed out-of-line BLOB references are relative to the table, and the batched reader is not given the table base path. - The broadcast storage configuration is a `HadoopStorageConfiguration` with a transient configuration; under Kryo without Hudi's registrator executors see null. - A scan that opens one reader per file keeps every exhausted reader reachable from the task's completion listeners until the task ends. ### Summary and Changelog - `BatchedBlobReadExec` / `BatchedBlobReader`: resolve the BLOB column by expression id and ordinal (`processRDDAtOrdinal`) instead of by name. - New `BlobTableBasePath`: finds the base path of the table a BLOB attribute comes from; `BatchedBlobReaderStrategy` passes it to the reader, which resolves managed relative references against it. - `BatchedBlobReader.broadcastStorageConf`: broadcast a Spark `SerializableConfiguration`, which works with any `spark.serializer`, and rebuild the storage configuration on the executor. - `CloseableIteratorListener.addListenerReleasedOnExhaustion`, used by `HoodieFileGroupReaderFunction`: release a file's reader once its iterator is exhausted or closed, not at task end. - Tests: same-named join columns in `TestReadBlobSQL` (fails on master: it reads the wrong bytes), `TestBatchedBlobReaderBasePath`, `TestBatchedBlobReaderKryo`, `TestFileScanReaderRelease`, `TestCloseableIteratorListener`, `TestBlobFileGroupReaderOutputType`, and `TestBlobPrunedProjection` (pruned BLOB projections on MOR, using master's `SparkSchemaTransformUtils` nested pruning). The scan test builds its files through the existing `SparkAdapter` factories, so it compiles on every supported Spark version. Master's general nested pruning and its identical-descriptor batching and join handling are kept unchanged. No code was copied from other projects. ### Impact No public API change. `read_blob()` returns the right bytes for same-named columns in a join, managed references resolve, Kryo jobs read out-of-line BLOBs, and long scans release readers as they go. ### Risk Level low: changes are confined to the BLOB read path and the per-file listener. Verified on JDK 11 / Spark 3.5 / Scala 2.12: 30 hudi-spark-client tests (listener, schema transform) and 30 JUnit plus 8 ScalaTest hudi-spark tests, including the whole `TestReadBlobSQL`, `TestHoodieFileGroupReaderFunction` and `TestSparkSchemaTransformUtils`; checkstyle, scalastyle and RAT pass. The changed reader sources compile on JDK 17 / Spark 4.0 / Scala 2.13. Spark 3.3 was not checked: test compilation on master already stops at `InputFormatTestUtil.java:584` (`AvroWriteSupport.WRITE_FIXED_AS_INT96` unavailable), which this PR does not touch. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
