LsomeYeah opened a new pull request, #9023: URL: https://github.com/apache/paimon/pull/9023
## Summary Expose opt-in, plan-level written-column metadata for each Spark Structured Streaming micro-batch. This lets consumers of wide sparse change streams obtain stable field IDs from the files admitted to the current batch before materializing business rows, instead of scanning every column to infer updates from null values. ## Changes - Add an explicit Core metadata model with `KnownWrittenColumns(fieldIds)` and the conservative `AllColumns` fallback. - Resolve file write columns through each file schema so field IDs remain stable across schema evolution and column renames. Legacy files without `writeCols` expand to their file schema, while unresolved metadata falls back to all columns. - Add the opt-in Spark option `read.stream.batch-written-columns.enabled`, disabled by default. - Attach the summary to InputPartitions created from the exact splits admitted for the `(startOffset, endOffset]` micro-batch. - Add `PaimonSparkMicroBatchMetadata.writtenColumns(Dataset)` as a Driver-side, zero-Action API for raw `foreachBatch` datasets. - Isolate Spark internal RDD/InputPartition access behind Spark 3.2, 3.3, 3.4/3.5, and Spark 4 shims. - Return an empty result for disabled metadata, non-Paimon input, incomplete lineage, or ambiguous multiple Paimon sources so callers can conservatively fall back. ## Testing - [x] Core unit tests for multi-file/schema unions, field-ID stability, legacy metadata expansion, caching, and conservative fallback. - [x] Spark integration tests for default-disabled behavior and metadata access after a projection. - [x] Verify metadata lookup does not start a Spark job. - [x] Verify `maxFilesPerTrigger` does not mix columns from the next batch. - [x] Verify an explicitly written all-null column is still reported as written. - [x] Compile Spark 3.2, 3.3, 3.4, and 3.5 modules. - [x] Compile the changed Spark 4 common shim with JDK 21. - [x] Run formatting, validation, and diff checks. ## Notes This PR is intentionally a draft for discussion of the public API and multi-source/source-identity behavior. The option is disabled by default and does not change source schema, offset/checkpoint semantics, split admission, or row reader behavior. It exposes planning metadata for early narrowing inside `foreachBatch`; it does not implement per-micro-batch physical reader column pruning because the Structured Streaming source schema remains fixed for the query. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
