leaves12138 opened a new pull request, #9069: URL: https://github.com/apache/paimon/pull/9069
## What - Add optional per-column maximum sequence numbers to `DataFileMeta`. - Propagate the per-column sequence watermark through data-evolution normal compaction. - Make global-index refresh planning compare the indexed columns' sequence watermarks instead of the compacted file's overall maximum sequence. - Preserve the new metadata when copying files and across current serializers. ## Why Data-evolution compaction rewrites a full normal data file. Today the compacted file's overall `maxSequenceNumber` advances when only a non-index column changes, so `SortedGlobalIndexScanner` treats the row range as an index-column update and rebuilds the global index unnecessarily. Snapshot history cannot reliably answer which column changed because snapshots may already have expired. Persisting the column watermarks in file metadata makes the decision independent of retained snapshot history and lets repeated compactions carry the information forward. ## Compatibility - The manifest field is nullable. Files written before this change fall back to the file-level maximum sequence and therefore retain the conservative rebuild behavior. - Add a serializer for the previous 20-field `DataFileMeta` layout. - Bump and route compatibility for `CommitMessage`, `DataSplit`, `IncrementalSplit`, `ChainSplit`, and `SplitSerializer`. - Bump explicitly savepoint-incompatible compaction task serializers, including the Flink changelog compaction task serializer. - Keep Spark copy-file metadata propagation intact. ## Impact For data-evolution tables using global-index column update action `IGNORE`, compaction after updating only non-index columns no longer rebuilds an already valid global index. Updates to indexed columns still trigger refresh. Legacy files remain conservative. ## Checks - 83 focused `paimon-core` tests, including historical `DataSplit` and `ManifestCommittable` compatibility resources - `SortedGlobalIndexScannerTest` with non-index updates, repeated compaction, and expired snapshots - `ChangelogCompactTaskSerializerTest` with column sequence metadata - Spark 3 common module compilation - Spotless and Checkstyle for core, Flink common, and Spark common This is a WIP draft for design and compatibility review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
