leaves12138 opened a new pull request, #9313: URL: https://github.com/apache/paimon/pull/9313
## What - Add an optional positional `long[]` of per-column maximum sequence numbers to `DataFileMeta`. - Reuse `writeCols` to resolve array positions; full-column files use the file schema field order and row-tracking system fields are excluded. - Propagate per-column sequence watermarks through data-evolution normal compaction by stable field id. - Persist the array only for `global-index.column-update-action=IGNORE`, and store `null` when every column already equals the file-level maximum. - Preserve the metadata across manifest, commit-message, split, compaction-task, Flink checkpoint, and Spark copy compatibility paths. - Return defensive copies from POJO metadata. ## Why Data-evolution compaction rewrites a full normal data file, so the file-level `maxSequenceNumber` cannot identify which physical columns changed. Snapshot history is not a reliable source for this information because snapshots may expire. Persisting column-level sequence watermarks in `DataFileMeta` provides durable metadata for consumers which need to distinguish column updates across repeated compactions. When all physical columns equal the file-level maximum, `null` is semantically equivalent because readers fall back to that maximum. Tables not configured with update action `IGNORE` do not persist the array. ## Compatibility - The manifest field is nullable. Files written before this change fall back to the file-level maximum sequence. - Missing or malformed positional arrays also fall back to the file-level maximum. - New Avro manifest files remain readable by the previous 20-field reader schema. - Add legacy serializers for the previous 20-field `DataFileMeta` and nested `ManifestEntry` layouts. - Bump and route compatibility for `CommitMessage`, `DataSplit`, `IncrementalSplit`, and `ChainSplit`. - `SplitSerializer` writes v2, reads v1 and v2, and routes v1 through the previous `DataFileMeta` layout. - Flink pending-split checkpoint recovery covers legacy `IncrementalSplit` v1 and `ChainSplit` v2 state. - Spark `sys.copy` clears the positional array when assigning a different schema id. ## Scope This PR only introduces and propagates the durable `DataFileMeta` metadata. It does not change global-index refresh planning or scanner decisions. Those changes remain in follow-up PR #9069. This was split from #9069 following review feedback. ## Tests - Core metadata, manifest, commit-message, split, and compatibility suites: 111 tests. - Data-evolution compaction E2E coverage for repeated partial-column updates, redundant-array elision, and non-`IGNORE` behavior. - Flink task serializer and pending-split checkpoint recovery: 7 tests. - Spark `CopyFilesUtilTest`. - Spotless and Checkstyle for API, core, Flink common, and Spark common. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
