csurong opened a new pull request, #10928: URL: https://github.com/apache/paimon/pull/10928
### Purpose Fixes #10927 Manual `expire_snapshots` currently ignores explicit independent changelog retention options when building the expiration configuration. For example, a table configured to keep one snapshot and seven days of changelog history can lose its old changelog files as soon as its snapshots expire. Forward the explicit values of `changelog.num-retained.min`, `changelog.num-retained.max`, and `changelog.time-retained` from the effective table options into `ExpireConfig.Builder` in `ProcedureUtils.fillInSnapshotOptions`. Unspecified changelog options remain unset, so the builder continues to inherit the effective snapshot options, including procedure/action overrides. Tables without independent changelog retention keep their existing behavior. When only some changelog options are set, the explicit values are honored and the remaining fields still fall back to the effective snapshot settings. The shared helper is used by the Flink snapshot expiration procedure/action and the Spark snapshot expiration procedure. Spark dynamic `options` are honored for the operation without persisting them to the table. The expiration algorithm and `expire_changelogs` are unchanged. ### Tests The focused suites passed locally with the normal quality checks enabled: | Suite | Result | Coverage added | | --- | --- | --- | | `SnapshotChangelogRetentionVerificationTest` | 7 passed | Explicit retention, partially configured options, snapshot overrides, and `older_than` fallback | | `ExpireSnapshotsProcedureITCase` | 8 passed, including 4 new cases | Flink SQL procedure and official action, each with and without independent retention | | `ExpireSnapshotsProcedureTest` | 8 passed, including 3 new cases | Spark procedure with no independent retention, table-level retention, and dynamic `options` | The real-table tests expire three of four snapshots and check changelog files, metadata, saved-split reads, resumed scans, and current-table contents. Before the fix, the two new Flink independent-retention cases fail; after the fix, they pass. Cases without independent retention still delete the old changelog files. The Spark dynamic-options test also verifies that the options are not persisted. A new reader uses the same retention configuration to discover the retained history. Real-table tests cover time retention; the common tests cover explicit minimum/maximum counts and partially configured policies. Full fork CI passed all Java jobs, including Spark 3 / Scala 2.12, Spark 3 / Scala 2.13, Spark 4 / Scala 2.13, the Flink 1 and Flink 2 jobs, and Flink/Spark interoperability. **CI baseline comparison** | Run | Commit | Result | | --- | --- | --- | | [Unmodified upstream baseline](https://github.com/csurong/paimon/actions/runs/41309003545) | `b4342332ea` | 31 successful jobs; Python Native CI fails; aggregate CI result fails | | [Fix and regression tests](https://github.com/csurong/paimon/actions/runs/41309711491) | `b221e7e9e9` | 31 successful jobs; Python Native CI fails; aggregate CI result fails | The full workflow remains red because of Python Native CI. Both Native jobs report **30 failed, 24,516 passed, and 142 skipped**. The failing test IDs and error messages were compared individually and match exactly. Both use `paimon-rust` commit `8df32eb6049415aa4bf12da0469874abde38564a`. These are the same existing Python/Rust index failures on the unmodified baseline. Native job logs: [baseline](https://github.com/csurong/paimon/actions/runs/41309003545/job/124855329059), [fix](https://github.com/csurong/paimon/actions/runs/41309711491/job/124857422582). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
