csurong opened a new issue, #10927: URL: https://github.com/apache/paimon/issues/10927
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version Upstream master at b4342332eafd2121daa95b89aaf215757f82edcc (2.2-SNAPSHOT). ### Compute Engine Flink 1.20.1, using the SQL procedure and the official action via ActionFactory in a local-filesystem integration test (JDK 11). The Spark procedure uses the same configuration helper. ### Minimal reproduce step I found that `expire_snapshots` can delete changelog files even when the table is configured to retain them longer than its snapshots. In a test with four commits, keeping one snapshot and setting `changelog.time-retained` to `7 d` still causes the old changelog files to be deleted immediately. This reproduction uses a local-filesystem integration test. The SQL and Java snippets share the same catalog setup. Full test: [`ExpireSnapshotsProcedureITCase.testChangelogRetention`](https://github.com/csurong/paimon/blob/b221e7e9e9af6f87a7e3481d861b70fcd5538dd0/paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/procedure/ExpireSnapshotsProcedureITCase.java#L224). 1. Use a fresh local-filesystem Paimon catalog and its `default` database. Create this primary-key table: ```sql CREATE TABLE changelog_retention ( id INT PRIMARY KEY NOT ENFORCED, v INT ) WITH ( 'bucket' = '1', 'write-only' = 'true', 'changelog-producer' = 'input', 'snapshot.num-retained.min' = '1', 'snapshot.num-retained.max' = '1', 'changelog.time-retained' = '7 d' ); ``` 2. Load `default.changelog_retention` from the same catalog as a `FileStoreTable` named `table`. Create four separate commits, containing `(1, 10)`, `(2, 20)`, `(3, 30)` and `(4, 40)`. Use the following API sequence to produce snapshots 1–4: ```java StreamWriteBuilder builder = table.newStreamWriteBuilder().withCommitUser("test"); try (StreamTableWrite write = builder.newWrite(); StreamTableCommit commit = builder.newCommit()) { for (int i = 1; i <= 4; i++) { write.write(GenericRow.of(i, i * 10)); commit.commit(i, write.prepareCommit(true, i)); } } ``` 3. Before expiration, save the first planned changelog split and verify that it reads row ID `1`: ```java StreamTableScan scan = table.newReadBuilder().newStreamScan(); scan.restore(1L); List<Split> firstChangelog = scan.plan().splits(); ``` 4. Run manual expiration without overriding any retention parameters: ```sql CALL sys.expire_snapshots(`table` => 'default.changelog_retention'); ``` The same case is reproduced by creating the official `expire_snapshots` action with `--warehouse`, `--database default`, and `--table changelog_retention`, then calling `run()`. 5. Check the remaining snapshots, the files referenced by the saved split, and the long-lived changelog metadata. The reproduction has no consumer or tag protection. ### What doesn't meet your expectations? Snapshots 1–3 should expire and snapshot 4 should remain. The changelog history should still be available: all four commits were just created and are within the configured retention period. Instead, expiration deletes the files referenced by the saved changelog split and does not retain the expected long-lived changelog history. A consumer that still needs this history can no longer read it. Current-table batch reads still return all four rows. Before the fix, both independent-retention cases fail (Flink procedure and action). The same cases without `changelog.time-retained` pass. ### Anything else? ## Source-code cause `ProcedureUtils.fillInSnapshotOptions` populates the snapshot fields of `ExpireConfig.Builder`, but does not forward the explicit values of: - `changelog.num-retained.min` - `changelog.num-retained.max` - `changelog.time-retained` The builder therefore treats these values as unset and falls back to the effective snapshot retention settings. The resulting configuration does not preserve the table's independent changelog retention policy. ## Verification With the fix, the real-table tests retain the changelog files and metadata, read the saved split, and resume a new scan from snapshot 1. Tests without independent retention keep their previous behavior. Spark tests also cover dynamic options. All Java CI jobs pass with the fix, including the three Spark matrices. The overall workflow is still red because of existing Python Native CI failures: the [unmodified baseline](https://github.com/csurong/paimon/actions/runs/41309003545) and [fix](https://github.com/csurong/paimon/actions/runs/41309711491) both report **30 failed, 24,516 passed, and 142 skipped** in that job. All 30 failing test IDs and error messages match, and both use `paimon-rust` commit `8df32eb6049415aa4bf12da0469874abde38564a`. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
