csurong opened a new issue, #10927:
URL: https://github.com/apache/paimon/issues/10927

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   
   ### Paimon version
   
   Upstream master at b4342332eafd2121daa95b89aaf215757f82edcc (2.2-SNAPSHOT).
   
   ### Compute Engine
   
   Flink 1.20.1, using the SQL procedure and the official action via 
ActionFactory in a local-filesystem integration test (JDK 11). The Spark 
procedure uses the same configuration helper.
   
   ### Minimal reproduce step
   
   I found that `expire_snapshots` can delete changelog files even when the 
table is configured to retain them longer than its snapshots. In a test with 
four commits, keeping one snapshot and setting `changelog.time-retained` to `7 
d` still causes the old changelog files to be deleted immediately.
   
   This reproduction uses a local-filesystem integration test. The SQL and Java 
snippets share the same catalog setup. Full test: 
[`ExpireSnapshotsProcedureITCase.testChangelogRetention`](https://github.com/csurong/paimon/blob/b221e7e9e9af6f87a7e3481d861b70fcd5538dd0/paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/procedure/ExpireSnapshotsProcedureITCase.java#L224).
   
   1. Use a fresh local-filesystem Paimon catalog and its `default` database. 
Create this primary-key table:
   
   ```sql
   CREATE TABLE changelog_retention (
       id INT PRIMARY KEY NOT ENFORCED,
       v INT
   ) WITH (
       'bucket' = '1',
       'write-only' = 'true',
       'changelog-producer' = 'input',
       'snapshot.num-retained.min' = '1',
       'snapshot.num-retained.max' = '1',
       'changelog.time-retained' = '7 d'
   );
   ```
   
   2. Load `default.changelog_retention` from the same catalog as a 
`FileStoreTable` named `table`. Create four separate commits, containing `(1, 
10)`, `(2, 20)`, `(3, 30)` and `(4, 40)`. Use the following API sequence to 
produce snapshots 1–4:
   
   ```java
   StreamWriteBuilder builder = 
table.newStreamWriteBuilder().withCommitUser("test");
   try (StreamTableWrite write = builder.newWrite();
           StreamTableCommit commit = builder.newCommit()) {
       for (int i = 1; i <= 4; i++) {
           write.write(GenericRow.of(i, i * 10));
           commit.commit(i, write.prepareCommit(true, i));
       }
   }
   ```
   
   3. Before expiration, save the first planned changelog split and verify that 
it reads row ID `1`:
   
   ```java
   StreamTableScan scan = table.newReadBuilder().newStreamScan();
   scan.restore(1L);
   List<Split> firstChangelog = scan.plan().splits();
   ```
   
   4. Run manual expiration without overriding any retention parameters:
   
   ```sql
   CALL sys.expire_snapshots(`table` => 'default.changelog_retention');
   ```
   
   The same case is reproduced by creating the official `expire_snapshots` 
action with `--warehouse`, `--database default`, and `--table 
changelog_retention`, then calling `run()`.
   
   5. Check the remaining snapshots, the files referenced by the saved split, 
and the long-lived changelog metadata. The reproduction has no consumer or tag 
protection.
   
   ### What doesn't meet your expectations?
   
   Snapshots 1–3 should expire and snapshot 4 should remain. The changelog 
history should still be available: all four commits were just created and are 
within the configured retention period.
   
   Instead, expiration deletes the files referenced by the saved changelog 
split and does not retain the expected long-lived changelog history. A consumer 
that still needs this history can no longer read it. Current-table batch reads 
still return all four rows.
   
   Before the fix, both independent-retention cases fail (Flink procedure and 
action). The same cases without `changelog.time-retained` pass.
   
   ### Anything else?
   
   ## Source-code cause
   
   `ProcedureUtils.fillInSnapshotOptions` populates the snapshot fields of 
`ExpireConfig.Builder`, but does not forward the explicit values of:
   
   - `changelog.num-retained.min`
   - `changelog.num-retained.max`
   - `changelog.time-retained`
   
   The builder therefore treats these values as unset and falls back to the 
effective snapshot retention settings. The resulting configuration does not 
preserve the table's independent changelog retention policy.
   
   ## Verification
   
   With the fix, the real-table tests retain the changelog files and metadata, 
read the saved split, and resume a new scan from snapshot 1. Tests without 
independent retention keep their previous behavior. Spark tests also cover 
dynamic options.
   
   All Java CI jobs pass with the fix, including the three Spark matrices. The 
overall workflow is still red because of existing Python Native CI failures: 
the [unmodified 
baseline](https://github.com/csurong/paimon/actions/runs/41309003545) and 
[fix](https://github.com/csurong/paimon/actions/runs/41309711491) both report 
**30 failed, 24,516 passed, and 142 skipped** in that job. All 30 failing test 
IDs and error messages match, and both use `paimon-rust` commit 
`8df32eb6049415aa4bf12da0469874abde38564a`.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to