csurong opened a new issue, #11039: URL: https://github.com/apache/paimon/issues/11039
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master at d7b521dd2d93e770da934c6a2022398d59d917d2 (2.2-SNAPSHOT), after #10928. ### Compute Engine Flink 1.20.1 and the Java stream-scan API. The shared reader path is also used by Spark. ### Minimal reproduce step A temporary snapshot-retention override can leave valid changelog history without changing the table properties. A streaming reader using those unchanged properties does not discover that history. In an existing Paimon catalog, create a fresh table in the default database: ```sql SET 'execution.runtime-mode' = 'batch'; CREATE TABLE t ( id INT, v INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'bucket' = '1', 'write-only' = 'true', 'changelog-producer' = 'input', 'snapshot.num-retained.min' = '1', 'snapshot.num-retained.max' = '4', 'changelog.num-retained.max' = '4' ); INSERT INTO t VALUES (1, 10); INSERT INTO t VALUES (2, 20); INSERT INTO t VALUES (3, 30); INSERT INTO t VALUES (4, 40); ``` Run each INSERT separately and wait for it to finish, producing snapshots 1–4. Then run: ```sql CALL sys.expire_snapshots( `table` => 'default.t', retain_max => 1 ); ``` Only snapshot 4 remains, while changelog history for commits 1–3 is retained. The persisted snapshot/changelog maximum counts are still 4/4. Start a fresh streaming query, without restored state or existing consumer progress: ```sql SET 'execution.runtime-mode' = 'streaming'; SELECT * FROM t /*+ OPTIONS('scan.snapshot-id' = '1') */; ``` ### What doesn't meet your expectations? Expected: read the retained changes from commits 1–4 and then follow new commits, without repeating retention overrides on the reader. Actual: the query starts at the remaining snapshot 4 and does not read the retained changes from commits 1–3. A related checkpoint-recovery scenario can report that the requested snapshot has expired even though its changelog metadata and data files are still available. ### Anything else? The reader uses `changelogLifecycleDecoupled()` to decide whether to discover retained changelog metadata. Operation-local expiration options do not update the table properties, so this decision can disagree with the history actually retained. This concerns reading history that survived snapshot expiration. It does not bypass consumer protection or mean that all downstream jobs are affected. Related: #10928 fixed retention in `expire_snapshots`; this report concerns discovery on the reader side. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
