csurong opened a new issue, #11039:
URL: https://github.com/apache/paimon/issues/11039

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   
   ### Paimon version
   
   master at d7b521dd2d93e770da934c6a2022398d59d917d2 (2.2-SNAPSHOT), after 
#10928.
   
   ### Compute Engine
   
   Flink 1.20.1 and the Java stream-scan API. The shared reader path is also 
used by Spark.
   
   ### Minimal reproduce step
   
   A temporary snapshot-retention override can leave valid changelog history 
without changing the table properties. A streaming reader using those unchanged 
properties does not discover that history.
   
   In an existing Paimon catalog, create a fresh table in the default database:
   
   ```sql
   SET 'execution.runtime-mode' = 'batch';
   
   CREATE TABLE t (
       id INT,
       v INT,
       PRIMARY KEY (id) NOT ENFORCED
   ) WITH (
       'bucket' = '1',
       'write-only' = 'true',
       'changelog-producer' = 'input',
       'snapshot.num-retained.min' = '1',
       'snapshot.num-retained.max' = '4',
       'changelog.num-retained.max' = '4'
   );
   
   INSERT INTO t VALUES (1, 10);
   INSERT INTO t VALUES (2, 20);
   INSERT INTO t VALUES (3, 30);
   INSERT INTO t VALUES (4, 40);
   ```
   
   Run each INSERT separately and wait for it to finish, producing snapshots 
1–4. Then run:
   
   ```sql
   CALL sys.expire_snapshots(
       `table` => 'default.t',
       retain_max => 1
   );
   ```
   
   Only snapshot 4 remains, while changelog history for commits 1–3 is 
retained. The persisted snapshot/changelog maximum counts are still 4/4.
   
   Start a fresh streaming query, without restored state or existing consumer 
progress:
   
   ```sql
   SET 'execution.runtime-mode' = 'streaming';
   
   SELECT *
   FROM t /*+ OPTIONS('scan.snapshot-id' = '1') */;
   ```
   
   ### What doesn't meet your expectations?
   
   Expected: read the retained changes from commits 1–4 and then follow new 
commits, without repeating retention overrides on the reader.
   
   Actual: the query starts at the remaining snapshot 4 and does not read the 
retained changes from commits 1–3.
   
   A related checkpoint-recovery scenario can report that the requested 
snapshot has expired even though its changelog metadata and data files are 
still available.
   
   ### Anything else?
   
   The reader uses `changelogLifecycleDecoupled()` to decide whether to 
discover retained changelog metadata. Operation-local expiration options do not 
update the table properties, so this decision can disagree with the history 
actually retained.
   
   This concerns reading history that survived snapshot expiration. It does not 
bypass consumer protection or mean that all downstream jobs are affected.
   
   Related: #10928 fixed retention in `expire_snapshots`; this report concerns 
discovery on the reader side.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to