csurong opened a new pull request, #11040:
URL: https://github.com/apache/paimon/pull/11040

   ### Purpose
   
   Fixes #11039.
   
   A snapshot-retention override can leave independent changelog history 
without changing the table properties. Readers still use those properties to 
decide whether to look for that history, so valid changes can be missed on 
startup or rejected during recovery.
   
   Allow ordinary changelog streaming scans to discover retained history, and 
use the same capability for startup, recovery, Spark offset checks and AuditLog 
scans. Timestamp startup also handles metadata gaps left by repeated 
expiration: readable ranges still work, while an ambiguous starting boundary 
fails explicitly instead of skipping potentially required history.
   
   Full reads, overwrite reads and Flink lookup joins keep their existing 
discovery rules. Retention policies, consumer protection, storage formats and 
checkpoint formats are unchanged.
   
   ### Tests
   
   Added real-table regressions for startup and recovery after expiration, 
repeated cleanup, equal commit timestamps, missing metadata/data files, and 
excluded scan modes. Spark tests exercise ordinary and AuditLog streaming 
queries, including checkpoint restarts.
   
   The Flink regression runs a real job in a 1.20.1 MiniCluster: complete a 
checkpoint, stop the reader, commit updates and deletes, expire their 
snapshots, then restore. An append-only sink checks every change and its 
multiplicity, followed by another commit to verify continued consumption.
   
   Local validation on d7b521dd2d plus this change passed with JDK 8 and 
quality checks enabled:
   
   - Core: 108 tests, including 41 retained-changelog cases.
   - Spark: 39 tests across streaming and expiration-procedure suites.
   - Flink Lookup: 20 tests across scan/lookup suites.
   - Flink job recovery: all 8 tests in `FlinkJobRecoveryITCase`.
   
   As a negative control, restoring only the old changelog-discovery condition 
makes the new Flink checkpoint test fail with an expired-snapshot error. This 
control changes only that condition; it is not a full unmodified-upstream run.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to