csurong opened a new pull request, #11040: URL: https://github.com/apache/paimon/pull/11040
### Purpose Fixes #11039. A snapshot-retention override can leave independent changelog history without changing the table properties. Readers still use those properties to decide whether to look for that history, so valid changes can be missed on startup or rejected during recovery. Allow ordinary changelog streaming scans to discover retained history, and use the same capability for startup, recovery, Spark offset checks and AuditLog scans. Timestamp startup also handles metadata gaps left by repeated expiration: readable ranges still work, while an ambiguous starting boundary fails explicitly instead of skipping potentially required history. Full reads, overwrite reads and Flink lookup joins keep their existing discovery rules. Retention policies, consumer protection, storage formats and checkpoint formats are unchanged. ### Tests Added real-table regressions for startup and recovery after expiration, repeated cleanup, equal commit timestamps, missing metadata/data files, and excluded scan modes. Spark tests exercise ordinary and AuditLog streaming queries, including checkpoint restarts. The Flink regression runs a real job in a 1.20.1 MiniCluster: complete a checkpoint, stop the reader, commit updates and deletes, expire their snapshots, then restore. An append-only sink checks every change and its multiplicity, followed by another commit to verify continued consumption. Local validation on d7b521dd2d plus this change passed with JDK 8 and quality checks enabled: - Core: 108 tests, including 41 retained-changelog cases. - Spark: 39 tests across streaming and expiration-procedure suites. - Flink Lookup: 20 tests across scan/lookup suites. - Flink job recovery: all 8 tests in `FlinkJobRecoveryITCase`. As a negative control, restoring only the old changelog-discovery condition makes the new Flink checkpoint test fail with an expired-snapshot error. This control changes only that condition; it is not a full unmodified-upstream run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
