zml1206 opened a new pull request, #4544: URL: https://github.com/apache/flink-cdc/pull/4544
## What is the purpose of this pull request? Add configurable Iceberg table maintenance to streaming CDC pipelines, supporting data file compaction, snapshot expiration, and orphan file cleanup. Maintenance supports both existing tables and tables created by CDC after startup. It is disabled by default and configured through `sink.maintenance.*`. ## Brief change log - Integrate Iceberg’s `TableMaintenance` API with configurable schedules, retention settings, and JDBC locks. - Discover maintenance targets from source capture rules and pipeline routing for MySQL, PostgreSQL, Oracle, and SQL Server, with explicit target lists taking precedence. - Defer maintenance initialization until target tables exist, allowing startup and checkpoints to proceed while CDC creates them. - Refresh cached table metadata and protect empty-table metadata during orphan cleanup. - Add execution graph adaptation for Flink 1.20 and 2.x, including catalog resource cleanup and stable operator identities. - Document configuration, deployment requirements, retention policies, and recovery constraints in English and Chinese. --- ## Verifying this change This change added tests and can be verified as follows: - Added unit tests in `MaintenanceOptionsTest`, `TargetTableDiscoveryTest`, and `MaintenanceGraphAdapterTest` for configuration validation, routed target discovery, and topology adaptation. - Added regression tests in `MaintenanceRegressionTest` and `DeferredTableLoaderTest` for resource cleanup and metadata refresh. Confirmed that the cached-snapshot regression fails before the fix. - Added MiniCluster tests in `TableMaintenanceExecutionTest` covering CDC writes, all three maintenance operations, and savepoint recovery before and after target creation. - Verified `clean package` with 72 maintenance tests passing on each of Flink 1.20 and Flink 2.2. Spotless and Checkstyle checks passed. ## Documentation - Does this pull request introduce a new feature? Yes. - If yes, how is the feature documented? Docs: updated the English and Chinese Iceberg pipeline connector documentation. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Codex GPT-6 Generated-by: Codex GPT-6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
