tchivs opened a new pull request, #4089: URL: https://github.com/apache/flink-cdc/pull/4089
This PR fixes [FLINK-38221](https://issues.apache.org/jira/browse/FLINK-38221) in PostgreSQL partition table discovery that was causing complete CDC data loss when users configured parent partition table names. The core fix adds `"PARTITIONED TABLE"` type support to TableDiscoveryUtils, enabling proper discovery of parent partition tables and eliminating the need for complex regex configurations. ## Critical Problem Statement **This is a data loss bug affecting production PostgreSQL CDC deployments:** ### The Bug 1. **User configures CDC** with parent partition table name: `"aia_t_icc_jjdb"` 2. **TableDiscoveryUtils fails** to discover parent partition (only looks for "TABLE" type) 3. **Table gets filtered out** - completely ignored by CDC 4. **Result: ZERO CDC data captured** - complete data loss for partition tables ### Current Painful Workarounds - Users must use complex regex: `"aia_t_icc_jjdb_\\d{6}"` to match child partitions - This causes **extremely slow initialization** (loads schemas for all individual partitions) - Requires users to know internal partition naming schemes - Error-prone and difficult to maintain configurations ## Solution Overview **Critical Fix**: Enhanced `TableDiscoveryUtils` to include `"PARTITIONED TABLE"` type, enabling proper discovery of parent partition tables. ### Key Improvements 1. **🔧 Data Loss Fix**: Parent partition tables now properly discovered - CDC captures all data 2. **🚀 Performance Fix**: Users configure simple parent names - no more slow regex patterns 3. **✅ User Experience**: Simple configuration `"aia_t_icc_jjdb"` now works correctly 4. **🏗️ Architecture**: Unified table discovery across all CDC components ### Root Cause & Fix ```sql -- PROBLEM: Parent partition tables have different relkind SELECT relname, relkind FROM pg_class WHERE relname = 'aia_t_icc_jjdb'; -- Result: relkind = 'p' (partitioned table) - not discovered by "TABLE" type filter -- FIX: Include "PARTITIONED TABLE" type in discovery SELECT * FROM pg_tables WHERE tabletype IN ('TABLE', 'PARTITIONED TABLE'); -- Now properly discovers parent partition tables ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
