[
https://issues.apache.org/jira/browse/FLINK-38221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yanquan Lv resolved FLINK-38221.
--------------------------------
Assignee: Vinh Pham
Resolution: Fixed
> PostgreSQL CDC fails to capture data from partition tables when configured
> with parent table names
> --------------------------------------------------------------------------------------------------
>
> Key: FLINK-38221
> URL: https://issues.apache.org/jira/browse/FLINK-38221
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
> Affects Versions: cdc-3.4.0
> Environment: PostgreSQL CDC with partitioned tables
> Reporter: tchivs
> Assignee: Vinh Pham
> Priority: Major
> Labels: pull-request-available
> Fix For: cdc-3.5.0
>
>
> h2. Problem Description
> h3. Critical Data Loss Bug
> PostgreSQL CDC connector completely fails to capture any changelog data when
> users configure partition tables using standard parent table names. This is a
> *critical data loss bug* affecting production deployments.
> h3. Reproduction Steps
> # *Create a partitioned table in PostgreSQL:*
> {{CREATE TABLE aia_t_icc_jjdb (
> id BIGINT,
> name VARCHAR(255),
> partition_date DATE NOT NULL,
> PRIMARY KEY (id, partition_date)
> ) PARTITION BY RANGE (partition_date);
> CREATE TABLE aia_t_icc_jjdb_202401 PARTITION OF aia_t_icc_jjdb
> FOR VALUES FROM ('2024-01-01') TO ('2024-02-01');}}
> # *Configure Flink CDC with parent table name:*
> {{source:
> type: postgres-cdc
> hostname: localhost
> port: 5432
> username: postgres
> password: password
> database-name: testdb
> schema-name: public
> table-name: aia_t_icc_jjdb # Parent partition table}}
> # *Insert data into the partition table:*
> {{INSERT INTO aia_t_icc_jjdb (id, name, partition_date)
> VALUES (1, 'test', '2024-01-15');}}
> h3. Expected Behavior
> * CDC should discover the parent partition table {{aia_t_icc_jjdb}}
> * CDC should capture INSERT/UPDATE/DELETE operations on the partition table
> * Changelog events should be emitted for data changes
> h3. Actual Behavior
> * *NO CDC data is captured* - complete data loss
> * Parent partition table {{aia_t_icc_jjdb}} is filtered out during table
> discovery
> * CDC connector runs without errors but produces zero changelog events
> * Users receive no indication that the table configuration is ineffective
> h2. Root Cause Analysis
> h3. Technical Root Cause
> The issue is in {{TableDiscoveryUtils.java}} line 55-56:
> {{Set<TableId> allTableIds = jdbc.readTableNames(
> database, null, null, new String[] \{"TABLE"}); // Missing "PARTITIONED
> TABLE"}}
> h3. PostgreSQL Partition Table Structure
> * {*}Parent partition tables{*}: {{pg_class.relkind = 'p'}} (partitioned
> table)
> * {*}Child partition tables{*}: {{pg_class.relkind = 'r'}} (regular table)
> * {*}Table discovery query{*}: Only looks for {{tabletype = 'TABLE'}}
> * {*}Problem{*}: Parent partitions need {{tabletype = 'PARTITIONED TABLE'}}
> to be discovered
> h3. Discovery Failure Chain
> # User configures {{table-name: "aia_t_icc_jjdb"}} (parent partition)
> # TableDiscoveryUtils queries {{pg_tables WHERE tabletype IN ('TABLE')}}
> # Parent partition table not found (it's type 'PARTITIONED TABLE')
> # Table filter excludes the configured table name
> # Result: Zero CDC data captured
> h2. Current User Workarounds (and their problems)
> h3. Workaround 1: Regex Pattern Matching
> {{table-name: "aia_t_icc_jjdb_\\d\{6}" # Match child partitions}}
> *Problems:*
> * Extremely slow initialization (loads schema for every child partition)
> * Complex configuration requiring internal partition naming knowledge
> * Error-prone and maintenance-intensive
> * Breaks when partition naming changes
> h3. Workaround 2: List All Child Partitions
> {{table-name: "aia_t_icc_jjdb_202401,aia_t_icc_jjdb_202402,..."}}
> *Problems:*
> * Manual maintenance required for new partitions
> * Configuration becomes unmanageable with many partitions
> * Still slower than parent table approach
> h2. Impact Assessment
> h3. Severity: Critical
> * {*}Data Loss{*}: Complete CDC data loss for partition tables
> * {*}Silent Failure{*}: No obvious error messages, difficult to diagnose
> * {*}Production Impact{*}: Affects real-time data pipelines using partition
> tables
> * {*}User Experience{*}: Forces complex, error-prone configurations
> h3. Affected Scenarios
> * Any PostgreSQL CDC deployment using partitioned tables
> * Range partitioned tables (most common use case)
> * List partitioned tables
> * Hash partitioned tables
> * Multi-level partition hierarchies
> h3. Business Impact
> * Real-time analytics miss partition table data changes
> * Data synchronization failures in production
> * Increased operational complexity due to workarounds
> * Potential compliance issues due to incomplete data capture
> h2. Proposed Solution
> h3. Core Fix
> Enhance {{TableDiscoveryUtils.java}} to include partitioned tables:
> {{Set<TableId> allTableIds = jdbc.readTableNames(
> database, null, null, new String[] \{"TABLE", "PARTITIONED TABLE"});}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
