YangyangHu created FLINK-40453:
----------------------------------
Summary: [postgres] Avoid eagerly loading all table schemas when
initializing PostgresPipelineRecordEmitter
Key: FLINK-40453
URL: https://issues.apache.org/jira/browse/FLINK-40453
Project: Flink
Issue Type: Improvement
Components: Flink CDC
Reporter: YangyangHu
Fix For: cdc-3.7.0
Currently PostgresPipelineRecordEmitter loads table schemas eagerly during
construction (generateCreateTableEvent()) and again in the applySplit()
override.
This opens a JDBC connection, lists all captured tables, and fetches every
table's
schema up front, before any split is actually processed.
The overhead grows with the number of captured tables and slows down reader
startup.
A table's schema is only needed when its first record (or low-watermark /
schema-change
event) is emitted, so loading everything at initialization is wasteful.
This proposes to load schemas lazily: remove the eager call from the
constructor and
drop the applySplit() override, and instead resolve/cache the CreateTableEvent
on demand,
falling back to a JDBC lookup only when the split state does not already carry
the schema.
Behavior for snapshot, incremental, and schema-change paths is preserved. A new
PostgresPipelineRecordEmitterTest verifies that construction opens no JDBC
connection
and that assigning a snapshot split does not trigger loading of all table
schemas.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
