locchipinti opened a new pull request, #4357:
URL: https://github.com/apache/flink-cdc/pull/4357
### Motivation
In high-throughput environments backed by AWS Aurora / RDS (or any MySQL
setup with read replicas), running snapshot scans against the primary writer
instance increases its I/O load significantly.
This change introduces a `scan.snapshot.hostname` configuration option that
routes all snapshot data reads and metadata queries (table discovery, chunk
splitting) to a designated read replica, while keeping all binlog-based
operations on the primary writer.
### Changes
#### New configuration option
- `scan.snapshot.hostname` (MySqlSourceConfig, MySqlSourceConfigFactory,
MySqlSourceBuilder, pipeline connector MySqlDataSourceOptions): optional
hostname of a MySQL read replica used exclusively for snapshot reads and
metadata queries. When not set, all operations fall back to the primary.
#### Connection routing
- DebeziumUtils.createSnapshotMySqlConnection(): new method that creates a
Debezium MySqlConnection pointing to the snapshot replica (used by
SnapshotSplitReader for SELECT data reads).
- DebeziumUtils.openSnapshotJdbcConnection(): new method that opens a
Hikari-backed JdbcConnection to the snapshot replica (used by
MySqlChunkSplitter, MySqlSnapshotSplitAssigner, MySqlSource for metadata
queries).
- DebeziumUtils.openJdbcConnection() is unchanged and always connects to
the primary — ensuring MySqlValidator (which checks binlog_format /
binlog_row_image) always runs against the writer, not the replica.
#### Split-level dual-connection handling (MySqlSnapshotSplitReadTask,
StatefulTaskContext)
- StatefulTaskContext now holds two connection references: connection
(primary, for binlog watermarks via SHOW MASTER STATUS) and snapshotConnection
(replica, for SELECT data reads). A legacy single-connection constructor is
kept for backward compatibility.
- close() uses reference-identity check (snapshotConnection != connection)
to avoid double-closing when both point to the same object.
#### Connection pool routing (JdbcConnectionFactory, JdbcConnectionPools,
PooledDataSourceFactory)
- Added hostnameOverride constructor to JdbcConnectionFactory and matching
overloads in JdbcConnectionPools / PooledDataSourceFactory to allow Hikari pool
connections to target the replica host.
#### Semantics
⚠️ At-least-once only. When `scan.snapshot.hostname` is set, exactly-once
semantics cannot be guaranteed.
Binlog watermark positions are recorded from the primary writer, but
snapshot data is read from the replica.
Aurora/RDS storage replication lag (typically milliseconds) means these may
not be perfectly consistent, potentially causing duplicate records during
changelog replay.
This is an accepted trade-off for reduced primary load, as discussed with
@leonardBang
### Tests
- MySqlSnapshotSplitReadTaskConnectionTest (new, 7 tests): verifies correct
connection assignment in StatefulTaskContext, legacy constructor behavior, and
close() double-close protection.
- DebeziumUtilsTest: extended with 3 tests covering snapshot connection
routing and fallback.
### Documentation
- Options table: added `scan.snapshot.hostname` row.
- New section `Read Replica Snapshot` explaining the feature, at-least-once
caveat, and an AWS Aurora/RDS example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
