paulo-t opened a new pull request, #4359: URL: https://github.com/apache/flink-cdc/pull/4359
## Summary This PR fixes [FLINK-39391](https://issues.apache.org/jira/browse/FLINK-39391). The user-configured `scan.snapshot.fetch.size` option has **no effect** on the actual JDBC `fetchSize` used during the incremental snapshot phase for Oracle, SqlServer, DB2, and Postgres connectors. The value is correctly parsed and stored in `JdbcSourceConfig.fetchSize`, but is never written into the Debezium `Properties` object that the snapshot execution path actually reads. ### Root Cause - **Oracle / SqlServer / DB2**: The snapshot task calls `connectorConfig.getQueryFetchSize()`, which reads Debezium's `query.fetch.size` property. This defaults to `0` when not set, causing the JDBC driver to fall back to its own small default (e.g., Oracle defaults to `fetchSize=10`). On high-latency networks, this results in **order-of-magnitude performance degradation** (~440 rows/s instead of ~3,500 rows/s with 4 parallel readers on ~31ms RTT). - **Postgres**: The snapshot task calls `connectorConfig.getSnapshotFetchSize()`, which reads `snapshot.fetch.size`. This defaults to `2000`, so the impact is less severe but the user-configured value is still silently ignored. MySQL is **NOT affected** because `MySqlSourceConfigFactory` already correctly sets `props.setProperty("database.fetchSize", ...)`. ### Fix In each affected `SourceConfigFactory.create()`, add a `props.setProperty()` call **before** `dbzProperties.putAll()`: - `OracleSourceConfigFactory`: `props.setProperty("query.fetch.size", String.valueOf(fetchSize))` - `SqlServerSourceConfigFactory`: `props.setProperty("query.fetch.size", String.valueOf(fetchSize))` - `Db2SourceConfigFactory`: `props.setProperty("query.fetch.size", String.valueOf(fetchSize))` - `PostgresSourceConfigFactory`: `props.setProperty("snapshot.fetch.size", String.valueOf(fetchSize))` Placing the `setProperty` before `dbzProperties.putAll()` ensures that explicit user overrides via `debezium.query.fetch.size` or `debezium.snapshot.fetch.size` still take precedence. ### Testing Added `SourceConfigFactoryTest` for all 4 connectors (12 tests total), verifying: 1. Custom `fetchSize` is correctly propagated to Debezium properties 2. Default `fetchSize` (1024) is propagated correctly 3. User-provided `debeziumProperties` can still override the propagated value All tests pass. Spotless and Checkstyle checks pass. ## Test Plan - [x] `OracleSourceConfigFactoryTest` — 3 tests pass - [x] `SqlServerSourceConfigFactoryTest` — 3 tests pass - [x] `PostgresSourceConfigFactoryTest` — 3 tests pass - [x] `Db2SourceConfigFactoryTest` — 3 tests pass - [x] Spotless (Google Java Format 1.24.0 AOSP) check passes - [x] Checkstyle check passes - [x] Validated workaround (`debezium.query.fetch.size: 10000`) on production Oracle 19c RAC environment: throughput improved from ~440 rows/s to ~3,500 rows/s Made with [Cursor](https://cursor.com) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
