JNSimba opened a new pull request, #70255: URL: https://github.com/apache/doris/pull/70255
### Background Streaming jobs read MySQL snapshot splits through JDBC before continuing with binlog events. The shared MySQL reader builds the source configuration used by snapshot readers in both read and write paths. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Bugfix Problem Summary: With the default JDBC options, a positive fetch size lets the driver buffer an entire snapshot split before rows reach the downstream queue. Large splits or wide rows can therefore exhaust the CDC client's heap before queue backpressure can help. This change enables the driver's row-streaming mode for snapshot queries. Most of the added code verifies the real driver's behavior and connection reuse. ### Details - **Frequency:** Affects snapshot queries using the default JDBC options; the memory impact depends on the split size and row width. - **Root cause:** The default positive fetch size of 1024 does not enable MySQL cursor fetching or row streaming on its own. - **Why existing tests missed it:** Small-data snapshot tests verified returned records without distinguishing a fully buffered result from a streaming result. - **Fix:** Set the shared MySQL source fetch size to `Integer.MIN_VALUE`, enabling row streaming with the existing forward-only, read-only snapshot statements. - **Other paths and branches:** Covers initial snapshots and snapshot-only reads through the shared reader configuration. Binlog consumption, offsets and checkpoint formats are unchanged. The inherited OceanBase reader shares this configuration path but was not tested against a live OceanBase instance. - **Behavior change:** Snapshot queries use row streaming, including when the JDBC URL supplies cursor-fetch, server-prepared-statement or default-fetch-size options. No new user setting is introduced. This does not bound memory retained elsewhere in the CDC pipeline. ### Testing Validated locally with JDK 17 and a MySQL 8.0 Testcontainers instance: - CDC client unit suite: 268 tests passed. The new configuration test failed with the original fetch size and passed after the fix. - `MySqlBasicReadITCase` and `MySqlSnapshotResumeITCase` passed, covering snapshot-to-binlog reading and snapshot split resume. - `MySqlSnapshotStreamingITCase`: four JDBC URL variants passed: default, `useCursorFetch=true`, `useServerPrepStmts=true`, and `defaultFetchSize=4096`. Each compares legacy fetch size 1024 with the generated production configuration through the snapshot statement builder. It checks the actual driver result implementation, ordered row contents, full consumption, early close, and subsequent reuse of the connection. - The legacy default result is `ResultsetRowsStatic`, with all 16 rows buffered before the first `next()`; the new result is `ResultsetRowsStreaming`. The legacy cursor-fetch variant uses `ResultsetRowsCursor`. - Removing the production streaming setting made all four integration variants fail; restoring it made all four pass. Command: `mvn -B -ntp -f fs_brokers/cdc_client/pom.xml test-compile failsafe:integration-test failsafe:verify -Dit.test=MySqlSnapshotStreamingITCase`. - Java compilation and the module's Spotless/Checkstyle validation passed. The earlier three-variant streaming test also passed five consecutive runs. ### Notes for Reviewers The integration fixture contains 16 rows with 256 KiB payloads (4 MiB total). It verifies driver behavior, not peak heap usage or throughput. The original large-data workload and a throughput benchmark were not run. Full package builds were attempted but blocked by missing local LLVM 20 and installed `protoc` prerequisites; Maven Java compilation succeeded. ### Release note MySQL streaming job snapshots now stream rows instead of caching an entire snapshot split in the JDBC driver by default, reducing driver-side memory usage for large splits and wide rows. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Partially verified: full package build, original large-data stress workload, throughput and live OceanBase compatibility remain unverified. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
