eric666666 commented on code in PR #4466:
URL: https://github.com/apache/flink-cdc/pull/4466#discussion_r3548775162
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/reader/MySqlPipelineRecordEmitter.java:
##########
@@ -130,11 +140,13 @@ public void applySplit(MySqlSplit split) {
protected void processElement(
SourceRecord element, SourceOutput<Event> output, MySqlSplitState
splitState)
throws Exception {
- if (shouldEmitAllCreateTableEventsInSnapshotMode && isBounded) {
- // In snapshot mode, we simply emit all schemas at once.
+ if (shouldEmitAllCreateTableEventsInSnapshotMode
+ && shouldBatchEmitCreateTableEvents()) {
+ // In snapshot mode or batch emit mode, we emit all schemas at
once.
Review Comment:
Thanks @yuxiqian. The goal of this PR is specifically the INITIAL-mode case:
without it, each snapshot split's low-watermark triggers a per-table
`CreateTableEvent` *interleaved with* the data stream. Each one makes the
downstream `SchemaOperator` emit a `FlushEvent`, which flushes whatever data is
currently buffered — so with thousands of tables the data stream is interrupted
thousands of times, and on Paimon/Iceberg that means thousands of tiny files +
throughput loss.
With the option on, all `CreateTableEvent`s are emitted together up front
(first record, before any `DataChangeEvent`). The `FlushEvent` count is
unchanged, but those flushes now hit an empty buffer (no data has flowed yet),
so no real data files are produced. Per-table JDBC (`SHOW CREATE TABLE`) is
also collapsed from N connections to one batched fetch.
The downstream per-event coordination you pointed at (one blocking RPC per
`CreateTableEvent`) is a separate concern; happy to address it in a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
