wgzhao opened a new issue, #13108:
URL: https://github.com/apache/seatunnel/issues/13108
### Search before asking
- [x] I had searched the issues and found no open issue with a root-cause
analysis for this problem. Two closed reports describe the same symptom without
a resolution: #7050 (2.3.5, multi-table CDC writing to one Doris table) and
#9048 (2.3.9, `LABEL_ALREADY_EXISTS`). Both were closed as stale. This issue is
not a duplicate: it identifies the label-generation path that causes the
collision on SeaTunnel 3.0.0 and proposes a fix.
### What happened
A MySQL-CDC streaming job reads 12 monthly shard tables with a
`table-pattern` and writes all of them into a single Doris table. `parallelism
= 1`, `sink.enable-2pc = false`, default `doris.batch.size = 1024`, checkpoint
interval 5s.
At a checkpoint boundary, two Doris sub-writers generated the same
stream-load label. Doris rejected the second load with `LABEL_ALREADY_EXISTS`.
The Doris sink writer treats that status as fatal, so the whole pipeline
failed. Restoring from the savepoint replayed the same window and the failure
repeated within seconds.
The job had been running for days with only one active source shard. The
failure started when a second shard table began receiving changes. The same
label-generation logic exists in 2.3.13, so this is a latent defect rather than
a 3.0.0-only regression; it is reported here on 3.0.0.
### SeaTunnel Version
3.0.0 (Zeta engine, single node; Java 8).
### SeaTunnel Config
```hocon
env {
parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 5000
read_limit.rows_per_second = 40000
}
source {
MySQL-CDC {
server-id = 5401
username = "repl"
password = "***"
database-names = ["shop"]
table-pattern = "shop\\.orders_2021.*" # 12 monthly shard tables
url = "jdbc:mysql://mysql.example.com:3306/?autoReconnect=true"
schema-changes.enabled = false
startup.mode = "latest"
}
}
sink {
Doris {
fenodes = "doris-fe.example.com:8030"
query-port = 19030
username = "loader"
password = "***"
database = "analytics"
table = "orders_all" # all 12 source tables write here
schema_save_mode = "ignore"
sink.enable-2pc = "false"
sink.label-prefix = "orders_label"
doris.config = {
format = "json"
read_json_by_line = "true"
}
}
}
```
### Running Command
```shell
bin/seatunnel.sh -c job/orders_cdc_demo.conf -n orders_cdc_demo -r
1234567890123456789 --async
```
### Error Exception
Anonymized excerpt (`T` = checkpoint completion time; the label suffix
`1700000000452` is the millisecond timestamp at which both writers generated
the label):
```log
2026-01-01 00:00:10.452 INFO [CheckpointCoordinator] - pending checkpoint
notify finished, job id: 1234567890123456789, checkpoint id: 100
# Two sub-writers generated the same label at 00:00:10.452:
2026-01-01 00:00:10.540 INFO [DorisStreamLoad] - stream load started for
orders_label_analytics_orders_all_1234567890123456789_0_1700000000452
2026-01-01 00:00:12.210 INFO [DorisStreamLoad] - load Result {
"TxnId": 100001,
"Label":
"orders_label_analytics_orders_all_1234567890123456789_0_1700000000452",
"Status": "Success",
"NumberTotalRows": 1024,
...
}
2026-01-01 00:00:12.568 INFO [DorisStreamLoad] - stream load started for
orders_label_analytics_orders_all_1234567890123456789_0_1700000000452
2026-01-01 00:00:12.600 INFO [DorisStreamLoad] - load Result {
"TxnId": -1,
"Label":
"orders_label_analytics_orders_all_1234567890123456789_0_1700000000452",
"Status": "Label Already Exists",
"Message": "[LABEL_ALREADY_EXISTS]TStatus: errCode = 2, detailMessage =
Label [...] has already been used, relate to txn [100001], status [VISIBLE].",
...
}
2026-01-01 00:00:12.610 ERROR [DorisSinkWriter] - stream load finished
unexpectedly: [LABEL_ALREADY_EXISTS] ...
2026-01-01 00:00:15.493 ERROR [SubPlan] - Task ... Failed in Job
orders_cdc_demo (1234567890123456789), Pipeline: [(1/1)], Begin to cancel other
tasks in this pipeline.
```
The second load starts 2.5 s after the first because `DorisStreamLoad` only
opens the HTTP load when the first record of the buffer arrives. The label
itself was generated at 00:00:10.452 by both writers. On every restore, the
engine also logs 12 `DorisSinkWriter - labelPrefix orders_label` lines within a
few milliseconds, one per source table.
### Root cause
Relevant code on `v3.0.0`:
-
[`DorisSinkWriter`](https://github.com/apache/seatunnel/blob/v3.0.0/seatunnel-connectors-v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/sink/writer/DorisSinkWriter.java#L121-L129)
builds the label prefix from the configured prefix, the **sink** table path,
the job id and `context.getIndexOfSubtask()`. In a multi-table sink where every
source table maps to one Doris table, all sub-writers get the same prefix.
`getIndexOfSubtask()` is the engine subtask index, not a per-sub-sink index.
-
[`LabelGenerator.generateLabel`](https://github.com/apache/seatunnel/blob/v3.0.0/seatunnel-connectors-v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/sink/writer/LabelGenerator.java#L30-L34)
uses `prefix + "_" + System.currentTimeMillis()` for non-2PC. Two sub-writers
that call it in the same millisecond produce identical labels.
-
[`MultiTableSink.createWriter`](https://github.com/apache/seatunnel/blob/v3.0.0/seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSink.java#L125-L143)
creates one writer per source table.
- At every checkpoint,
[`MultiTableSinkWriter.snapshotState`](https://github.com/apache/seatunnel/blob/v3.0.0/seatunnel-api/src/main/java/org/apache/seatunnel/api/sink/multitablesink/MultiTableSinkWriter.java#L681-L721)
iterates all sub-writers and each
[`DorisSinkWriter.snapshotState`](https://github.com/apache/seatunnel/blob/v3.0.0/seatunnel-connectors-v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/sink/writer/DorisSinkWriter.java#L299-L304)
calls `startLoad(generateLabel(...))`. This makes several label generations
within the same millisecond likely.
-
[`DorisSinkWriter.flush`](https://github.com/apache/seatunnel/blob/v3.0.0/seatunnel-connectors-v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/sink/writer/DorisSinkWriter.java#L274-L286)
and
[`checkDone`](https://github.com/apache/seatunnel/blob/v3.0.0/seatunnel-connectors-v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/sink/writer/DorisSinkWriter.java#L326-L341)
treat any status other than `Success`/`Publish Timeout` as fatal.
Doris requires labels to be unique per database; a second load with a used
label returns `LABEL_ALREADY_EXISTS`. The two loads carry different data
(different source tables), so treating the second as success would silently
drop data. The only correct fix is to generate unique labels.
Why it is intermittent: the collision requires two sub-writers to generate a
label in the same millisecond and both to start their load. With one active
source table there is no collision. In the reported job, 2 of the 12 shard
tables were active; one had far more changes than the other, and a single
change on the second table was enough to trigger the failure at the next
checkpoint.
### Reproduction
1. Create two source tables under `shop` (for example `orders_202101`,
`orders_202102`) and one Doris table `analytics.orders_all`.
2. Run a MySQL-CDC job with `table-pattern = "shop\\.orders_2021.*"`, `table
= "orders_all"`, `sink.enable-2pc = false`, the default batch size and a 5 s
checkpoint interval.
3. Write continuously to both source tables (or replay a binlog backlog) so
both sub-writers flush at the same checkpoints. The job fails with
`LABEL_ALREADY_EXISTS`.
4. Deterministic micro-reproduction of the label generator: create two
`LabelGenerator` instances with the same prefix and call `generateLabel(0)`
from both in a tight loop; labels duplicate when calls land in the same
millisecond. With a unique per-call suffix, 2,000,000 generated labels were
unique in a local check.
### Expected behavior
Every stream-load label generated by the sub-writers of one job must be
unique, even when several sub-writers share the same sink table, job id and
subtask index and generate labels in the same millisecond. A multi-table CDC
job writing to a single Doris table must run without `LABEL_ALREADY_EXISTS`.
2PC labels (`prefix + checkpointId`) are used by abort/recovery and should
remain deterministic.
### Workaround
Split the job into one job per source table so each job gets a different job
id in the label prefix. This avoids the collision but multiplies jobs and
MySQL-CDC server ids. It is not a fix.
### Related issues
- #7050 (2.3.5): multi-table CDC to one Doris table, `Label already exists`;
closed stale.
- #9048 (2.3.9): `LABEL_ALREADY_EXISTS`; closed stale.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
