DanielLeens commented on issue #13108: URL: https://github.com/apache/seatunnel/issues/13108#issuecomment-6595074220
Hi @wgzhao, thanks for the very thorough root-cause write-up. I checked it against the current `dev` code and the analysis holds. **Confirmed on `dev`** - `DorisSinkWriter` builds `labelPrefix` from `sink.label-prefix` + sink table path + job id + `context.getIndexOfSubtask()` (`DorisSinkWriter.java` constructor, around L129-L137). - In `MultiTableSink.createWriter`, every source table gets its own writer with `index = subtaskIndex * replicaNum + i` (`MultiTableSink.java` L143-L152). That index is the same for every source table, so when N source tables map to one Doris table, all N sub-writers end up with an identical prefix. - `LabelGenerator.generateLabel` then only appends `System.currentTimeMillis()` in non-2PC mode, and `snapshotState`, `write` (batch-size flush) and `timerFlush` all call it, so same-millisecond collisions across sub-writers are expected. **One point to add to the scope (inferred from the code, not reproduced):** the 2PC path is likely affected too, and deterministically. With `sink.enable-2pc = true` every sub-writer generates `labelPrefix + "_" + checkpointId`, so two sub-writers sharing a prefix would request the same label at the same checkpoint. `abortPreCommit(labelPrefix, lastCheckpointId + 1)` is also prefix-based, so on restore one sub-writer could abort another's pre-committed transaction. A per-call random suffix only fixes non-2PC; the uniqueness should come from a stable per-sub-writer discriminator in the prefix, which keeps 2PC labels deterministic as you intended. **Compatibility constraint for the PR:** `DorisSinkState` persists `labelPrefix`, but on restore the writer recomputes the prefix and uses that for `abortPreCommit`. If the prefix format changes for every job, existing 2PC jobs restored after an upgrade would no longer abort their old pre-committed labels. Please either keep the prefix unchanged for the single-sub-writer case, or abort with the restored prefix from state. Since you offered a PR, please go ahead. Suggested scope: 1. Unique labels per sub-writer in both modes, confined to `connector-doris` (no change to `MultiTableSink` needed). 2. Unit tests: two writers sharing sink table, job id and subtask index produce distinct labels in non-2PC and 2PC; restore from an old-format state still aborts the old prefix. 3. If feasible, extend the existing Doris E2E with a two-source-tables-to-one-sink-table case. For reference, #13060 is addressing the same class of problem for Kafka transactional IDs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
