wgzhao commented on issue #13108:
URL: https://github.com/apache/seatunnel/issues/13108#issuecomment-6596014225
Thanks Daniel, both points are right, and I have updated #13111 with the
revised design.
- **2PC**: confirmed. Every sub-writer currently builds `prefix +
checkpointId`, so the collision is deterministic at each checkpoint, and
`abortPreCommit` is prefix-based, so one sub-writer can abort another
sub-writer's pre-committed transaction. My first patch only fixed the non-2PC
path.
- **Prefix compatibility**: the revised design keeps
`DorisSinkState.labelPrefix` as the recovery anchor:
1. Each writer gets `basePrefix + "_" + randomToken`; the token is part of
the state's `labelPrefix` (state format unchanged).
2. On restore, the writer uses the persisted prefix for the initial
`abortPreCommit`, then uses a freshly generated prefix for the new run. An
old-format state still aborts its old shared prefix; the new run's labels are
unique.
3. `DorisStreamLoad.abortPreCommit` now builds the label from the prefix
passed by the caller. Previously it used the stream load's own
`LabelGenerator`, so passing the restored prefix alone would have been ignored
by the actual pre-commit request.
4. 2PC labels stay `prefix + checkpointId` (deterministic per
writer/checkpoint); non-2PC labels additionally keep the per-call sequence.
5. No `MultiTableSink` changes; the fix is confined to `connector-doris`.
Tests added in #13111:
- two writers with the same sink table, job id and subtask index produce
distinct 2PC labels;
- the same for non-2PC labels;
- a writer restored from a persisted state calls `abortPreCommit` with the
persisted prefix and starts the new run with a different prefix;
- the pre-commit request uses the prefix passed to `abortPreCommit`, not the
stream load's generator;
- the E2E covers two source tables writing to one Doris table.
The full connector unit suite passes locally (162 tests). Thanks for the
#13060 pointer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
