zhangshenghang opened a new pull request, #12692:
URL: https://github.com/apache/seatunnel/pull/12692
## Purpose of this pull request
Fixes a silent data-loss path in the StarRocks sink: a Stream Load that
returns **Publish Timeout** is currently trusted as success () and the buffered
batch is released **without querying the transaction's final state**. This
closes the residual gap left by #11431.
## Background
When a StarRocks cluster has just been restarted (e.g. staggered FE
restarts), a stream load transaction can report and **still be aborted
afterwards**. The connector then clears its buffer while the rows never become
durable:
- the job stays and every checkpoint keeps succeeding (no exception is
thrown),
- the data for that flush window is silently lost, one table and one
contiguous time range at a time.
We reproduced this against the real connector classes with a stub StarRocks
HTTP endpoint (customer-like config, ): → 1 PUT, 0 calls, buffer cleared, no
exception — data lost.
## What this PR changes
A batch now has exactly two outcomes: **released only after StarRocks
confirms the label /**, or retained until the flush fails closed and the job
replays it from its checkpoint (retries reuse the same label, so an
already-committed load is never duplicated; PRIMARY-KEY tables are idempotent
under upsert).
1. `StarRocksStreamLoadVisitor#doStreamLoad`: removed the `Publish Timeout →
return true` fast path. Only `Success` is accepted directly and a plain `Fail`
still fails immediately; every other non-terminal response (`Publish Timeout`,
`Label Already Exists`, a reused-label failure, an unrecognized status) is
resolved through `get_load_state`:
- `VISIBLE`/`COMMITTED` → confirmed, batch released;
- `ABORTED` → the manager resends under a new label;
- unresolved after a bounded wait → exception, batch retained, checkpoint
replay.
2. `StarRocksStreamLoadVisitor#checkLabelState`: `UNKNOWN` states and
unreadable `get_load_state` responses are retried within the deadline instead
of failing immediately (a front-end recovering from a restart may briefly
report `UNKNOWN` for a committed label).
3. New `label_state_timeout_ms` sink option (default 3 minutes); the visitor
prefers an explicit test timeout, then the sink config, then the built-in
default.
4. `StarRocksSinkManager` is unchanged — it already retains `pendingFlush`
and fails closed since #11431.
## Tests
- Updated the two tests that locked in the old behaviour and added three
regression tests (`resolvesPublishTimeoutByConfirmingLabelState`,
`recreatesLabelWhenPublishTimeoutLabelIsAborted`,
`failsWhenPublishTimeoutLabelNeverResolves`).
- Full module suite: **28/28 pass** with this change; running the same suite
against unpatched `dev` fails **exactly the 4 new/updated tests**, so they
reliably capture the defect.
- End-to-end verification with a stub StarRocks endpoint over real HTTP:
| Scenario | Before | After |
|---|---|---|
| Publish Timeout → VISIBLE | silent loss | released after state confirmed |
| Publish Timeout → ABORTED | silent loss | resent under a new label,
Success |
| Publish Timeout → never resolves | silent loss | fail-closed, batch
retained, replay |
| Fail + `Label already used` → VISIBLE | ok | ok (verified) |
| HTTP 500 → retry same label → Success | ok | ok |
| Unrecognized status → VISIBLE | fail (task restart) | released after state
confirmed |
| Success | ok | ok |
## Note
Connector option docs live in the seatunnel-docs repo;
`label_state_timeout_ms` should be added there:
```
| label_state_timeout_ms | long | no | 180000 | Total time the sink polls
get_load_state to resolve a non-final stream load outcome (Publish Timeout,
Label Already Exists, reused-label failure) to VISIBLE/COMMITTED/ABORTED before
failing the flush, default is 3 minutes. |
```
## Check list
- [x] Code changed
- [x] Unit tests added / updated (`connector-starrocks`)
- [x] Integration tests not affected (no API change; `StarRocksSinkManager`
untouched)
- [ ] Documentation (seatunnel-docs, noted above)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
