stinger1206 opened a new pull request, #113:
URL: https://github.com/apache/doris-kafka-connector/pull/113
Fixes #111
## Problem
In combined-flush mode a background stream load runs on a single executor
thread
(`AsyncDorisStreamLoad.LoadAsyncExecutor`). When a load fails (BE
restarting, connection
refused, a timeout, a failed Doris load result), the failure is dropped
together with its
data, and the next offset commit runs past the lost records:
1. `LoadAsyncExecutor.run()` catches the failure, stores it in `exception`,
clears
`flushQueue` and exits. The failed `RecordBuffer` is gone from memory —
the writer
already swapped its own buffer when it enqueued it
(`AsyncStreamLoadWriter.flushBuffer`).
2. The next `put()` calls `insert()`, which throws from
`checkFlushException()`.
`DorisSinkTask.put()` catches it, calls `sink.init()` — which restarts
the executor and
resets `exception` to null — and throws `RetriableException`.
3. Kafka Connect re-delivers only the batch of that `put()`. The records of
the failed
buffer came from earlier `put()` calls and are never re-delivered.
4. The next `preCommit()` flushes the new buffer, and
`DorisCombinedSinkService.getOffset()`
returns the offsets from `topicPartitionOffset` — the last offsets *seen*
by `insert()`,
not confirmed by Doris. The commit succeeds past the lost records.
The reporter's reproduction (issue #111) lost 29-39% of all rows this way
while the tasks
stayed RUNNING.
## Change
`AsyncDorisStreamLoad` now latches the failure for the lifetime of the
object:
- `LoadAsyncExecutor.run()` sets a `loadFailed` latch in its catch blocks,
next to the
existing `exception` reference and the queue clear.
- `checkFlushException()` throws not only while `exception` is set, but also
while the latch
is set. `start()` keeps restarting the executor thread and keeps clearing
`exception`
(unchanged), but it cannot clear the latch.
Consequences, in order of the four steps above:
- After the failure, every `insert()`, `flush()` and `forceFlush()` on the
affected writer
throws `DorisException` (RuntimeException), including across executor
restarts.
- `DorisSinkTask.put()` retries stay `RetriableException` until
`max.retries` is exhausted;
then the original exception escapes `put()` and the task fails. Kafka
Connect does not
restart a failed task by itself — the task stays FAILED until it is
restarted (manually
or by external automation). On restart, new writers are created and
consumption resumes
from the last committed offsets.
- While the task is still running, `preCommit()` can never return the unsafe
offsets: it
calls `sink.commit(offsets)` first, which calls `commitFlush()` →
`forceFlush()` →
`checkFlushException()` and throws before `getOffset()` is consulted.
Kafka Connect's
`WorkerSinkTask.commitOffsets()` catches such an exception, skips the
commit and rewinds
to the last committed offsets; it does not fail the task — the task keeps
running and
keeps failing on `put()` and `preCommit()` until the `put()` retries are
exhausted.
Two different restarts are involved, and only the second one recovers:
- **Executor restart** (`DorisSinkTask.put()` → `sink.init()` →
`AsyncStreamLoadWriter.start()` → `AsyncDorisStreamLoad.start()`):
re-`execute`s the
executor runnable and clears `exception`, but **cannot clear the
`loadFailed` latch**.
It keeps the connector failing on every redelivered `put()` and on every
`preCommit()`;
it is not a recovery.
- **Kafka Connect task-instance restart** (after the failed task is
restarted — manually
or by external automation — or after a rebalance): creates new writers and
therefore new
`AsyncDorisStreamLoad` instances with fresh latches, and resumes
consumption from the
last committed offsets — this is the recovery, and it re-delivers
everything since the
last commit.
A failed load does not prove the records were not written: if the HTTP
response was lost
(timeout, connection cut), the load outcome is unknown and the re-delivered
records may
already be in Doris. Recovery is therefore at-least-once; with a primary-key
table and
Debezium-style upserts, re-delivery is idempotent (duplicate rows are
otherwise possible
for tables without a unique key).
`forceFlush()` (the `preCommit` barrier) also cannot miss an in-flight
failure:
`waitAsyncLoadFinish()` enqueues two eof markers into the capacity-1 flush
queue served by
a single sequential executor thread, so the second marker's `put` cannot
complete until
the executor has polled the first — which happens only after `load()`
returned. If that
load failed, `exception`/`loadFailed` are set before the blocked `put`
unblocks, and the
`checkFlushException()` right after it throws. Covered by
`forceFlushWaitsForInFlightLoadAndSurfacesItsFailure`, which pins the HTTP
request with a
`CountDownLatch`, calls `forceFlush()` on a separate thread, and asserts it
neither
returns while the request is in flight nor returns normally after the
request fails.
This is deliberately the "fail the commit/task" variant of the fix suggested
in #111: the
failed buffer is not re-sent (its records are no longer available for
retry), so the only
safe recovery is replaying from the last committed offsets.
## Relation to PR #112
PR #112 makes every unexpected HTTP response (non-200, null entity, empty
body) fail the
load in the first place. This PR is independent of it: it handles every
failure that does
throw out of the load path (connection refused, timeout, Doris
`Failed`/`Cancelled` JSON).
With both PRs, the proxy-503 scenario of #110 also reaches this latch
instead of returning
normally.
## Tests
- `TestAsyncStreamLoadFailureLatch` (5 tests):
- a Doris `Failed` load result latches: the failure surfaces on
`forceFlush()`, and
survives `start()` (the `sink.init()` executor restart) for
`checkException()`,
`flush()` and `forceFlush()`;
- the same for an `IOException` from the HTTP client (connection refused
while the BE
restarts);
- `forceFlush()` cannot return while a load is still in flight, and
surfaces that load's
failure when it completes: the HTTP request is pinned by a
`CountDownLatch`, and a
separate thread calling `forceFlush()` stays blocked until the request
fails, then
receives the `DorisException`;
- a normal `close()` interrupting an idle executor poll does not latch;
- successful loads do not latch: two full flush cycles succeed, and a
restart is
harmless.
- `TestCombineFlushFailedLoadCommits` (2 tests, real
`DorisCombinedSinkService` /
`DorisSinkTask`, only the HTTP layer mocked via the injectable-httpClient
constructor):
- service level: after a failed load, `commit()` throws, and keeps
throwing across
`init()` and further inserts, while `getOffset()` still reports the
last-seen offset —
reachable only through a successful `commit()`, which `preCommit` runs
first;
- task level: `put()` buffers, `preCommit()` throws, each re-delivered
`put()` throws
`RetriableException` until `max.retries` (2 in the test) is exhausted,
then the
original `DorisException` escapes (the task fails; on restart,
consumption resumes
from the last committed offsets), and `preCommit()` still never returns
offsets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]