stinger1206 opened a new pull request, #115:
URL: https://github.com/apache/doris-kafka-connector/pull/115
Fixes #114
## Problem 1: string-prefix writer selection can aggregate unrelated
topic-partitions
`DorisDefaultSinkService` keyed its writer map by strings
(`topic_partition[_table]`) and
selected writers with `entry.getKey().startsWith(topic + "_" + partition)`.
That matches
writers of *other* topic-partitions whenever one key is a string-prefix of
another:
partition 1 vs partition 10 (`topic_test_1` prefixes `topic_test_10`), and
topic `X`
partition 1 vs topic `X_1` partition 0. Both `getOffset()` and `commit()`
were affected:
wrong offset aggregates, and a commit for one partition also committed
another partition's
writers.
## Problem 2: MAX offset aggregation across writers of one topic-partition
With `record.tablename.field`, one topic-partition fans out to one writer
per table.
`getOffset()` reported `max(writer.getOffset())` — safe only for the most
advanced writer;
records still buffered in a lagging writer are below it and lost after a
crash.
(`max -> min` is not a fix: an idle table writer would stall the partition
forever.)
## Problem 3: a failed flush discarded its buffered records
`DorisWriter.flushBuffer()` and the size/count-triggered flush in
`insertRecord()` replaced
the active buffer *before* flushing:
```java
RecordBuffer tmpBuff = buffer;
this.buffer = new RecordBuffer();
flush(tmpBuff); // throws -> the records are no longer tracked
```
Scenario (two `put()` calls): put() #1 buffers offset 100 for table B and
returns
successfully; put() #2 routes offset 101 to table A and triggers B's
time-based flush,
which fails. B's buffer — holding offset 100 from the *first, successful*
put() — is
discarded. Kafka Connect redelivers only put() #2's batch. When A eventually
commits past
100, the watermark passes B's lost record: silent data loss the watermark
cannot see, because
it only inspects records *currently* in buffers.
**Fix:** swap the buffer only *after* a successful flush. A failed flush
keeps its records
buffered, so the watermark (min first-buffered offset) stays pinned below
them until they
are durably persisted. `insertRecord` additionally skips redelivered records
at or below
the buffer's last offset, so Kafka Connect's redelivery of the failed put()
batch does not
duplicate already-buffered records. `AsyncStreamLoadWriter` (combined mode)
already swapped
after enqueue and needed no change.
## Problem 4: early return skipped writer commit-state finalization
`DorisWriter.getOffset()` is not a pure accessor: `StreamLoadWriter` advances
`committedOffset` from `flushedOffset` and clears its `committableList`;
`CopyIntoWriter`
clears its uploaded-file list. Returning the buffered-watermark early —
without calling any
writer's `getOffset()` — left completed transactions pending; the next
preCommit cycle would
re-commit them.
**Fix:** `DorisDefaultSinkService.getOffset()` now finalizes *every* writer
of the
topic-partition first (calling `getOffset()` on each), then computes the
aggregate watermark.
## Change
- Writers are keyed by a structured `WriterKey(topic, partition, table)` with
`equals`/`hashCode`; `getOffset()` and `commit()` select writers by exact
key.
- Watermark rule: if any writer of the topic-partition still buffers
records, the
**minimum first-buffered offset** across writers (nothing at/after it is
claimed as
consumed); otherwise the **max of the writers' committed offsets** (every
record of every
table up to it is durably in Doris).
## Lifecycle analysis (verified, not assumed)
- Ordering: `preCommit` -> `sink.commit()` (durable flush or 2PC commit,
throws on failure
-> no offsets returned) -> `sink.getOffset()`.
- In-flight window: default-service writers (`StreamLoadWriter`,
`CopyIntoWriter`) flush
synchronously on the Connect worker thread; a failed flush throws out of
`put()` and now
preserves its buffer. The async writer belongs only to
`DorisCombinedSinkService`, whose
offset accounting is unchanged here (see #111 / #113 for combined mode).
- Crash-consistency: any undurable record is either still buffered (pins the
watermark) or
its flush threw (put() fails; Kafka Connect commits nothing past it; on
restart,
consumption resumes from the last committed offset and the records are
re-loaded —
at-least-once; a lost HTTP response leaves the load outcome unknown, so
idempotent
writes (primary-key upserts) are required to be duplicate-free).
- Single-table routing: identical when the buffer is empty; with a non-empty
buffer the new
value is the first buffered offset — strictly safer (the old value could
pass unflushed
records).
- Concurrency: buffer/offset mutations happen on the Connect worker thread
(put/preCommit
are sequential per task); offsets are AtomicLong-backed.
## Relation to PRs #112 and #113
Independent and file-disjoint: #112 changes only the stream-load response
handling
(`DorisStreamLoad`/`AsyncDorisStreamLoad`/`DataLoad`), #113 changes only the
combined-flush
failure latch (`AsyncDorisStreamLoad`). This PR changes
`DorisDefaultSinkService`,
`DorisCombinedSinkService` (writer-map key only), and `DorisWriter`. The
regression tests
inject failures via Doris `Failed` JSON responses, which fail the load on
pristine master —
no dependency on #112's non-200 handling.
## Tests
`TestDorisDefaultSinkServiceGetOffset` (14 tests) drive **real
`StreamLoadWriter`s** (and
`CopyIntoWriter` in the COPY INTO tests) through the real insert -> buffer
-> flush -> commit
lifecycle; only the HTTP layer is mocked (plus `DorisCommitter`/`CopyLoad`
via existing
test seams, and one forced `shouldFlush()` via spy in the time-path test):
- `partitionIdentityIsolationPartitions1And10` /
`topicPrefixCollisionIsolation` — exact
selection (old code crossed partitions and topics, in `getOffset` and
`commit`);
- `commitIsolationPartitions1And10` / `commitIsolationTopicPrefixCollision`
— with both
conflicting writers present (partitions 1+10 of one topic; topic `X` p1 +
topic `X_1`
p0), a `commit()` for partition 1 only commits partition 1's writer and
never the other
writer. Against the original implementation this fails: the string-prefix
selection
committed both writers, the wrong one even with partition 1's id as the
argument;
- `multiTableRoutingWatermarkBlocksOnSlowTable` — interleaved multi-table
routing
(0,2,4 -> A; 1,3 -> B): while B buffers, watermark = 1 (MAX = 5 would lose
B's records);
after B commits, watermark = max committed;
- `failedFlushIsNeverAcknowledged` — a failed load pins the watermark; the
preserved
buffer re-flushes successfully on retry;
- `failedSizeFlushPreservesBufferedRecordsAcrossPutBatches` /
`failedTimeFlushPreservesBufferedRecordsAcrossPutBatches` —
cross-put-batch loss
regression for both flush triggers: put #1 buffers 100 for B; put #2's
flush of B fails;
watermark pinned at 100; redelivered put #2 does not duplicate; B's later
flush persists
100 and the watermark advances exactly past durable data;
- `committedWritersFinalizedDespiteBufferedWriter` — two preCommit cycles:
while B buffers,
A's committables are cleared each cycle (no pending re-commit); once B is
durable the
watermark advances to the max committed offset;
- `idleWriterDoesNotStallWatermark` — an idle writer neither stalls nor
shrinks the
watermark;
- `twoPhaseCommitPartialCommitFailureRetriesIdempotently` — 2PC enabled: A's
transaction
commits, B's fails -> preCommit aborts before any offset is returned; the
retry re-commits
B and A idempotently; only then the watermark advances to the durable
state;
- `copyIntoFailedUploadPreservesBufferedRecords` /
`copyIntoCopyFailureRetriesBeforeAdvancingOffsets` — COPY INTO: a failed
upload preserves
the buffer (watermark pinned); upload-success + COPY-failure keeps the
file pending, the
retry executes it exactly twice, and the file list is finalized exactly
once;
- `dorisSinkTaskPutPreCommitLifecycleWithFailedFlush` — the real
`DorisSinkTask.put()` /
`preCommit()` entry points: a failed load during a later put() surfaces as
`RetriableException`, redelivery does not duplicate, and preCommit()
returns the safe
watermark (101 while A still buffers; 102 after everything is durable).
`DorisCombinedSinkServiceTest` and `TestDorisSinkService` are adapted to the
new writer-map
key type (semantics unchanged).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]