stinger1206 opened a new pull request, #115:
URL: https://github.com/apache/doris-kafka-connector/pull/115

   
   Fixes #114
   
   ## Problem 1: string-prefix writer selection can aggregate unrelated 
topic-partitions
   
   `DorisDefaultSinkService` keyed its writer map by strings 
(`topic_partition[_table]`) and
   selected writers with `entry.getKey().startsWith(topic + "_" + partition)`. 
That matches
   writers of *other* topic-partitions whenever one key is a string-prefix of 
another:
   partition 1 vs partition 10 (`topic_test_1` prefixes `topic_test_10`), and 
topic `X`
   partition 1 vs topic `X_1` partition 0. Both `getOffset()` and `commit()` 
were affected:
   wrong offset aggregates, and a commit for one partition also committed 
another partition's
   writers.
   
   ## Problem 2: MAX offset aggregation across writers of one topic-partition
   
   With `record.tablename.field`, one topic-partition fans out to one writer 
per table.
   `getOffset()` reported `max(writer.getOffset())` — safe only for the most 
advanced writer;
   records still buffered in a lagging writer are below it and lost after a 
crash.
   (`max -> min` is not a fix: an idle table writer would stall the partition 
forever.)
   
   ## Problem 3: a failed flush discarded its buffered records
   
   `DorisWriter.flushBuffer()` and the size/count-triggered flush in 
`insertRecord()` replaced
   the active buffer *before* flushing:
   
   ```java
   RecordBuffer tmpBuff = buffer;
   this.buffer = new RecordBuffer();
   flush(tmpBuff);          // throws -> the records are no longer tracked
   ```
   
   Scenario (two `put()` calls): put() #1 buffers offset 100 for table B and 
returns
   successfully; put() #2 routes offset 101 to table A and triggers B's 
time-based flush,
   which fails. B's buffer — holding offset 100 from the *first, successful* 
put() — is
   discarded. Kafka Connect redelivers only put() #2's batch. When A eventually 
commits past
   100, the watermark passes B's lost record: silent data loss the watermark 
cannot see, because
   it only inspects records *currently* in buffers.
   
   **Fix:** swap the buffer only *after* a successful flush. A failed flush 
keeps its records
   buffered, so the watermark (min first-buffered offset) stays pinned below 
them until they
   are durably persisted. `insertRecord` additionally skips redelivered records 
at or below
   the buffer's last offset, so Kafka Connect's redelivery of the failed put() 
batch does not
   duplicate already-buffered records. `AsyncStreamLoadWriter` (combined mode) 
already swapped
   after enqueue and needed no change.
   
   ## Problem 4: early return skipped writer commit-state finalization
   
   `DorisWriter.getOffset()` is not a pure accessor: `StreamLoadWriter` advances
   `committedOffset` from `flushedOffset` and clears its `committableList`; 
`CopyIntoWriter`
   clears its uploaded-file list. Returning the buffered-watermark early — 
without calling any
   writer's `getOffset()` — left completed transactions pending; the next 
preCommit cycle would
   re-commit them.
   
   **Fix:** `DorisDefaultSinkService.getOffset()` now finalizes *every* writer 
of the
   topic-partition first (calling `getOffset()` on each), then computes the 
aggregate watermark.
   
   ## Change
   
   - Writers are keyed by a structured `WriterKey(topic, partition, table)` with
     `equals`/`hashCode`; `getOffset()` and `commit()` select writers by exact 
key.
   - Watermark rule: if any writer of the topic-partition still buffers 
records, the
     **minimum first-buffered offset** across writers (nothing at/after it is 
claimed as
     consumed); otherwise the **max of the writers' committed offsets** (every 
record of every
     table up to it is durably in Doris).
   
   ## Lifecycle analysis (verified, not assumed)
   
   - Ordering: `preCommit` -> `sink.commit()` (durable flush or 2PC commit, 
throws on failure
     -> no offsets returned) -> `sink.getOffset()`.
   - In-flight window: default-service writers (`StreamLoadWriter`, 
`CopyIntoWriter`) flush
     synchronously on the Connect worker thread; a failed flush throws out of 
`put()` and now
     preserves its buffer. The async writer belongs only to 
`DorisCombinedSinkService`, whose
     offset accounting is unchanged here (see #111 / #113 for combined mode).
   - Crash-consistency: any undurable record is either still buffered (pins the 
watermark) or
     its flush threw (put() fails; Kafka Connect commits nothing past it; on 
restart,
     consumption resumes from the last committed offset and the records are 
re-loaded —
     at-least-once; a lost HTTP response leaves the load outcome unknown, so 
idempotent
     writes (primary-key upserts) are required to be duplicate-free).
   - Single-table routing: identical when the buffer is empty; with a non-empty 
buffer the new
     value is the first buffered offset — strictly safer (the old value could 
pass unflushed
     records).
   - Concurrency: buffer/offset mutations happen on the Connect worker thread 
(put/preCommit
     are sequential per task); offsets are AtomicLong-backed.
   
   ## Relation to PRs #112 and #113
   
   Independent and file-disjoint: #112 changes only the stream-load response 
handling
   (`DorisStreamLoad`/`AsyncDorisStreamLoad`/`DataLoad`), #113 changes only the 
combined-flush
   failure latch (`AsyncDorisStreamLoad`). This PR changes 
`DorisDefaultSinkService`,
   `DorisCombinedSinkService` (writer-map key only), and `DorisWriter`. The 
regression tests
   inject failures via Doris `Failed` JSON responses, which fail the load on 
pristine master —
   no dependency on #112's non-200 handling.
   
   ## Tests
   
   `TestDorisDefaultSinkServiceGetOffset` (14 tests) drive **real 
`StreamLoadWriter`s** (and
   `CopyIntoWriter` in the COPY INTO tests) through the real insert -> buffer 
-> flush -> commit
   lifecycle; only the HTTP layer is mocked (plus `DorisCommitter`/`CopyLoad` 
via existing
   test seams, and one forced `shouldFlush()` via spy in the time-path test):
   
   - `partitionIdentityIsolationPartitions1And10` / 
`topicPrefixCollisionIsolation` — exact
     selection (old code crossed partitions and topics, in `getOffset` and 
`commit`);
   - `commitIsolationPartitions1And10` / `commitIsolationTopicPrefixCollision` 
— with both
     conflicting writers present (partitions 1+10 of one topic; topic `X` p1 + 
topic `X_1`
     p0), a `commit()` for partition 1 only commits partition 1's writer and 
never the other
     writer. Against the original implementation this fails: the string-prefix 
selection
     committed both writers, the wrong one even with partition 1's id as the 
argument;
   - `multiTableRoutingWatermarkBlocksOnSlowTable` — interleaved multi-table 
routing
     (0,2,4 -> A; 1,3 -> B): while B buffers, watermark = 1 (MAX = 5 would lose 
B's records);
     after B commits, watermark = max committed;
   - `failedFlushIsNeverAcknowledged` — a failed load pins the watermark; the 
preserved
     buffer re-flushes successfully on retry;
   - `failedSizeFlushPreservesBufferedRecordsAcrossPutBatches` /
     `failedTimeFlushPreservesBufferedRecordsAcrossPutBatches` — 
cross-put-batch loss
     regression for both flush triggers: put #1 buffers 100 for B; put #2's 
flush of B fails;
     watermark pinned at 100; redelivered put #2 does not duplicate; B's later 
flush persists
     100 and the watermark advances exactly past durable data;
   - `committedWritersFinalizedDespiteBufferedWriter` — two preCommit cycles: 
while B buffers,
     A's committables are cleared each cycle (no pending re-commit); once B is 
durable the
     watermark advances to the max committed offset;
   - `idleWriterDoesNotStallWatermark` — an idle writer neither stalls nor 
shrinks the
     watermark;
   - `twoPhaseCommitPartialCommitFailureRetriesIdempotently` — 2PC enabled: A's 
transaction
     commits, B's fails -> preCommit aborts before any offset is returned; the 
retry re-commits
     B and A idempotently; only then the watermark advances to the durable 
state;
   - `copyIntoFailedUploadPreservesBufferedRecords` /
     `copyIntoCopyFailureRetriesBeforeAdvancingOffsets` — COPY INTO: a failed 
upload preserves
     the buffer (watermark pinned); upload-success + COPY-failure keeps the 
file pending, the
     retry executes it exactly twice, and the file list is finalized exactly 
once;
   - `dorisSinkTaskPutPreCommitLifecycleWithFailedFlush` — the real 
`DorisSinkTask.put()` /
     `preCommit()` entry points: a failed load during a later put() surfaces as
     `RetriableException`, redelivery does not duplicate, and preCommit() 
returns the safe
     watermark (101 while A still buffers; 102 after everything is durable).
   
   `DorisCombinedSinkServiceTest` and `TestDorisSinkService` are adapted to the 
new writer-map
   key type (semantics unchanged).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to