stinger1206 opened a new pull request, #112:
URL: https://github.com/apache/doris-kafka-connector/pull/112

   # [Fix] Fail stream load loudly on unexpected HTTP responses (non-200, null 
entity, empty body)
   
   Fixes #110
   
   ## Problem
   
   `DorisStreamLoad.load` and `AsyncDorisStreamLoad.LoadAsyncExecutor.load` 
only parsed the
   response when `statusCode == 200 && entity != null`. On any other response — 
proxy 502/503/504
   when a BE dies, FE 500, or a 200 with null entity — the method returned 
normally, upper layers
   advanced the Kafka offsets, and the batch was silently lost (see issue for 
the full loss-path
   matrix). A 200 with an empty body threw only a cryptic Jackson 
`MismatchedInputException`.
   
   ## Error-propagation analysis (verified, not assumed)
   
   - Sync path: `DorisStreamLoad.load` throws `StreamLoadException` ->
     `StreamLoadWriter.flush` rethrows -> `DorisWriter.flushBuffer` rethrows ->
     `DorisSinkTask.put` lets it escape -> Kafka Connect fails the task; no 
offsets are committed.
   - Async path (combined-flush mode): `LoadAsyncExecutor.run` catches the 
exception, stores it
     in `exception` (AtomicReference), clears the queue and stops the thread. 
The next
     `flush`/`forceFlush`/`checkException` rethrows it, so the failure is no 
longer silent and
     `put()`/`preCommit()` surface it instead of returning success.
     Note: in combined-flush mode the retry in `DorisSinkTask.put()` restarts 
the executor and
     resets that exception, which is a separate offset-accounting problem 
tracked in #111; this
     PR only makes every unexpected HTTP response fail the load instead of 
returning normally.
   - 2PC: after a successful load, `commit` -> `DorisCommitter.commit` (with 
backoff) throws on
     failure before offsets advance.
   
   Regression tests drive both paths end-to-end:
   `streamLoadWriterFlushPropagatesHttpFailure` (sync writer lifecycle, asserts 
`getOffset()==0`
   after a 503) and `asyncLoadThreadSurfacesHttpFailureOnNextFlush` (real 
background executor
   thread, asserts the 503 surfaces on `forceFlush`).
   
   ## Change
   
   Both load methods now, in order:
   
   1. throw `StreamLoadException` on any non-200 status — message includes 
host, table, status
      code, reason phrase and a bounded body snippet (see below);
   2. throw on a 200 with null entity;
   3. throw a clear exception on a 200 with an empty body;
   4. unchanged: parse the body, throw on unparsable/Doris-`Failed` status, keep
      `Success` / `Publish Timeout` as success.
   
   ## Bounded error-body read (review round 2)
   
   The snippet helper `errorBodySnippet(HttpEntity)` in `DataLoad` reads at most
   **512 + 1 bytes** (the +1 detects truncation) directly from the entity's 
stream — the body is
   never read fully, so a hostile or misbehaving server or proxy cannot force 
an unbounded memory
   allocation on the error path. Verified by `errorSnippetReadsBoundedBytes`, 
which streams a
   200 KB body through a counting entity and asserts at most 513 bytes were 
consumed.
   The 200-success path still reads the full body (required to parse the Doris 
JSON result;
   upstream behavior, Doris responses are small) — unchanged by this fix.
   
   The snippet bounds length only; it does not redact sensitive data that a 
server or
   intermediary may include in an error page. Users should treat connector 
error logs
   accordingly.
   
   ## Tests
   
   `TestStreamLoadResponseHandling` (14 tests): for the sync and async paths 
each — success still
   passes; non-200 (503/502 with body), null entity, empty body, and Doris 
`Failed` JSON all
   throw with the expected message fragments; direct tests of the snippet helper
   (`<no body>`, `<empty body>`, `<unreadable body>`, long-body truncation, 
bounded read);
   and the two end-to-end propagation tests above.
   
   ## Loss-path matrix (verified against this change)
   
   | BE death as seen by connector | Before | After |
   |---|---|---|
   | TCP reset / truncated body | IOException | IOException (unchanged) |
   | Proxy 502/503/504, FE 500 | **silent loss** | throws + host/status/body in 
message |
   | 200 + null entity | **silent loss** | throws |
   | 200 + empty body | cryptic Jackson error | throws, clear message |
   | 200 + Doris `Failed` JSON | throws | throws (unchanged) |
   | Load ok, BE dies before 2PC commit | committer retries/throws | unchanged |
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to