Manas-Bhole opened a new issue, #110: URL: https://github.com/apache/doris-kafka-connector/issues/110
**Versions** - doris-kafka-connector 26.1.0 (same code on master, 40b80e9) - Doris 4.1.5, Kafka Connect 4.3.0, Debezium 3.6.3 (Postgres source) - On Kafka 4.3 the connector needs commons-lang3 in its plugin directory (#109); these runs had it - Connector configured as in the "Synchronizing data captured by Debezium" guide (`converter.mode=debezium_ingestion`, `enable.delete=true`, `enable.combine.flush=true`, `buffer.flush.time=30`) **What happens** [`AsyncDorisStreamLoad.LoadAsyncExecutor.load()`](https://github.com/apache/doris-kafka-connector/blob/40b80e9897508baf56728b3757725947d103957c/src/main/java/org/apache/doris/kafka/connector/writer/load/AsyncDorisStreamLoad.java#L258) and [`DorisStreamLoad.load()`](https://github.com/apache/doris-kafka-connector/blob/40b80e9897508baf56728b3757725947d103957c/src/main/java/org/apache/doris/kafka/connector/writer/load/DorisStreamLoad.java#L123) only look at the response when the status is 200: ```java int statusCode = response.getStatusLine().getStatusCode(); if (statusCode == 200 && response.getEntity() != null) { ... parse, throw on a failed Status, record the response ... } // any other status: no exception, nothing recorded ``` For any other status the method returns normally. The buffer is gone, no error is logged, the task stays RUNNING, and the next commit returns offsets past the dropped records. The rows never reach Doris. Doris itself answers errors with HTTP 200 and a JSON body, so this path is reached when something between the connector and the FE answers instead: a load balancer, ingress or gateway returning 502/503/504 while an FE restarts or is overloaded. **Reproduction** An HTTP proxy in front of the FE HTTP port (`doris.urls` points at it) that answers `503` with nginx's HTML body for 10 s out of every 30 s, and forwards everything else. Postgres changes (inserts, updates, deletes on 20,000 keys at 500 transactions/s) for 240 s: - 108 stream loads started, 37 got a load result, 71 were answered 503 - no ERROR or WARN from the connector, no retry, tasks RUNNING throughout, 28 offset commits - after the source stopped and the replica had not changed for 120 s, 1,215 of Postgres's 17,526 rows were missing from Doris and 9,643 had an older value, and 1,481 rows that Postgres had deleted were still in Doris - the Kafka topic held no duplicates, so every difference comes from the sink - two more runs with the same setup ended the same way, with 1,018 and 1,213 rows missing **Expected** A non-200 response should fail the load like any other failed load (throw `StreamLoadException`), so that the records are retried or the commit fails and Kafka Connect rewinds. **Suggested fix** Treat every status other than 200 as a failure in both `load()` methods, including the status and body in the exception message. cc @JNSimba -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
