diggle1 opened a new pull request, #99:
URL: https://github.com/apache/doris-kafka-connector/pull/99
Previously DorisStreamLoad / AsyncDorisStreamLoad / DorisCommitter
called BackendUtils.getAvailableBackend() before every stream load,
which round-robined all BEs and ran tryHttpConnection() on each one
with a hard-coded 60s connect timeout. On a hot write path this added
unnecessary HTTP probes per request, and a single unreachable BE could
block writes for up to 60s per node.
Changes:
- BackendUtils: add a TTL cache for the picked backend with double-checked
locking; expose invalidateCache() so callers can drop the cache after a
failure. Shorten the default probe timeout from 60s to 5s and apply it to
both connect and read timeouts; ensure the HttpURLConnection is always
disconnected.
- DorisStreamLoad / AsyncDorisStreamLoad: invalidate the cached backend on
load failure so the next attempt re-probes and may pick another BE.
- DorisCommitter: invalidate the cache before re-selecting a BE on commit
failure to avoid sticking to the failing node.
- DorisOptions / DorisSinkConnectorConfig: add two new config options,
backend.cache.ttl.ms (default 60000, 0 disables the cache) and
backend.probe.timeout.ms (default 5000), both backward compatible.
- Add BackendUtilsTest covering cache hit, TTL expiry, invalidateCache,
ttl=0 disabling the cache, multi-BE selection and the short probe
timeout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
