matanper opened a new issue, #69984: URL: https://github.com/apache/doris/issues/69984
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4. The same code is on master (`0fe5cc1e319`). ### What's Wrong? With `enable_fqdn_mode`, when a BE restarts and comes back on a new IP (the normal case for a recreated Kubernetes pod), other BEs keep dialing its old IP for up to a minute after it is Ready. Loads whose tablets are on that BE fail during that time with: ``` failed to open tablet writer, error=Host is down ``` Timeline from one restart (seconds relative to the moment the restarted BE was Ready): | t | Event | |---|---| | −126 s | The peer's hostname stops resolving while its pod is down. The other BEs log `Failed to resolve hostname …, use cached ip: <old ip>` and keep the old IP | | 0 s | The peer is Ready on its new IP | | +7 s … +55 s | Every Stream Load with tablets on the peer fails with `Host is down` | | +54.9 s | The periodic refresh logs `update hostname …'s ip to <new ip>`, then `get_client` logs `Cached ip changed`, and the failures stop | ### Cause 1. **`DNSCache` refreshes only once a minute and keeps the old IP while resolution fails** (`be/src/util/dns_cache.cpp`, `_refresh_cache` waits `std::chrono::minutes(1)`; `get` falls back to `use cached ip`). `BrpcClientCache::get_client()` dials whatever `DNSCache::get()` returns. When it finds a stub whose channel is unhealthy, it rebuilds the stub against the same stale IP, because nothing asks DNS again before the next periodic refresh. So the window can be up to the full 60 s after the peer is back. 2. **Only `EHOSTDOWN` marks a channel unhealthy** (`FailureDetectClosure` / `FailureDetectChannel` in `be/src/util/brpc_client_cache.h`). If the old IP has already been reused by another pod, connects fail with `ECONNREFUSED`, the channel stays "healthy", and the cached stub is reused as-is. 3. **`VNodeChannel::open_wait()` erases the failed stub by the wrong key** (`be/src/exec/sink/writer/vtablet_writer.cpp`): ```cpp ExecEnv::GetInstance()->brpc_internal_client_cache()->erase( open_callback->cntl_->remote_side()); ``` `remote_side()` is the `ip:port` endpoint, but `BrpcClientCache` is keyed by `hostname:port` in FQDN mode, so this erase never matches anything. #63358 / #63363 handled a dropped BE whose hostname never resolves again. They don't help here: the hostname resolves again within seconds, just to a new IP. #64708 (a load-stream socket that is never revived) is a different problem. In this case the cluster recovers by itself after the periodic refresh. ### What You Expected? When a connection to a peer BE fails as unreachable, the next `get_client()` should re-resolve that peer's hostname straight away instead of waiting for the periodic refresh. That way a restarted BE is reachable as soon as it is Ready. ### How to Reproduce? 1. Run a cluster with `enable_fqdn_mode=true` on Kubernetes (doris-operator), replication ≥ 2, and a steady Stream Load workload. 2. Delete one BE pod so it is recreated with a new pod IP. 3. Watch the other BEs' logs: `use cached ip: <old>` while the pod is down, then `Host is down` load failures after the pod is Ready, until `update hostname …'s ip to <new>` appears up to 60 s later. ### Anything Else? Proposed fix (we have it implemented and tested, with new unit tests passing under ASAN): - `DNSCache::refresh(hostname)`: re-resolve one hostname now. It is rate-limited per hostname by a new mutable config, `dns_cache_min_refresh_interval_ms` (default 1000). A failed resolution keeps the cached IP. The refresh thread waits on a condition variable so shutdown doesn't block for up to a minute. - `BrpcClientCache::get_client()`: when the cached stub's channel is unhealthy, call `refresh()` before rebuilding the stub, so the new stub dials the current IP. - Treat `ECONNREFUSED` as unreachable as well as `EHOSTDOWN`. - Add `BrpcClientCache::invalidate(host, port)`, which erases by `hostname:port` and re-resolves, and use it in `VNodeChannel::open_wait()` instead of `erase(remote_side())`. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
