matanper opened a new pull request, #70762:
URL: https://github.com/apache/doris/pull/70762

   ### What problem does this PR solve?
   
   Issue Number: close #70382
   
   Related PR: #59904, #69423, #69450
   
   Problem Summary:
   
   Since #59904 the FE dials the IP that `DNSCache` resolved for a backend. 
`DNSCache.get()` is `computeIfAbsent`, and only the periodic task re-resolves 
(every 60 s, first run after 120 s). `BackendServiceProxy.getProxy` rebuilds a 
non-normal channel, but to the same stale IP. On top of that, 
`NetUtils.getIpByHost` goes through `InetAddress`, whose own cache keeps 
answers for `networkaddress.cache.ttl=60` (set in `DorisFE`), so even a forced 
lookup can return the old IP for another minute. The heartbeat client resolves 
the hostname through the same JVM cache.
   
   When a BE restarts under the same hostname with a new IP (e.g. a Kubernetes 
pod), every query that schedules a fragment on it fails with `send fragments 
failed ... UNAVAILABLE: io exception` for up to a minute or more after the BE 
is alive again. In a rolling restart this happens on every BE rejoin. #69450 
fixed this for heartbeat connections only.
   
   This is the narrow, default-on FE DNS part of #69423 (which also carries 
graceful-restart, Stream Load and BE changes), with the first-lookup race from 
its latest review handled, plus eviction of the JVM DNS entry, which #69423 
does not do.
   
   #### Fix
   
   - **`DNSCache.invalidate(host)`** marks the host for a forced lookup on its 
next `get()`:
     - lookups for a host are serialized with `compute`, so an older result 
cannot overwrite a newer one;
     - a failed lookup (exception or empty result) keeps the last known IP;
     - an invalidation that arrives while the host's first lookup is still 
running is kept, so the next `get()` resolves again (the race found in #69423's 
review);
     - warm readers wait for an in-flight forced lookup instead of returning 
the address it replaces;
     - the periodic refresh resolves **outside** the map lock (a slow resolver 
does not block other hosts in the same bin) and publishes only if no other 
lookup published for the host since it started, tracked by a per-host version 
bumped under the map lock. An address comparison would not be enough: a newer 
lookup may confirm the same address the refresh started from. A refresh does 
not consume a pending invalidation.
   - A forced lookup first evicts the host from the JVM `InetAddress` cache 
(`NetUtils.evictJvmDnsCache`, by reflection; `start_fe.sh` already passes 
`--add-opens=java.base/java.net=ALL-UNNAMED`, and the fe-core surefire 
`argLine` now does too). If the field is not reachable, nothing changes: the 
entry expires by its TTL as before.
   - **`DNSCache.invalidateAfterRpcFailure(host, failedIp)`** does nothing when 
the cache already moved past `failedIp` (a late failure from an old channel), 
and allows at most one forced lookup per host per second, so a BE that keeps 
refusing connections at an unchanged address does not turn every failed RPC 
into a DNS query.
   - **`BackendServiceProxy.invalidateForAll(address)`** drops the cached 
channel to the address from every proxy instance in `Holder` and shuts it down 
without waiting, so the heartbeat or replay thread never blocks.
   
   #### Triggers
   
   1. **RPC failure.** A gRPC client interceptor on every 
`BackendServiceClient` channel calls `invalidateAfterRpcFailure` when a call 
closes with `UNAVAILABLE`, or with `DEADLINE_EXCEEDED` while the channel is not 
`READY` (how calls end when the old address silently drops packets). A deadline 
on a `READY` channel is a slow call to a reachable BE and is ignored; 
application/query errors are never a trigger. Because it sits on the channel, 
it covers fragment sends and every other backend RPC, including failures that 
surface asynchronously through `Future.get()` in `Coordinator` / 
`PipelineExecutionTask`. Only the DNS entry is invalidated; `getProxy` rebuilds 
the channel only if the new lookup returns a different IP.
   2. **Failing channel.** When `getProxy` finds the cached channel in a 
non-normal state, it claims a forced lookup (same throttle) and always reads 
the cache again before rebuilding, so it uses a lookup already marked by a 
racing RPC failure even when the throttle drops its own claim. An empty DNS 
result in FQDN mode is re-resolved on the next RPC instead of staying cached 
until the periodic refresh.
   3. **Heartbeat.** `Backend.handleHbResponse` calls 
`invalidateLocalConnections()` (DNS invalidation plus `invalidateForAll` on the 
brpc address) when an OK heartbeat shows the BE back alive or with a new start 
time. It runs on the master and on every follower's heartbeat replay (each FE 
has its own DNS cache and channels), and is skipped on the checkpoint thread so 
historical replay never touches serving connections. A failed heartbeat on the 
master evicts the host's JVM DNS entry, so the next heartbeat reaches a BE that 
moved.
   
   No new config. When the IP did not change, the cost is one DNS lookup, plus 
one new channel on a BE restart.
   
   Out of scope (as suggested in the issue triage): transparent RPC retries, 
BE-to-BE exchange caches (BE side), Stream Load, graceful-shutdown admission.
   
   #### Before / after (restart at a new IP)
   
   This is an availability fix, not a throughput change, so the measurement is 
time to recovery. Setup: a local FE + BE in FQDN mode 
(`enable_fqdn_mode=true`), BE added as `fedns-be.test` via `/etc/hosts`. 
Restart simulation: stop the BE, make the old IP (127.0.0.2) reset connections 
to the BE ports with iptables, remap the hostname to 127.0.0.3, start the BE. 
Query loop at 4/s (`select count(*) ... where v + rand() > 0`, never served 
from cache). T0 = the BE accepts connections at the new IP. "Worst case" 
restarts the BE 4 s after a periodic DNS refresh. These runs were done with the 
same FE change on a 4.1.4-based build (the touched FE code is identical between 
4.1.4 and master apart from the conflicts noted below).
   
   | FE | restart timing | errors after T0 | `UNAVAILABLE` after T0 | first OK 
query after T0 |
   |---|---|---|---|---|
   | before | worst case | 215 | 105 | 57.7 s |
   | before | arbitrary | 37 | 37 | 10.4 s |
   | after | worst case | 36 | 0 | 9.7 s |
   | after | arbitrary | 0 | 0 | 0.2 s |
   
   Blackhole variant (old IP drops packets): before 70.3 s to the first OK 
query (3 queries hung for the loop's 20 s timeout), after 8.8 s.
   
   The remaining errors after the fix are all `tablet has no queryable 
replicas` while the scheduler still has the BE marked down; that window is the 
same when a BE restarts at an unchanged IP and is not changed here.
   Port notes vs. master: master dropped `channelConfigVersion` in 
`BackendServiceClient` and moved the fe-core surefire flags to 
`${fe.ut.jvm.args}`; the change is adapted to both.
   
   ### Release note
   
   Fix FE sending query RPCs to the old IP of a backend that restarted on a new 
IP under the same hostname (FQDN mode) for up to a minute or more.
   
   ### Check List (For Author)
   
   - Test <!-- At least one of them must be included. -->
       - [ ] Regression test
       - [x] Unit Test
       - [x] Manual test (add detailed scripts or steps below)
       - [ ] No need to test or manual test. Explain why:
   
     New: `DNSCacheInvalidationTest` (10) and `BackendRestartReconnectTest` 
(11): invalidate then re-resolve; a failed forced lookup keeps the last IP; a 
stale periodic refresh does not overwrite a newer lookup, including one that 
confirmed the same address; the refresh does not hold the lock during DNS; 
invalidation during the first lookup; warm readers wait for a forced lookup; 
RPC-failure invalidation scoped to the failing IP and throttled; JVM 
`InetAddress` cache eviction of `localhost`; `getProxy` rebuilds to the new IP 
after invalidation and for a `TRANSIENT_FAILURE` channel; a real `UNAVAILABLE` 
RPC (closed port) forces a lookup; `DEADLINE_EXCEEDED` while connecting (socket 
that never completes the HTTP/2 handshake) forces a lookup and on a `READY` 
channel (in-process server that never answers) does not; a racing RPC failure's 
lookup is used when the throttle drops `getProxy`'s claim; heartbeat with a new 
start time, follower replay and recovery after failed heartbeats drop ch
 annels and re-resolve; checkpoint replay leaves them alone. Each test was 
checked to fail with the corresponding piece of the fix reverted.
   
     `./run-fe-ut.sh --run` on master + this PR: `DNSCacheInvalidationTest`, 
`BackendRestartReconnectTest`, `DNSCacheTest`, `BackendServiceProxyTest`, 
`BackendServiceClientTest`, `HeartbeatMgrTest`, `BackendHeartbeatHandlerTest`, 
`BackendRemoteSpillBytesTest`: 61 tests, 0 failures, 0 skipped.
   
   - Behavior changed:
       - [ ] No.
       - [x] Yes. The FE re-resolves a backend hostname on demand after an 
unreachable RPC or a BE restart/recovery heartbeat, instead of only every 60 s, 
and bypasses the JVM DNS cache for those lookups.
   
   - Does this need documentation?
       - [x] No.
       - [ ] Yes.
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to