matanper commented on issue #70382: URL: https://github.com/apache/doris/issues/70382#issuecomment-6729293784
One more piece is needed beyond re-resolving through `DNSCache`. `DNSCache.resolveHostname` calls `NetUtils.getIpByHost`, which goes through `InetAddress`, and `DorisFE` sets `networkaddress.cache.ttl=60` (`fe/fe-core/src/main/java/org/apache/doris/DorisFE.java:193` on master). So a forced re-resolve right after a BE restart can still return the old IP from the JVM's own address cache for up to another 60 s. The heartbeat client resolves the hostname through the same JVM cache. In our patch, a forced lookup first evicts the host from `InetAddress.cache` (`start_fe.sh` already adds `--add-opens=java.base/java.net=ALL-UNNAMED`). If that field is not reachable, the eviction does nothing and the entry expires by its TTL. We also evict the host after a failed heartbeat, so the next heartbeat reaches a BE that moved. In a local reproduction (FQDN mode; BE restarted on a new IP right after a periodic `DNSCache` refresh; 4 queries/s), the current code returned 215 errors (105 of them `UNAVAILABLE: io exception`) and its first successful query came 57.7 s after the BE was back. With the patch there were 0 `UNAVAILABLE` errors and the first successful query came after about 10 s. The remaining errors are `no queryable replicas` while the scheduler still has the BE marked down, which also happens when a BE restarts on the same IP. #69423 does not evict the JVM entry, so with it alone, a forced re-resolve can still return the old IP. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
