This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a27bf6f179f [fix](dns-cache) evict hostnames that fail to resolve
repeatedly (#63363)
a27bf6f179f is described below
commit a27bf6f179f7157a0e59b3bdf047f294cbb3ed14
Author: zhaorongsheng <[email protected]>
AuthorDate: Mon Aug 3 09:48:10 2026 +0800
[fix](dns-cache) evict hostnames that fail to resolve repeatedly (#63363)
Proposed changes
Issue Number: close #63358
DNSCache currently never evicts an entry once it has been inserted. When
a backend host is permanently dropped from the cluster and its DNS
record is removed, every other BE in the cluster keeps:
1. logging failed to get ip from host: <removed> every 60 s,
indefinitely (from DNSCache::_refresh_cache -> hostname_to_ipv4);
2. handing the stale cached IP back to brpc through BrpcClientCache /
ClientCache, which then keeps emitting Fail to wait EPOLLOUT ...
Connection timed out at the brpc socket layer.
This PR adds a simple consecutive-failure counter to DNSCache. When the
counter reaches a configurable threshold, the entry is removed from the
cache so that callers no longer get a stale IP and the refresh thread
stops logging about
it. WARNING logs for the same host are also throttled to avoid flooding
be.WARNING.
Configs introduced
Name: dns_cache_max_consecutive_failures
Type: mInt32
Default: 30
Behavior: Evict a hostname after this many consecutive resolution
failures. At the default 60 s refresh interval, that means ~30 minutes
of grace. Set <= 0 to disable eviction (legacy behavior).
────────────────────────────────────────
Name: dns_cache_log_every_n_failures
Type: mInt32
Default: 60
Behavior: Throttle the Failed to resolve ... use cached ip warning to
once per N failures per hostname. Set <= 1 to log every failure (legacy
behavior).
Both are mutable so operators can tune without restarting BE.
Backward compatibility
Setting dns_cache_max_consecutive_failures = 0 and
dns_cache_log_every_n_failures = 1 reproduces exactly the pre-PR
behavior. Successful resolution clears the failure counter, so transient
DNS hiccups don't accumulate across hours. No
public API or wire format changes.
Further comments
- I deliberately did not touch the FE-side DNSCache.java. If the same
fix is wanted on FE, happy to send a follow-up PR.
- The eviction threshold default (30) is conservative; please push back
if you'd prefer a smaller / larger default. Operators with very flaky
DNS can lower it via the mutable config without redeploying.
---
Checklist
- I have read the Contributing document.
- I have created an issue (#63358) on (or commented on) the related
issue.
- I have added unit tests for my change.
- All new and existing tests passed (verified locally → fails to build
on macOS due to unrelated contrib/openblas issue; rely on CI).
- My change requires a change to the documentation. — No (config doc
auto-generated)
---------
Signed-off-by: zhaorongsheng <[email protected]>
Co-authored-by: zhaorongsheng <[email protected]>
Co-authored-by: morningman <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
---
be/src/cloud/cloud_warm_up_manager.cpp | 2 +-
be/src/common/config.cpp | 8 +
be/src/common/config.h | 33 ++
be/src/util/brpc_client_cache.h | 8 +
be/src/util/dns_cache.cpp | 309 +++++++++++--
be/src/util/dns_cache.h | 107 ++++-
be/src/util/network_util.cpp | 35 +-
be/src/util/network_util.h | 16 +-
be/test/util/dns_cache_test.cpp | 819 +++++++++++++++++++++++++++++++++
9 files changed, 1287 insertions(+), 50 deletions(-)
diff --git a/be/src/cloud/cloud_warm_up_manager.cpp
b/be/src/cloud/cloud_warm_up_manager.cpp
index dc7ddc01b59..9ede36b9883 100644
--- a/be/src/cloud/cloud_warm_up_manager.cpp
+++ b/be/src/cloud/cloud_warm_up_manager.cpp
@@ -986,7 +986,7 @@ void CloudWarmUpManager::_recycle_cache(int64_t tablet_id,
if (!status.ok()) {
LOG(WARNING) << "failed to get ip from host " <<
replica.replica.host << ": "
<< status.to_string();
- return;
+ continue;
}
}
std::string brpc_addr = get_host_port(host, replica.replica.brpc_port);
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index 3065d5b0fba..dede5133117 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -285,6 +285,14 @@ DEFINE_mInt32(download_binlog_meta_timeout_ms, "30000");
// the interval time(seconds) for agent report index policy to FE
DEFINE_mInt32(report_index_policy_interval_seconds, "10");
+// DNS cache: throttle "use cached ip" warning to once per N failures per host.
+DEFINE_mInt32(dns_cache_log_every_n_failures, "10");
+// DNS cache: evict a hostname after this many consecutive resolution failures.
+DEFINE_mInt32(dns_cache_max_consecutive_failures, "30");
+// DNS cache: after eviction, block re-resolve attempts for this many seconds.
+// Set <= 0 to disable the negative cache (legacy behavior).
+DEFINE_mInt32(dns_cache_negative_ttl_seconds, "60");
+
DEFINE_String(sys_log_dir, "");
DEFINE_String(user_function_dir, "${DORIS_HOME}/lib/udf");
// INFO, WARNING, ERROR, FATAL
diff --git a/be/src/common/config.h b/be/src/common/config.h
index ec48e288fec..9954f3b3e41 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -342,6 +342,39 @@ DECLARE_mInt32(download_binlog_meta_timeout_ms);
// the interval time(seconds) for agent report index policy to FE
DECLARE_mInt32(report_index_policy_interval_seconds);
+// DNS cache: log the "Failed to resolve hostname ... use cached ip" warning
+// only once per N consecutive failures for the same hostname, to avoid
+// flooding be.WARNING. Set <= 1 to log every failure (legacy behavior).
+// Should be set <= dns_cache_max_consecutive_failures, otherwise only the
+// first-failure log is ever emitted before a host is evicted.
+DECLARE_mInt32(dns_cache_log_every_n_failures);
+
+// DNS cache: evict a hostname after this many consecutive resolution failures.
+// At the default refresh interval of 60s, the default value of 30 means a
+// hostname that was once successfully resolved is evicted after ~30 minutes of
+// being un-resolvable. Hostnames that have never been successfully resolved
are
+// not tracked and are unaffected by this threshold.
+// Eviction additionally requires the most recent failure to be an
authoritative
+// NXDOMAIN (getaddrinfo returning EAI_NONAME), i.e. the resolver positively
+// stating that the name does not exist. Transient failures such as EAI_AGAIN
+// (resolver unreachable or timed out) never evict, so a DNS-server outage
+// degrades to serving the last known IP instead of emptying the cache for
every
+// hostname at once and turning a DNS incident into a cluster-wide RPC outage.
+// Set <= 0 to disable eviction (legacy behavior, kept for backward
compatibility).
+DECLARE_mInt32(dns_cache_max_consecutive_failures);
+
+// DNS cache: seconds to suppress re-resolve attempts for a hostname that could
+// not be resolved -- either because it was evicted after repeated failures, or
+// because it never resolved in the first place. During this window get()
+// returns an error immediately (no blocking getaddrinfo) so request threads
are
+// not stalled while the backend is being drained or while a bad hostname is
+// being retried.
+// This also bounds recovery latency: the refresh thread does not retry
hostnames
+// that are no longer in the cache, so a host comes back only when a caller's
+// get() runs after this TTL expires (one such retry per host per TTL).
+// Set <= 0 to disable the negative cache.
+DECLARE_mInt32(dns_cache_negative_ttl_seconds);
+
// deprecated, use env var LOG_DIR in be.conf
DECLARE_String(sys_log_dir);
// for udf
diff --git a/be/src/util/brpc_client_cache.h b/be/src/util/brpc_client_cache.h
index 993838699e8..aa0ce4524e0 100644
--- a/be/src/util/brpc_client_cache.h
+++ b/be/src/util/brpc_client_cache.h
@@ -174,6 +174,14 @@ public:
Status status = dns_cache->get(host, &realhost);
if (!status.ok()) {
LOG(WARNING) << "failed to get ip from host:" <<
status.to_string();
+ // The hostname is no longer resolvable, which normally means
the backend
+ // was dropped from the cluster. Returning early is not
enough: any stub
+ // cached under this host:port still holds a brpc Channel
bound to the last
+ // resolved (now dead) IP, and brpc keeps health-checking that
socket
+ // forever, which is the source of the endless
+ // "Fail to wait EPOLLOUT ... Connection timed out" warnings.
Drop it here
+ // so the socket is closed along with the last reference to
the stub.
+ _stub_map.erase(fmt::format("{}:{}", host, port));
return nullptr;
}
}
diff --git a/be/src/util/dns_cache.cpp b/be/src/util/dns_cache.cpp
index 0ea794872c1..9a4883bc399 100644
--- a/be/src/util/dns_cache.cpp
+++ b/be/src/util/dns_cache.cpp
@@ -17,6 +17,13 @@
#include "util/dns_cache.h"
+#include <netdb.h>
+
+#include <algorithm>
+#include <atomic>
+#include <unordered_set>
+
+#include "common/config.h"
#include "service/backend_options.h"
#include "util/network_util.h"
@@ -26,14 +33,21 @@ DNSCache::DNSCache() {
refresh_thread = std::thread(&DNSCache::_refresh_cache, this);
}
+DNSCache::DNSCache(Resolver resolver) : _resolver(std::move(resolver)) {}
+
DNSCache::~DNSCache() {
- stop_refresh = true;
+ {
+ std::lock_guard<std::mutex> lk(_cv_mutex);
+ stop_refresh = true;
+ }
+ _cv.notify_all();
if (refresh_thread.joinable()) {
refresh_thread.join();
}
}
Status DNSCache::get(const std::string& hostname, std::string* ip) {
+ bool has_negative_entry = false;
{
std::shared_lock<std::shared_mutex> lock(mutex);
auto it = cache.find(hostname);
@@ -41,20 +55,67 @@ Status DNSCache::get(const std::string& hostname,
std::string* ip) {
*ip = it->second;
return Status::OK();
}
+ auto neg_it = _negative_cache.find(hostname);
+ if (neg_it != _negative_cache.end()) {
+ int32_t ttl = config::dns_cache_negative_ttl_seconds;
+ if (ttl > 0) {
+ auto deadline = neg_it->second + std::chrono::seconds(ttl);
+ if (std::chrono::steady_clock::now() < deadline) {
+ // No stack trace: this is an expected steady state, not
an anomaly, and
+ // it is returned once per caller request for as long as
the host stays
+ // unresolvable. Capturing a stack here would make
Status::Error() log a
+ // WARNING per call (and every caller logs
status.to_string() again),
+ // recreating exactly the be.WARNING flood this cache
exists to stop.
+ return Status::InternalError<false>(
+ "Hostname {} is in negative DNS cache (recently
evicted or "
+ "unresolvable), skipping resolve",
+ hostname);
+ }
+ }
+ has_negative_entry = true;
+ }
}
- // Update if not found
- RETURN_IF_ERROR(_update(hostname));
- {
- std::shared_lock<std::shared_mutex> lock(mutex);
- *ip = cache[hostname];
- return Status::OK();
+
+ // If the host was in the negative cache with an expired (or disabled) TTL,
+ // claim the single-flight retry under unique_lock before the blocking DNS
+ // call. Re-arming the eviction_time to now() makes concurrent callers see
+ // an unexpired entry, bounding retries to one per host per TTL period.
+ if (has_negative_entry) {
+ std::unique_lock<std::shared_mutex> lock(mutex);
+ auto neg_it = _negative_cache.find(hostname);
+ if (neg_it != _negative_cache.end()) {
+ int32_t ttl = config::dns_cache_negative_ttl_seconds;
+ if (ttl <= 0) {
+ _negative_cache.erase(neg_it);
+ } else {
+ auto deadline = neg_it->second + std::chrono::seconds(ttl);
+ if (std::chrono::steady_clock::now() >= deadline) {
+ neg_it->second = std::chrono::steady_clock::now();
+ } else {
+ // Lost the single-flight race; see above for why this
carries no stack.
+ return Status::InternalError<false>(
+ "Hostname {} is in negative DNS cache (recently
evicted or "
+ "unresolvable), skipping resolve",
+ hostname);
+ }
+ }
+ }
}
+
+ // First access (or negative TTL expired): resolve and populate the cache.
+ // Consume the IP returned by _update() directly to avoid a second cache
+ // lookup — operator[] under a shared_lock would mutate the map and could
+ // reinsert an empty entry if a concurrent refresh cycle evicted the
hostname
+ // between _update() and here.
+ return _update(hostname, nullptr, ip);
}
// Resolve hostname to IP address, similar to Java's DNSCache.resolveHostname.
// If resolution fails, falls back to cached IP if available.
// Returns the resolved IP, or cached IP on failure, or empty string if no
cache available.
-std::string DNSCache::_resolve_hostname(const std::string& hostname) {
+// *is_fresh (if non-null) is set to true when DNS returned a live result,
false
+// when the IP comes from the stale cached fallback path.
+std::string DNSCache::_resolve_hostname(const std::string& hostname, bool*
is_fresh) {
// Get cached IP first (if any)
std::string cached_ip;
{
@@ -67,55 +128,245 @@ std::string DNSCache::_resolve_hostname(const
std::string& hostname) {
// Try to resolve hostname
std::string resolved_ip;
- Status status = hostname_to_ip(hostname, resolved_ip,
BackendOptions::is_bind_ipv6());
+ int gai_err = 0;
+ Status status =
+ _resolver ? _resolver(hostname, resolved_ip,
BackendOptions::is_bind_ipv6(), &gai_err)
+ : hostname_to_ip(hostname, resolved_ip,
BackendOptions::is_bind_ipv6(),
+ &gai_err);
if (!status.ok() || resolved_ip.empty()) {
- // Resolution failed
+ if (is_fresh) {
+ *is_fresh = false;
+ }
+ // EAI_NONAME is the resolver authoritatively answering "this name
does not exist",
+ // which is the only evidence that a backend is really gone.
Everything else
+ // (EAI_AGAIN = resolver unreachable or timed out, EAI_SYSTEM,
EAI_FAIL, ...) means
+ // DNS itself is unhealthy while the host is most likely still up at
its last known
+ // address, so those failures must never lead to eviction — otherwise
a resolver
+ // outage would wipe every hostname at once and turn a DNS incident
into a
+ // cluster-wide RPC outage.
+ const bool authoritative = (gai_err == EAI_NONAME);
if (!cached_ip.empty()) {
- LOG(WARNING) << "Failed to resolve hostname " << hostname
- << ", use cached ip: " << cached_ip;
+ // Only track failure counts for hosts that are currently in the
cache.
+ // Hosts that were never cached or have already been evicted are
not
+ // tracked, which prevents unbounded growth of failure_count.
+ uint32_t failures = 0;
+ {
+ std::unique_lock<std::shared_mutex> lock(mutex);
+ // Re-check that the host is still cached under the
unique_lock:
+ // it may have been evicted by the refresh thread between our
+ // earlier shared_lock read of cached_ip and now
(hostname_to_ip
+ // can block for seconds on DNS timeout, widening the window).
+ // Skipping the bump here preserves keys(failure_count) ⊆
keys(cache).
+ if (cache.find(hostname) != cache.end()) {
+ FailureState& state = failure_count[hostname];
+ // The counter tracks failures of any kind so the
throttled log below
+ // stays informative during a resolver outage; only
`last_authoritative`
+ // gates eviction.
+ failures = ++state.count;
+ state.last_authoritative = authoritative;
+ }
+ }
+ // Throttle the log: only every N failures or the first failure.
+ if (failures > 0) {
+ int32_t every_n = std::max(1,
config::dns_cache_log_every_n_failures);
+ if (failures == 1 || failures % static_cast<uint32_t>(every_n)
== 0) {
+ LOG(WARNING) << "Failed to resolve hostname " << hostname
+ << " (consecutive failures: " << failures <<
", error: "
+ << (authoritative ? "NXDOMAIN, host is gone"
+ : "transient, DNS
unhealthy")
+ << "), use cached ip: " << cached_ip;
+ }
+ }
return cached_ip;
} else {
- LOG(WARNING) << "Failed to resolve hostname " << hostname << ", no
cached ip available";
+ // Throttle to avoid flooding be.WARNING when callers repeatedly
+ // query an evicted or never-resolvable hostname. This branch
+ // deliberately does not maintain a per-hostname counter (that
+ // would break the keys(failure_count) ⊆ keys(cache) invariant),
+ // so the throttle is a coarse global rate limit shared across
+ // all hostnames hitting this code path.
+ static std::atomic<uint64_t> no_cache_warn_counter {0};
+ uint64_t n = no_cache_warn_counter.fetch_add(1,
std::memory_order_relaxed) + 1;
+ int32_t every_n = std::max(1,
config::dns_cache_log_every_n_failures);
+ if (n == 1 || n % static_cast<uint64_t>(every_n) == 0) {
+ LOG(WARNING) << "Failed to resolve hostname " << hostname
+ << ", no cached ip available";
+ }
return "";
}
}
+ // Resolution succeeded - clear failure counter for this hostname.
+ if (is_fresh) {
+ *is_fresh = true;
+ }
+ {
+ std::unique_lock<std::shared_mutex> lock(mutex);
+ failure_count.erase(hostname);
+ }
return resolved_ip;
}
-Status DNSCache::_update(const std::string& hostname) {
- std::string real_ip = _resolve_hostname(hostname);
+void DNSCache::_evict_locked(const std::string& hostname) {
+ cache.erase(hostname);
+ failure_count.erase(hostname);
+ int32_t ttl = config::dns_cache_negative_ttl_seconds;
+ if (ttl > 0) {
+ _negative_cache[hostname] = std::chrono::steady_clock::now();
+ }
+}
+
+void DNSCache::_remember_unresolvable(const std::string& hostname) {
+ int32_t ttl = config::dns_cache_negative_ttl_seconds;
+ if (ttl <= 0) {
+ return;
+ }
+ std::unique_lock<std::shared_mutex> lock(mutex);
+ // try_emplace, not operator[]: get()'s single-flight path may have just
re-armed this
+ // tombstone to now(); overwriting it would be harmless there but would
also let two
+ // callers racing on the same host each reset the deadline, loosening the
rate limit.
+ _negative_cache.try_emplace(hostname, std::chrono::steady_clock::now());
+}
+
+void DNSCache::_erase(const std::string& hostname) {
+ std::unique_lock<std::shared_mutex> lock(mutex);
+ _evict_locked(hostname);
+}
+
+bool DNSCache::_erase_if_still_failing(const std::string& hostname, uint32_t
threshold) {
+ std::unique_lock<std::shared_mutex> lock(mutex);
+ auto fc_it = failure_count.find(hostname);
+ if (fc_it == failure_count.end() || fc_it->second.count < threshold ||
+ !fc_it->second.last_authoritative) {
+ // Either a concurrent successful resolution cleared or reset the
counter between
+ // _update() returning and this call — do not erase a now-healthy
entry — or the
+ // most recent failure was transient (resolver unreachable) rather
than an
+ // authoritative NXDOMAIN, in which case the host is probably still
alive.
+ return false;
+ }
+ _evict_locked(hostname);
+ return true;
+}
+
+Status DNSCache::_update(const std::string& hostname, FailureState* out_state,
+ std::string* out_ip) {
+ bool is_fresh = false;
+ std::string real_ip = _resolve_hostname(hostname, &is_fresh);
if (real_ip.empty()) {
- return Status::InternalError("Failed to resolve hostname {} and no
cached ip available",
- hostname);
+ if (out_state) {
+ *out_state = FailureState {};
+ }
+ if (out_ip) {
+ out_ip->clear();
+ }
+ // The host could not be resolved and has no cached IP to fall back
on, so it never
+ // entered `cache` and will therefore never reach the eviction path
that writes a
+ // tombstone. Record one here as well: otherwise every single get() on
a hostname
+ // that has never resolved (a typo in the FE, a backend registered
before its DNS
+ // record propagated) pays a full blocking getaddrinfo, and many of
those calls run
+ // on bthreads where long blocking is especially costly.
+ _remember_unresolvable(hostname);
+ return Status::InternalError<false>(
+ "Failed to resolve hostname {} and no cached ip available",
hostname);
}
std::unique_lock<std::shared_mutex> lock(mutex);
+ // _resolve_hostname may have captured a stale cached_ip before a
concurrent
+ // eviction completed. If the host is now in the negative cache we must
not
+ // reinsert the stale IP: that would silently undo the eviction and clear
the
+ // tombstone, defeating the whole purpose of eviction. Only a fresh DNS
+ // result (is_fresh == true, meaning DNS actually resolved) may override an
+ // eviction — which indicates the backend is genuinely back.
+ if (!is_fresh && _negative_cache.count(hostname)) {
+ if (out_state) {
+ *out_state = FailureState {};
+ }
+ if (out_ip) {
+ out_ip->clear();
+ }
+ // No stack trace: like the negative-cache hits in get(), this is an
expected
+ // outcome that can repeat on every request while the host stays
evicted.
+ return Status::InternalError<false>(
+ "Hostname {} was concurrently evicted; stale-fallback not
reinserted", hostname);
+ }
auto it = cache.find(hostname);
if (it == cache.end() || it->second != real_ip) {
cache[hostname] = real_ip;
LOG(INFO) << "update hostname " << hostname << "'s ip to " << real_ip;
}
+ // DNS resolved successfully — remove any negative cache tombstone so
+ // subsequent get() calls go straight to the main cache.
+ _negative_cache.erase(hostname);
+ if (out_ip) {
+ *out_ip = real_ip;
+ }
+ // Read failure_count under the same lock we already hold, so _refresh_once
+ // does not need a second lock acquisition to decide on eviction.
+ if (out_state) {
+ auto fc_it = failure_count.find(hostname);
+ *out_state = fc_it != failure_count.end() ? fc_it->second :
FailureState {};
+ }
return Status::OK();
}
+void DNSCache::_refresh_once() {
+ std::unordered_set<std::string> keys;
+ {
+ std::shared_lock<std::shared_mutex> lock(mutex);
+ std::transform(cache.begin(), cache.end(), std::inserter(keys,
keys.end()),
+ [](const auto& pair) { return pair.first; });
+ }
+ for (auto& key : keys) {
+ // Each _update() below performs a blocking getaddrinfo, so one cycle
over a
+ // cluster whose DNS is timing out can take minutes. Without this
check the
+ // destructor's join() would be held up for exactly that long, which
would
+ // undo the point of making the wait itself interruptible.
+ if (stop_refresh.load(std::memory_order_acquire)) {
+ break;
+ }
+ FailureState state;
+ Status st = _update(key, &state);
+ if (!st.ok()) {
+ // _update returns an error either when _resolve_hostname returns
""
+ // (no fallback IP) or when a stale fallback was suppressed because
+ // the host was concurrently evicted. Either way, log and move on;
+ // the threshold check below handles the normal eviction path.
+ LOG(WARNING) << "Failed to update DNS cache for hostname " << key
<< ": "
+ << st.to_string();
+ }
+ // Evict hostnames that have failed to resolve for too long.
+ // This avoids two pathological symptoms after a backend is dropped
+ // from the cluster and its DNS record is removed:
+ // 1) be.WARNING gets flooded with `failed to get ip from host`.
+ // 2) brpc keeps re-using the stale IP from cache, producing
+ // `Fail to wait EPOLLOUT ... Connection timed out`.
+ // `last_authoritative` keeps this restricted to hosts the resolver
has positively
+ // reported as non-existent; a DNS outage yields transient errors for
every host at
+ // once and must leave the cache (and the stale-IP fallback) intact.
+ int32_t threshold = config::dns_cache_max_consecutive_failures;
+ if (threshold > 0 && state.last_authoritative &&
+ state.count >= static_cast<uint32_t>(threshold)) {
+ // Re-read failure_count under the mutex that also performs the
erase
+ // to fence any concurrent success that cleared the counter between
+ // _update() returning and this point.
+ if (_erase_if_still_failing(key,
static_cast<uint32_t>(threshold))) {
+ LOG(WARNING) << "Evicting hostname " << key << " from DNS
cache after "
+ << state.count << " consecutive resolution
failures";
+ }
+ }
+ }
+}
+
void DNSCache::_refresh_cache() {
while (!stop_refresh) {
- // refresh every 1 min
- std::this_thread::sleep_for(std::chrono::minutes(1));
- std::unordered_set<std::string> keys;
{
- std::shared_lock<std::shared_mutex> lock(mutex);
- std::transform(cache.begin(), cache.end(), std::inserter(keys,
keys.end()),
- [](const auto& pair) { return pair.first; });
+ std::unique_lock<std::mutex> lk(_cv_mutex);
+ // Wake up either after 1 minute or when the destructor signals
stop.
+ _cv.wait_for(lk, std::chrono::minutes(1), [this] { return
stop_refresh.load(); });
}
- for (auto& key : keys) {
- Status st = _update(key);
- if (!st.ok()) {
- LOG(WARNING) << "Failed to update DNS cache for hostname " <<
key << ": "
- << st.to_string();
- }
+ if (!stop_refresh) {
+ _refresh_once();
}
}
}
diff --git a/be/src/util/dns_cache.h b/be/src/util/dns_cache.h
index 51ffb6567ec..4adec2f7206 100644
--- a/be/src/util/dns_cache.h
+++ b/be/src/util/dns_cache.h
@@ -17,8 +17,13 @@
#pragma once
+#include <atomic>
#include <chrono>
+#include <condition_variable>
+#include <cstdint>
+#include <functional>
#include <iostream>
+#include <mutex>
#include <shared_mutex>
#include <string>
#include <thread>
@@ -32,7 +37,28 @@ namespace doris {
// fe/fe-core/src/main/java/org/apache/doris/common/DNSCache.java
class DNSCache {
public:
+ // (hostname, out_ip, is_ipv6, out_gai_err) -> Status.
+ // out_gai_err receives the raw getaddrinfo() return code so the cache can
tell an
+ // authoritative "no such host" (EAI_NONAME) apart from a transient
resolver failure.
+ using Resolver = std::function<Status(const std::string&, std::string&,
bool, int*)>;
+
+ // Per-hostname failure bookkeeping. Only tracked for hostnames currently
present in
+ // `cache`, which keeps the map bounded (invariant: keys(failure_count) ⊆
keys(cache)).
+ struct FailureState {
+ // Consecutive resolution failures of any kind. Drives the log
throttle.
+ uint32_t count = 0;
+ // Whether the most recent failure was an authoritative NXDOMAIN.
Eviction requires
+ // this to be true, so that a DNS-server outage (which yields
EAI_AGAIN for every
+ // hostname at once) degrades to the stale cached IP instead of wiping
the cache.
+ bool last_authoritative = false;
+ };
+
DNSCache();
+
+ // Test-only constructor: uses a custom resolver and does NOT start the
+ // background refresh thread. Call refresh_for_test() to drive one cycle.
+ explicit DNSCache(Resolver resolver);
+
~DNSCache();
// get ip by hostname
@@ -42,21 +68,94 @@ private:
// Resolve hostname to IP address.
// If resolution fails, falls back to cached IP if available.
// Returns the resolved IP, or cached IP on failure, or empty string if no
cache available.
- std::string _resolve_hostname(const std::string& hostname);
+ // *is_fresh is set to true when DNS returned a live result, false when the
+ // returned IP is the stale cached fallback from a failed lookup.
+ std::string _resolve_hostname(const std::string& hostname, bool* is_fresh
= nullptr);
+
+ // update the ip of hostname in cache; out_state (if non-null) is set to
the
+ // current failure bookkeeping read under the same lock; out_ip (if
non-null)
+ // receives the resolved IP so callers can use it without a second cache
lookup
+ // (avoids operator[] mutation under shared_lock).
+ Status _update(const std::string& hostname, FailureState* out_state =
nullptr,
+ std::string* out_ip = nullptr);
+
+ // erase a hostname from cache unconditionally (with unique_lock)
+ void _erase(const std::string& hostname);
+
+ // Erase a hostname from cache only if it still meets the eviction
criteria:
+ // failure_count >= threshold AND the most recent failure was
authoritative.
+ // Re-reads the live state under the same lock that performs the erase, so
a
+ // concurrent successful resolution that cleared it is not lost.
+ // Returns true if the host was erased, false otherwise.
+ bool _erase_if_still_failing(const std::string& hostname, uint32_t
threshold);
+
+ // Drop a hostname from the cache and write a negative-cache tombstone
(subject to
+ // dns_cache_negative_ttl_seconds). Caller must already hold a unique_lock
on `mutex`.
+ void _evict_locked(const std::string& hostname);
- // update the ip of hostname in cache
- Status _update(const std::string& hostname);
+ // Record a tombstone for a hostname that could not be resolved and has no
cached IP,
+ // so repeated get() calls do not each pay a full blocking getaddrinfo.
Uses
+ // try_emplace so an entry just re-armed by get()'s single-flight path is
preserved.
+ void _remember_unresolvable(const std::string& hostname);
+
+ // one refresh cycle: update every cached hostname and evict if needed
+ void _refresh_once();
// a function for refresh daemon thread
// update cache at fix internal
void _refresh_cache();
+ // ── test helpers (accessible via friend class DNSCacheTest)
──────────────
+ size_t size_for_test() const {
+ std::shared_lock<std::shared_mutex> lock(mutex);
+ return cache.size();
+ }
+
+ size_t negative_cache_size_for_test() const {
+ std::shared_lock<std::shared_mutex> lock(mutex);
+ return _negative_cache.size();
+ }
+
+ uint32_t failure_count_for_test(const std::string& hostname) const {
+ std::shared_lock<std::shared_mutex> lock(mutex);
+ auto it = failure_count.find(hostname);
+ return it != failure_count.end() ? it->second.count : 0;
+ }
+
+ // Run one refresh cycle synchronously (no sleep). Only meaningful when
+ // the object was constructed with the test constructor (no background
thread).
+ void refresh_for_test() { _refresh_once(); }
+
+ // Backdate all negative-cache entries far into the past so they appear
+ // expired without removing them. Use this to simulate TTL expiry in tests
+ // that need the re-arm path in get() to trigger.
+ // Backdating relative to now() (rather than to steady_clock's epoch,
which is
+ // typically boot time) keeps this correct on a freshly booted host.
+ void _expire_negative_cache_for_test() {
+ std::unique_lock<std::shared_mutex> lock(mutex);
+ auto backdated = std::chrono::steady_clock::now() -
std::chrono::hours(24 * 365);
+ for (auto& [k, v] : _negative_cache) {
+ v = backdated;
+ }
+ }
+
+ friend class DNSCacheTest;
+
private:
+ Resolver _resolver; // null → use global hostname_to_ip
// hostname -> ip
std::unordered_map<std::string, std::string> cache;
+ // hostname -> consecutive resolution failure bookkeeping
+ std::unordered_map<std::string, FailureState> failure_count;
+ // hostname -> eviction timestamp; effective deadline is computed as
+ // eviction_time + dns_cache_negative_ttl_seconds to honor live config
changes.
+ std::unordered_map<std::string, std::chrono::steady_clock::time_point>
_negative_cache;
mutable std::shared_mutex mutex;
std::thread refresh_thread;
- bool stop_refresh = false;
+ // Protects stop_refresh and signals _refresh_cache to wake early on
destroy.
+ std::mutex _cv_mutex;
+ std::condition_variable _cv;
+ std::atomic<bool> stop_refresh {false};
};
} // end of namespace doris
diff --git a/be/src/util/network_util.cpp b/be/src/util/network_util.cpp
index 85e16801648..ad1315dc121 100644
--- a/be/src/util/network_util.cpp
+++ b/be/src/util/network_util.cpp
@@ -119,13 +119,13 @@ bool parse_endpoint(const std::string& endpoint,
std::string* host, uint16_t* po
return true;
}
-Status hostname_to_ip(const std::string& host, std::string& ip) {
+Status hostname_to_ip(const std::string& host, std::string& ip, int* gai_err) {
auto start = std::chrono::high_resolution_clock::now();
- Status status = hostname_to_ipv4(host, ip);
+ Status status = hostname_to_ipv4(host, ip, gai_err);
if (status.ok()) {
return status;
}
- status = hostname_to_ipv6(host, ip);
+ status = hostname_to_ipv6(host, ip, gai_err);
auto current = std::chrono::high_resolution_clock::now();
auto duration =
std::chrono::duration_cast<std::chrono::milliseconds>(current - start);
@@ -136,15 +136,15 @@ Status hostname_to_ip(const std::string& host,
std::string& ip) {
return status;
}
-Status hostname_to_ip(const std::string& host, std::string& ip, bool ipv6) {
+Status hostname_to_ip(const std::string& host, std::string& ip, bool ipv6,
int* gai_err) {
if (ipv6) {
- return hostname_to_ipv6(host, ip);
+ return hostname_to_ipv6(host, ip, gai_err);
} else {
- return hostname_to_ipv4(host, ip);
+ return hostname_to_ipv4(host, ip, gai_err);
}
}
-Status hostname_to_ipv4(const std::string& host, std::string& ip) {
+Status hostname_to_ipv4(const std::string& host, std::string& ip, int*
gai_err) {
addrinfo hints, *res;
in_addr addr;
@@ -152,10 +152,17 @@ Status hostname_to_ipv4(const std::string& host,
std::string& ip) {
hints.ai_socktype = SOCK_STREAM;
hints.ai_family = AF_INET;
int err = getaddrinfo(host.c_str(), NULL, &hints, &res);
+ if (gai_err != nullptr) {
+ *gai_err = err;
+ }
if (err != 0) {
LOG(WARNING) << "failed to get ip from host: " << host << "err:" <<
gai_strerror(err);
- return Status::InternalError("failed to get ip from host: {}, err:
{}", host,
- gai_strerror(err));
+ // No stack trace: the WARNING above already carries the host and the
resolver error,
+ // and this failure is expected often enough (a decommissioned
backend, a hostname
+ // whose DNS record has not propagated yet) that attaching a stack to
every
+ // occurrence floods be.WARNING without adding information.
+ return Status::InternalError<false>("failed to get ip from host: {},
err: {}", host,
+ gai_strerror(err));
}
addr.s_addr = ((sockaddr_in*)(res->ai_addr))->sin_addr.s_addr;
@@ -165,7 +172,7 @@ Status hostname_to_ipv4(const std::string& host,
std::string& ip) {
return Status::OK();
}
-Status hostname_to_ipv6(const std::string& host, std::string& ip) {
+Status hostname_to_ipv6(const std::string& host, std::string& ip, int*
gai_err) {
char ipstr2[128];
struct sockaddr_in6* sockaddr_ipv6;
@@ -175,10 +182,14 @@ Status hostname_to_ipv6(const std::string& host,
std::string& ip) {
hint.ai_socktype = SOCK_STREAM;
int err = getaddrinfo(host.c_str(), NULL, &hint, &answer);
+ if (gai_err != nullptr) {
+ *gai_err = err;
+ }
if (err != 0) {
LOG(WARNING) << "failed to get ip from host: " << host << "err:" <<
gai_strerror(err);
- return Status::InternalError("failed to get ip from host: {}, err:
{}", host,
- gai_strerror(err));
+ // See hostname_to_ipv4() for why this error carries no stack trace.
+ return Status::InternalError<false>("failed to get ip from host: {},
err: {}", host,
+ gai_strerror(err));
}
sockaddr_ipv6 = reinterpret_cast<struct sockaddr_in6*>(answer->ai_addr);
diff --git a/be/src/util/network_util.h b/be/src/util/network_util.h
index fe8864bd1bd..e4521c4b0a6 100644
--- a/be/src/util/network_util.h
+++ b/be/src/util/network_util.h
@@ -47,13 +47,21 @@ bool is_valid_ip(const std::string& ip);
bool parse_endpoint(const std::string& endpoint, std::string* host, uint16_t*
port);
-Status hostname_to_ip(const std::string& host, std::string& ip);
+// The `gai_err` out-parameter, when non-null, receives the raw getaddrinfo()
return code
+// (0 on success). Callers need it to tell an authoritative "no such host"
(EAI_NONAME)
+// apart from a transient resolver problem (EAI_AGAIN, EAI_SYSTEM, ...): the
former means
+// the host is really gone, the latter means DNS itself is unhealthy while the
host is
+// most likely still alive. DNSCache relies on that distinction so a resolver
outage does
+// not get mistaken for every backend disappearing at once.
+// For the two-argument hostname_to_ip(), which falls back from IPv4 to IPv6,
`gai_err`
+// reports the code of the last attempt.
+Status hostname_to_ip(const std::string& host, std::string& ip, int* gai_err =
nullptr);
-Status hostname_to_ipv4(const std::string& host, std::string& ip);
+Status hostname_to_ipv4(const std::string& host, std::string& ip, int* gai_err
= nullptr);
-Status hostname_to_ipv6(const std::string& host, std::string& ip);
+Status hostname_to_ipv6(const std::string& host, std::string& ip, int* gai_err
= nullptr);
-Status hostname_to_ip(const std::string& host, std::string& ip, bool ipv6);
+Status hostname_to_ip(const std::string& host, std::string& ip, bool ipv6,
int* gai_err = nullptr);
// Finds the first non-localhost IP address in the given list. Returns
// true if such an address was found, false otherwise.
diff --git a/be/test/util/dns_cache_test.cpp b/be/test/util/dns_cache_test.cpp
new file mode 100644
index 00000000000..dab8bbdead1
--- /dev/null
+++ b/be/test/util/dns_cache_test.cpp
@@ -0,0 +1,819 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "util/dns_cache.h"
+
+#include <gtest/gtest-message.h>
+#include <gtest/gtest-test-part.h>
+#include <netdb.h>
+
+#include "common/config.h"
+#include "gtest/gtest_pred_impl.h"
+
+namespace doris {
+
+class DNSCacheTest : public testing::Test {
+public:
+ DNSCacheTest() = default;
+ ~DNSCacheTest() override = default;
+
+protected:
+ void SetUp() override {
+ _saved_threshold = config::dns_cache_max_consecutive_failures;
+ _saved_log_every = config::dns_cache_log_every_n_failures;
+ _saved_negative_ttl = config::dns_cache_negative_ttl_seconds;
+ }
+
+ void TearDown() override {
+ config::dns_cache_max_consecutive_failures = _saved_threshold;
+ config::dns_cache_log_every_n_failures = _saved_log_every;
+ config::dns_cache_negative_ttl_seconds = _saved_negative_ttl;
+ }
+
+ // Build a resolver whose failure behaviour can be flipped at runtime.
Failures are
+ // reported as authoritative NXDOMAIN by default, which is what eviction
requires.
+ static DNSCache::Resolver make_resolver(bool* should_fail, const
std::string& ip = "1.2.3.4",
+ int fail_gai_err = EAI_NONAME) {
+ return [should_fail, ip, fail_gai_err](const std::string&,
std::string& out, bool,
+ int* gai_err) -> Status {
+ if (*should_fail) {
+ if (gai_err != nullptr) {
+ *gai_err = fail_gai_err;
+ }
+ return Status::InternalError("mock failure");
+ }
+ if (gai_err != nullptr) {
+ *gai_err = 0;
+ }
+ out = ip;
+ return Status::OK();
+ };
+ }
+
+ // Build a resolver that counts its invocations; failures are
authoritative NXDOMAIN
+ // unless `fail_gai_err` says otherwise.
+ static DNSCache::Resolver make_counting_resolver(bool* should_fail, int*
calls,
+ const std::string& ip =
"1.2.3.4",
+ int fail_gai_err =
EAI_NONAME) {
+ return [should_fail, calls, ip, fail_gai_err](const std::string&,
std::string& out, bool,
+ int* gai_err) -> Status {
+ ++(*calls);
+ if (*should_fail) {
+ if (gai_err != nullptr) {
+ *gai_err = fail_gai_err;
+ }
+ return Status::InternalError("mock failure");
+ }
+ if (gai_err != nullptr) {
+ *gai_err = 0;
+ }
+ out = ip;
+ return Status::OK();
+ };
+ }
+
+private:
+ int32_t _saved_threshold = 0;
+ int32_t _saved_log_every = 0;
+ int32_t _saved_negative_ttl = 0;
+};
+
+// ── existing tests
────────────────────────────────────────────────────────────
+
+// Sanity: localhost resolves successfully and is cached.
+TEST_F(DNSCacheTest, resolve_localhost) {
+ DNSCache cache;
+ std::string ip;
+ EXPECT_TRUE(cache.get("localhost", &ip).ok());
+ EXPECT_FALSE(ip.empty());
+ // Second call hits the cache fast path and returns the same IP.
+ std::string ip2;
+ EXPECT_TRUE(cache.get("localhost", &ip2).ok());
+ EXPECT_EQ(ip, ip2);
+ EXPECT_EQ(1u, cache.size_for_test());
+}
+
+// Unresolvable hostname on first access returns InternalError and is NOT
cached.
+TEST_F(DNSCacheTest, first_miss_does_not_cache) {
+ config::dns_cache_negative_ttl_seconds = 3600;
+
+ DNSCache cache;
+ std::string ip;
+ Status st = cache.get("this-host-does-not-exist.invalid", &ip);
+ EXPECT_FALSE(st.ok());
+ EXPECT_EQ(0u, cache.size_for_test());
+ // It is tombstoned instead, so the next caller does not pay another
getaddrinfo.
+ EXPECT_EQ(1u, cache.negative_cache_size_for_test());
+}
+
+// Repeated successful resolution does not grow the cache and does not
accumulate
+// any failure state.
+TEST_F(DNSCacheTest, success_keeps_cache_stable) {
+ DNSCache cache;
+ std::string ip;
+ for (int i = 0; i < 8; ++i) {
+ EXPECT_TRUE(cache.get("localhost", &ip).ok());
+ }
+ EXPECT_EQ(1u, cache.size_for_test());
+}
+
+// The eviction config can be disabled by setting threshold <= 0 (legacy
behavior):
+// a cached host whose DNS record disappears keeps serving its stale IP
forever,
+// exactly as it did before this fix. Drive many refresh cycles (well past any
+// plausible threshold) and assert the entry is never dropped.
+TEST_F(DNSCacheTest, eviction_disabled_when_threshold_zero) {
+ config::dns_cache_max_consecutive_failures = 0;
+
+ bool should_fail = false;
+ DNSCache cache(make_resolver(&should_fail));
+
+ // Populate the cache with one successful resolution.
+ std::string ip;
+ ASSERT_TRUE(cache.get("fake-host.test", &ip).ok());
+ ASSERT_EQ("1.2.3.4", ip);
+ ASSERT_EQ(1u, cache.size_for_test());
+
+ // DNS now fails permanently.
+ should_fail = true;
+
+ // Far more cycles than the default threshold (30) would need to evict.
+ for (int i = 0; i < 50; ++i) {
+ cache.refresh_for_test();
+ ASSERT_EQ(1u, cache.size_for_test())
+ << "eviction must be disabled when threshold <= 0 (cycle " <<
i << ")";
+ }
+
+ // No tombstone was written, and callers still get the stale IP.
+ EXPECT_EQ(0u, cache.negative_cache_size_for_test());
+ ip.clear();
+ EXPECT_TRUE(cache.get("fake-host.test", &ip).ok())
+ << "legacy mode must keep serving the cached ip";
+ EXPECT_EQ("1.2.3.4", ip);
+
+ // The failure counter still accumulates; only the eviction action is
disabled.
+ EXPECT_GT(cache.failure_count_for_test("fake-host.test"), 0u);
+}
+
+// ── new tests for eviction logic
─────────────────────────────────────────────
+
+// A hostname that was once successfully resolved is evicted from the cache
after
+// dns_cache_max_consecutive_failures refresh cycles of continuous DNS failure.
+TEST_F(DNSCacheTest, evicts_after_threshold) {
+ config::dns_cache_max_consecutive_failures = 3;
+
+ bool should_fail = false;
+ DNSCache cache(make_resolver(&should_fail));
+
+ // Populate the cache with one successful resolution.
+ std::string ip;
+ ASSERT_TRUE(cache.get("fake-host.test", &ip).ok());
+ ASSERT_EQ("1.2.3.4", ip);
+ ASSERT_EQ(1u, cache.size_for_test());
+
+ // Now make DNS fail.
+ should_fail = true;
+
+ // Each refresh_for_test() call is one _refresh_cache iteration.
+ // The entry must survive the first threshold-1 cycles and disappear on the
+ // threshold-th cycle.
+ for (int i = 0; i < 2; ++i) {
+ cache.refresh_for_test();
+ EXPECT_EQ(1u, cache.size_for_test()) << "should not be evicted yet
(i=" << i << ")";
+ }
+ cache.refresh_for_test(); // third failure → threshold reached → eviction
+ EXPECT_EQ(0u, cache.size_for_test()) << "host should have been evicted
after threshold";
+}
+
+// One successful resolution resets the failure counter, so a full threshold of
+// additional failures is required before the next eviction.
+TEST_F(DNSCacheTest, success_resets_failure_count) {
+ config::dns_cache_max_consecutive_failures = 3;
+
+ bool should_fail = false;
+ DNSCache cache(make_resolver(&should_fail));
+
+ std::string ip;
+ ASSERT_TRUE(cache.get("fake-host.test", &ip).ok());
+
+ // Accumulate threshold-1 failures — must NOT evict.
+ should_fail = true;
+ cache.refresh_for_test();
+ cache.refresh_for_test();
+ EXPECT_EQ(1u, cache.size_for_test()) << "should not be evicted yet";
+
+ // One success clears the counter.
+ should_fail = false;
+ cache.refresh_for_test();
+ EXPECT_EQ(1u, cache.size_for_test()) << "success should keep the cache
entry";
+
+ // A full new round of threshold failures is needed before eviction.
+ should_fail = true;
+ cache.refresh_for_test();
+ cache.refresh_for_test();
+ EXPECT_EQ(1u, cache.size_for_test()) << "still not enough failures after
counter reset";
+ cache.refresh_for_test(); // third failure post-reset → eviction
+ EXPECT_EQ(0u, cache.size_for_test()) << "evicted after second run of
threshold failures";
+}
+
+// A hostname that was never successfully cached must not accumulate entries in
+// failure_count, regardless of how many times get() is called (fix for 7.2).
+TEST_F(DNSCacheTest, failure_count_does_not_grow_for_never_cached_host) {
+ config::dns_cache_negative_ttl_seconds = 0; // no tombstone, so every call
reaches the resolver
+
+ int resolver_calls = 0;
+ auto always_fail = [&resolver_calls](const std::string&, std::string&,
bool,
+ int* gai_err) -> Status {
+ ++resolver_calls;
+ if (gai_err != nullptr) {
+ *gai_err = EAI_NONAME;
+ }
+ return Status::InternalError("always fails");
+ };
+ DNSCache cache(always_fail);
+
+ std::string ip;
+ for (int i = 0; i < 5; ++i) {
+ EXPECT_FALSE(cache.get("never-cached.test", &ip).ok());
+ }
+
+ EXPECT_EQ(5, resolver_calls) << "with the negative cache disabled every
call resolves";
+ EXPECT_EQ(0u, cache.size_for_test());
+ EXPECT_EQ(0u, cache.failure_count_for_test("never-cached.test"))
+ << "failure_count must not grow for a host that was never
successfully resolved";
+}
+
+// After a hostname is evicted, subsequent get() calls must not re-accumulate
+// entries in failure_count (fix for 7.2).
+TEST_F(DNSCacheTest, failure_count_does_not_grow_after_eviction) {
+ config::dns_cache_max_consecutive_failures = 2;
+ config::dns_cache_negative_ttl_seconds = 3600; // keep negative cache
active
+
+ bool should_fail = false;
+ DNSCache cache(make_resolver(&should_fail));
+
+ // Populate cache, then evict.
+ std::string ip;
+ ASSERT_TRUE(cache.get("fake-host.test", &ip).ok());
+ should_fail = true;
+ cache.refresh_for_test();
+ cache.refresh_for_test();
+ ASSERT_EQ(0u, cache.size_for_test()) << "prerequisite: host must be
evicted";
+ EXPECT_EQ(0u, cache.failure_count_for_test("fake-host.test"))
+ << "failure_count must be cleared on eviction";
+ EXPECT_EQ(1u, cache.negative_cache_size_for_test())
+ << "evicted host must be in the negative cache";
+
+ // Further get() calls on the evicted host are served from the negative
cache
+ // and must not reach the resolver, so failure_count stays zero.
+ for (int i = 0; i < 5; ++i) {
+ EXPECT_FALSE(cache.get("fake-host.test", &ip).ok());
+ }
+ EXPECT_EQ(0u, cache.failure_count_for_test("fake-host.test"))
+ << "failure_count must not grow for an evicted host";
+}
+
+// Race defense: simulate concurrent eviction happening between
_resolve_hostname's
+// shared_lock read of cached_ip and the unique_lock used to ++failure_count.
+// The injected resolver erases the host while DNS resolution is "in flight",
+// mimicking what the refresh thread would do. Under the cache.find() re-check
+// added to the unique_lock section, failure_count must NOT be re-introduced.
+TEST_F(DNSCacheTest, failure_count_not_reintroduced_on_eviction_race) {
+ config::dns_cache_max_consecutive_failures = 1000; // disable auto-eviction
+
+ DNSCache* cache_ptr = nullptr;
+ auto racing_resolver = [&cache_ptr](const std::string& host, std::string&,
bool,
+ int* gai_err) -> Status {
+ // Simulate the refresh thread's _erase() landing during the
+ // small window when _resolve_hostname holds no lock.
+ if (cache_ptr != nullptr) {
+ cache_ptr->_erase(host);
+ }
+ if (gai_err != nullptr) {
+ *gai_err = EAI_NONAME;
+ }
+ return Status::InternalError("mock DNS failure during eviction race");
+ };
+
+ DNSCache cache(racing_resolver);
+ cache_ptr = &cache;
+
+ // Pre-populate so cached_ip is non-empty at the shared_lock read,
+ // bypassing _update (which would re-insert after the racing erase).
+ {
+ std::unique_lock<std::shared_mutex> lock(cache.mutex);
+ cache.cache["racing.test"] = "1.2.3.4";
+ }
+ ASSERT_EQ(1u, cache.size_for_test());
+
+ // Call _resolve_hostname directly (via friend access) so the caller-side
+ // re-insert in _update does not mask the behavior we want to verify.
+ std::string returned = cache._resolve_hostname("racing.test");
+
+ // _resolve_hostname returns the cached_ip captured before the race.
+ EXPECT_EQ("1.2.3.4", returned);
+ // The racing erase removed the host from cache.
+ EXPECT_EQ(0u, cache.size_for_test());
+ // The re-check under unique_lock prevented re-creating a failure_count
entry.
+ EXPECT_EQ(0u, cache.failure_count_for_test("racing.test"))
+ << "failure_count must not be re-introduced for a host evicted
mid-resolution";
+}
+
+// ── negative cache tests
──────────────────────────────────────────────────────
+
+// After a hostname is evicted, get() must return an error immediately without
+// invoking the resolver, as long as the negative-cache TTL has not expired.
+TEST_F(DNSCacheTest, negative_cache_blocks_resolver_after_eviction) {
+ config::dns_cache_max_consecutive_failures = 2;
+ config::dns_cache_negative_ttl_seconds = 3600; // will not expire during
test
+
+ int resolver_calls = 0;
+ bool should_fail = false;
+ DNSCache cache(make_counting_resolver(&should_fail, &resolver_calls));
+
+ std::string ip;
+ ASSERT_TRUE(cache.get("fake-host.test", &ip).ok());
+ ASSERT_EQ(1, resolver_calls);
+
+ // Evict via failures.
+ should_fail = true;
+ cache.refresh_for_test();
+ cache.refresh_for_test();
+ ASSERT_EQ(0u, cache.size_for_test()) << "prerequisite: host must be
evicted";
+ ASSERT_EQ(1u, cache.negative_cache_size_for_test());
+
+ int calls_at_eviction = resolver_calls;
+ for (int i = 0; i < 5; ++i) {
+ EXPECT_FALSE(cache.get("fake-host.test", &ip).ok());
+ }
+ EXPECT_EQ(calls_at_eviction, resolver_calls)
+ << "resolver must not be called while host is in the negative
cache";
+}
+
+// Once the negative-cache TTL expires (simulated by clearing the map), get()
+// must attempt a fresh resolve; on success the host re-enters the main cache
+// and the negative-cache entry is removed.
+TEST_F(DNSCacheTest, negative_cache_retries_after_ttl_expiry) {
+ config::dns_cache_max_consecutive_failures = 1;
+ config::dns_cache_negative_ttl_seconds = 3600;
+
+ int resolver_calls = 0;
+ bool should_fail = false;
+ DNSCache cache(make_counting_resolver(&should_fail, &resolver_calls));
+
+ std::string ip;
+ ASSERT_TRUE(cache.get("fake-host.test", &ip).ok());
+
+ // Evict.
+ should_fail = true;
+ cache.refresh_for_test();
+ ASSERT_EQ(0u, cache.size_for_test());
+ ASSERT_EQ(1u, cache.negative_cache_size_for_test());
+
+ // Negative cache blocks the resolver.
+ EXPECT_FALSE(cache.get("fake-host.test", &ip).ok());
+ int calls_while_blocked = resolver_calls;
+
+ // Simulate TTL expiry by backdating the entry.
+ cache._expire_negative_cache_for_test();
+ EXPECT_EQ(1u, cache.negative_cache_size_for_test()); // entry exists but
expired
+
+ // DNS recovers.
+ should_fail = false;
+ EXPECT_TRUE(cache.get("fake-host.test", &ip).ok());
+ EXPECT_GT(resolver_calls, calls_while_blocked) << "resolver must be called
after TTL expiry";
+ EXPECT_EQ(1u, cache.size_for_test()) << "host must be re-cached after
successful re-resolve";
+ EXPECT_EQ(0u, cache.negative_cache_size_for_test())
+ << "negative cache entry must be removed on successful re-resolve";
+}
+
+// A concurrent successful resolution between _update() returning and the
+// threshold check must prevent eviction: _erase_if_still_failing() re-reads
+// the live failure_count under its own lock so a reset counter is not lost.
+TEST_F(DNSCacheTest, concurrent_success_prevents_stale_eviction) {
+ config::dns_cache_max_consecutive_failures = 1;
+ config::dns_cache_negative_ttl_seconds = 3600;
+
+ bool should_fail = false;
+ DNSCache cache(make_resolver(&should_fail));
+
+ std::string ip;
+ ASSERT_TRUE(cache.get("fake-host.test", &ip).ok());
+ ASSERT_EQ(1u, cache.size_for_test());
+
+ // Simulate: _update() returned failures == threshold, but before _erase()
+ // was called a concurrent success cleared failure_count.
+ {
+ std::unique_lock<std::shared_mutex> lock(cache.mutex);
+ cache.failure_count["fake-host.test"] = {1, true}; // at threshold,
authoritative
+ cache.failure_count.erase("fake-host.test"); // concurrent
success clears it
+ }
+
+ bool erased = cache._erase_if_still_failing("fake-host.test", 1u);
+ EXPECT_FALSE(erased) << "must not erase when failure_count was
concurrently reset to zero";
+ EXPECT_EQ(1u, cache.size_for_test()) << "host must survive after
concurrent success";
+ EXPECT_EQ(0u, cache.negative_cache_size_for_test());
+}
+
+// When _resolve_hostname returns a stale cached IP after a concurrent
eviction,
+// _update must not reinsert it (which would undo the eviction and clear the
+// negative-cache tombstone).
+TEST_F(DNSCacheTest, stale_fallback_not_reinserted_after_concurrent_eviction) {
+ config::dns_cache_max_consecutive_failures = 1000; // disable
threshold-based eviction
+ config::dns_cache_negative_ttl_seconds = 3600;
+
+ DNSCache* cache_ptr = nullptr;
+ auto racing_resolver = [&cache_ptr](const std::string& host, std::string&,
bool,
+ int* gai_err) -> Status {
+ if (cache_ptr) {
+ cache_ptr->_erase(host); // evict mid-DNS-call
+ }
+ if (gai_err != nullptr) {
+ *gai_err = EAI_NONAME;
+ }
+ return Status::InternalError("mock DNS failure during concurrent
eviction");
+ };
+
+ DNSCache cache(racing_resolver);
+ cache_ptr = &cache;
+
+ // Pre-populate so _resolve_hostname reads a non-empty cached_ip before the
+ // DNS call, then the racing erase fires during the call.
+ {
+ std::unique_lock<std::shared_mutex> lock(cache.mutex);
+ cache.cache["racing.test"] = "1.2.3.4";
+ }
+ ASSERT_EQ(1u, cache.size_for_test());
+
+ // Drive one refresh cycle: resolver evicts mid-DNS, returns failure.
+ // _resolve_hostname returns the stale "1.2.3.4". Without the guard
_update
+ // would reinsert it; with the guard it must not.
+ cache.refresh_for_test();
+
+ EXPECT_EQ(0u, cache.size_for_test()) << "stale IP must not be reinserted
after eviction";
+ EXPECT_EQ(1u, cache.negative_cache_size_for_test()) << "host must be in
negative cache";
+ EXPECT_EQ(0u, cache.failure_count_for_test("racing.test"));
+}
+
+// After negative-cache TTL expiry and DNS still failing, get() must re-arm the
+// negative cache so the retry rate stays bounded at one attempt per TTL
period.
+TEST_F(DNSCacheTest,
negative_cache_rearms_on_continued_failure_after_ttl_expiry) {
+ config::dns_cache_max_consecutive_failures = 1;
+ config::dns_cache_negative_ttl_seconds = 3600;
+
+ int resolver_calls = 0;
+ bool should_fail = false;
+ DNSCache cache(make_counting_resolver(&should_fail, &resolver_calls));
+
+ std::string ip;
+ ASSERT_TRUE(cache.get("fake-host.test", &ip).ok());
+
+ // Evict.
+ should_fail = true;
+ cache.refresh_for_test();
+ ASSERT_EQ(0u, cache.size_for_test());
+ ASSERT_EQ(1u, cache.negative_cache_size_for_test());
+
+ // Simulate TTL expiry by backdating the entry (entry still exists, time
is past).
+ // Use _expire (not _clear) so the entry is visible to get()'s TTL check,
+ // which sets expired_negative=true and triggers the re-arm path on
failure.
+ cache._expire_negative_cache_for_test();
+ ASSERT_EQ(1u, cache.negative_cache_size_for_test()); // entry exists but
expired
+
+ // DNS still failing: one retry is allowed, then negative cache re-arms.
+ int calls_before = resolver_calls;
+ EXPECT_FALSE(cache.get("fake-host.test", &ip).ok());
+ EXPECT_GT(resolver_calls, calls_before) << "resolver called after TTL
expiry";
+ EXPECT_EQ(1u, cache.negative_cache_size_for_test())
+ << "negative cache re-armed on continued failure";
+
+ // Subsequent calls are now blocked without hitting the resolver.
+ calls_before = resolver_calls;
+ EXPECT_FALSE(cache.get("fake-host.test", &ip).ok());
+ EXPECT_EQ(calls_before, resolver_calls) << "resolver NOT called while
re-armed";
+}
+
+// ── Finding 1: mutable TTL is honored by existing tombstones
─────────────────
+
+// Setting dns_cache_negative_ttl_seconds to 0 after eviction must immediately
+// disable the negative cache for existing entries.
+TEST_F(DNSCacheTest, negative_cache_disabled_when_ttl_set_to_zero) {
+ config::dns_cache_max_consecutive_failures = 1;
+ config::dns_cache_negative_ttl_seconds = 3600;
+
+ int resolver_calls = 0;
+ bool should_fail = false;
+ DNSCache cache(make_counting_resolver(&should_fail, &resolver_calls));
+
+ std::string ip;
+ ASSERT_TRUE(cache.get("fake-host.test", &ip).ok());
+
+ // Evict with TTL=3600.
+ should_fail = true;
+ cache.refresh_for_test();
+ ASSERT_EQ(0u, cache.size_for_test());
+ ASSERT_EQ(1u, cache.negative_cache_size_for_test());
+
+ // Negative cache blocks.
+ int calls_at_eviction = resolver_calls;
+ EXPECT_FALSE(cache.get("fake-host.test", &ip).ok());
+ EXPECT_EQ(calls_at_eviction, resolver_calls);
+
+ // Now disable negative cache by setting TTL to 0 — simulates config
change.
+ config::dns_cache_negative_ttl_seconds = 0;
+ should_fail = false;
+
+ // get() must now retry immediately (TTL disabled) and succeed.
+ EXPECT_TRUE(cache.get("fake-host.test", &ip).ok());
+ EXPECT_GT(resolver_calls, calls_at_eviction) << "resolver must be called
when TTL is disabled";
+ EXPECT_EQ(1u, cache.size_for_test());
+}
+
+// Decreasing dns_cache_negative_ttl_seconds applies to existing tombstones:
+// a tombstone created with TTL=3600 must honor the new smaller TTL.
+TEST_F(DNSCacheTest, negative_cache_honors_decreased_ttl) {
+ config::dns_cache_max_consecutive_failures = 1;
+ config::dns_cache_negative_ttl_seconds = 3600;
+
+ bool should_fail = false;
+ DNSCache cache(make_resolver(&should_fail));
+
+ std::string ip;
+ ASSERT_TRUE(cache.get("fake-host.test", &ip).ok());
+
+ // Evict — tombstone stores eviction_time = now.
+ should_fail = true;
+ cache.refresh_for_test();
+ ASSERT_EQ(1u, cache.negative_cache_size_for_test());
+
+ // Decrease TTL to 1 second and backdate the entry to make it look older.
+ config::dns_cache_negative_ttl_seconds = 1;
+ cache._expire_negative_cache_for_test(); // backdate far → past any 1s
deadline
+
+ should_fail = false;
+ EXPECT_TRUE(cache.get("fake-host.test", &ip).ok())
+ << "reduced TTL must be honored for existing tombstones";
+}
+
+// ── Finding 2: single-flight retry ──────────────────────────────────────────
+
+// Only one thread claims the expired negative-cache retry; a second concurrent
+// caller sees the re-armed entry and gets the cheap error.
+TEST_F(DNSCacheTest, single_flight_retry_after_negative_ttl_expiry) {
+ config::dns_cache_max_consecutive_failures = 1;
+ config::dns_cache_negative_ttl_seconds = 3600;
+
+ int resolver_calls = 0;
+ bool should_fail = false;
+ DNSCache cache(make_counting_resolver(&should_fail, &resolver_calls));
+
+ std::string ip;
+ ASSERT_TRUE(cache.get("fake-host.test", &ip).ok());
+
+ // Evict and expire the tombstone.
+ should_fail = true;
+ cache.refresh_for_test();
+ cache._expire_negative_cache_for_test();
+
+ // First get() claims the retry — DNS still fails, re-arm happens.
+ int calls_before = resolver_calls;
+ EXPECT_FALSE(cache.get("fake-host.test", &ip).ok());
+ EXPECT_EQ(calls_before + 1, resolver_calls) << "first caller invokes DNS";
+
+ // Second get() must NOT invoke DNS — the entry was re-armed by the first
call.
+ calls_before = resolver_calls;
+ EXPECT_FALSE(cache.get("fake-host.test", &ip).ok());
+ EXPECT_EQ(calls_before, resolver_calls)
+ << "second caller must be blocked by the re-armed entry";
+}
+
+// ── Finding 4: refresh cleanup doesn't break re-arm ─────────────────────────
+
+// After _refresh_once() runs, the negative-cache tombstone must still be
present
+// so that get() can recognize the host as evicted and bound retry rate.
+TEST_F(DNSCacheTest, refresh_does_not_erase_negative_cache_tombstone) {
+ config::dns_cache_max_consecutive_failures = 1;
+ config::dns_cache_negative_ttl_seconds = 3600;
+
+ int resolver_calls = 0;
+ bool should_fail = false;
+ DNSCache cache(make_counting_resolver(&should_fail, &resolver_calls));
+
+ std::string ip;
+ ASSERT_TRUE(cache.get("fake-host.test", &ip).ok());
+
+ // Evict and expire the tombstone to simulate passage of time.
+ should_fail = true;
+ cache.refresh_for_test();
+ ASSERT_EQ(1u, cache.negative_cache_size_for_test());
+ cache._expire_negative_cache_for_test();
+
+ // Run another refresh cycle — must NOT erase the expired tombstone. The
+ // tombstone is what keeps get() from issuing a blocking getaddrinfo on
every
+ // call, so it must outlive refresh cycles regardless of age.
+ cache.refresh_for_test();
+ EXPECT_EQ(1u, cache.negative_cache_size_for_test())
+ << "refresh must not erase negative-cache tombstones";
+
+ // get() still recognizes this as an evicted host: one retry then re-arm.
+ int calls_before = resolver_calls;
+ EXPECT_FALSE(cache.get("fake-host.test", &ip).ok());
+ EXPECT_EQ(calls_before + 1, resolver_calls) << "one retry after expiry";
+
+ // Subsequent call is blocked.
+ calls_before = resolver_calls;
+ EXPECT_FALSE(cache.get("fake-host.test", &ip).ok());
+ EXPECT_EQ(calls_before, resolver_calls) << "blocked after re-arm";
+}
+
+// ── only authoritative NXDOMAIN may evict
────────────────────────────────────
+
+// A resolver outage reports transient errors (EAI_AGAIN) for every hostname
at once.
+// Those must never evict: the backends are still alive at their last known
IP, and
+// dropping the cache would turn a DNS incident into a cluster-wide RPC outage.
+TEST_F(DNSCacheTest, transient_failure_never_evicts) {
+ config::dns_cache_max_consecutive_failures = 3;
+ config::dns_cache_negative_ttl_seconds = 3600;
+
+ bool should_fail = false;
+ DNSCache cache(make_resolver(&should_fail, "1.2.3.4", EAI_AGAIN));
+
+ std::string ip;
+ ASSERT_TRUE(cache.get("fake-host.test", &ip).ok());
+ ASSERT_EQ(1u, cache.size_for_test());
+
+ // The resolver is now unreachable, but the host itself is fine.
+ should_fail = true;
+
+ // Far more cycles than the threshold would need if the failures counted.
+ for (int i = 0; i < 20; ++i) {
+ cache.refresh_for_test();
+ ASSERT_EQ(1u, cache.size_for_test())
+ << "transient DNS failures must not evict (cycle " << i << ")";
+ }
+ EXPECT_EQ(0u, cache.negative_cache_size_for_test()) << "no tombstone for a
transient failure";
+ // The failure counter still advances so the throttled warning stays
informative.
+ EXPECT_EQ(20u, cache.failure_count_for_test("fake-host.test"));
+
+ // Callers keep getting the last known good IP - this is the graceful
degradation
+ // that lets the cluster ride out a resolver outage.
+ ip.clear();
+ EXPECT_TRUE(cache.get("fake-host.test", &ip).ok());
+ EXPECT_EQ("1.2.3.4", ip);
+}
+
+// Once the resolver comes back and authoritatively answers NXDOMAIN, the host
is
+// evicted on that cycle - the failures accumulated during the outage still
count
+// toward the threshold, only the eviction action was withheld.
+TEST_F(DNSCacheTest, authoritative_failure_after_transient_evicts) {
+ config::dns_cache_max_consecutive_failures = 3;
+ config::dns_cache_negative_ttl_seconds = 3600;
+
+ bool should_fail = false;
+ int fail_gai_err = EAI_AGAIN;
+ DNSCache cache([&should_fail, &fail_gai_err](const std::string&,
std::string& out, bool,
+ int* gai_err) -> Status {
+ if (should_fail) {
+ *gai_err = fail_gai_err;
+ return Status::InternalError("mock failure");
+ }
+ *gai_err = 0;
+ out = "1.2.3.4";
+ return Status::OK();
+ });
+
+ std::string ip;
+ ASSERT_TRUE(cache.get("fake-host.test", &ip).ok());
+
+ // Resolver outage: well past the threshold, still no eviction.
+ should_fail = true;
+ for (int i = 0; i < 5; ++i) {
+ cache.refresh_for_test();
+ }
+ ASSERT_EQ(1u, cache.size_for_test()) << "prerequisite: transient failures
did not evict";
+
+ // The resolver recovers and states the name does not exist.
+ fail_gai_err = EAI_NONAME;
+ cache.refresh_for_test();
+
+ EXPECT_EQ(0u, cache.size_for_test())
+ << "an authoritative NXDOMAIN past the threshold must evict";
+ EXPECT_EQ(1u, cache.negative_cache_size_for_test());
+}
+
+// _erase_if_still_failing() re-reads the state under the erase lock, so a
host whose
+// most recent failure turned transient is spared even if the count is past
threshold.
+TEST_F(DNSCacheTest, erase_skipped_when_last_failure_is_transient) {
+ config::dns_cache_max_consecutive_failures = 1;
+ config::dns_cache_negative_ttl_seconds = 3600;
+
+ bool should_fail = false;
+ DNSCache cache(make_resolver(&should_fail));
+
+ std::string ip;
+ ASSERT_TRUE(cache.get("fake-host.test", &ip).ok());
+ ASSERT_EQ(1u, cache.size_for_test());
+
+ {
+ std::unique_lock<std::shared_mutex> lock(cache.mutex);
+ // Past the threshold, but the latest failure was a resolver problem.
+ cache.failure_count["fake-host.test"] = {5, false};
+ }
+
+ EXPECT_FALSE(cache._erase_if_still_failing("fake-host.test", 1u))
+ << "must not erase when the most recent failure was not
authoritative";
+ EXPECT_EQ(1u, cache.size_for_test());
+ EXPECT_EQ(0u, cache.negative_cache_size_for_test());
+}
+
+// ── negative cache covers hosts that never resolved
──────────────────────────
+
+// A hostname that has never resolved never enters `cache`, so it never
reaches the
+// eviction path. It must still be tombstoned, otherwise every get() pays a
full
+// blocking getaddrinfo - and many of those calls run on bthreads.
+TEST_F(DNSCacheTest, first_miss_is_negative_cached) {
+ config::dns_cache_negative_ttl_seconds = 3600;
+
+ int resolver_calls = 0;
+ bool should_fail = true;
+ DNSCache cache(make_counting_resolver(&should_fail, &resolver_calls));
+
+ std::string ip;
+ EXPECT_FALSE(cache.get("never-resolved.test", &ip).ok());
+ EXPECT_EQ(1, resolver_calls) << "the first call must actually try to
resolve";
+ EXPECT_EQ(0u, cache.size_for_test());
+ EXPECT_EQ(1u, cache.negative_cache_size_for_test())
+ << "a host that never resolved must still be tombstoned";
+
+ // Every later call is served from the negative cache without touching DNS.
+ for (int i = 0; i < 5; ++i) {
+ EXPECT_FALSE(cache.get("never-resolved.test", &ip).ok());
+ }
+ EXPECT_EQ(1, resolver_calls) << "resolver must not be called again within
the TTL";
+ EXPECT_EQ(0u, cache.failure_count_for_test("never-resolved.test"))
+ << "failure_count must stay empty for a host that was never
cached";
+
+ // Once the TTL lapses and DNS starts working, the host is picked up
normally.
+ cache._expire_negative_cache_for_test();
+ should_fail = false;
+ EXPECT_TRUE(cache.get("never-resolved.test", &ip).ok());
+ EXPECT_EQ("1.2.3.4", ip);
+ EXPECT_EQ(1u, cache.size_for_test());
+ EXPECT_EQ(0u, cache.negative_cache_size_for_test());
+}
+
+// Setting the TTL to 0 keeps the legacy behavior: no tombstone for a first
miss.
+TEST_F(DNSCacheTest, first_miss_not_negative_cached_when_ttl_disabled) {
+ config::dns_cache_negative_ttl_seconds = 0;
+
+ int resolver_calls = 0;
+ bool should_fail = true;
+ DNSCache cache(make_counting_resolver(&should_fail, &resolver_calls));
+
+ std::string ip;
+ for (int i = 0; i < 3; ++i) {
+ EXPECT_FALSE(cache.get("never-resolved.test", &ip).ok());
+ }
+ EXPECT_EQ(0u, cache.negative_cache_size_for_test());
+ EXPECT_EQ(3, resolver_calls) << "every call resolves when the negative
cache is disabled";
+}
+
+// ── shutdown responsiveness
──────────────────────────────────────────────────
+
+// _refresh_once() must bail out as soon as the stop flag is set. Each
_update() inside
+// it blocks in getaddrinfo, so a cycle over a cluster with dead DNS can run
for minutes
+// and would otherwise hold up ~DNSCache()'s join() for exactly that long.
+TEST_F(DNSCacheTest, refresh_stops_early_when_stopping) {
+ config::dns_cache_max_consecutive_failures = 0; // isolate: no eviction in
play
+
+ int resolver_calls = 0;
+ bool should_fail = false;
+ DNSCache cache(make_counting_resolver(&should_fail, &resolver_calls));
+
+ std::string ip;
+ ASSERT_TRUE(cache.get("host-a.test", &ip).ok());
+ ASSERT_EQ(1u, cache.size_for_test());
+
+ int calls_before = resolver_calls;
+ cache.stop_refresh = true;
+ cache.refresh_for_test();
+
+ EXPECT_EQ(calls_before, resolver_calls)
+ << "a refresh cycle must not start resolving once the stop flag is
set";
+ EXPECT_EQ(1u, cache.size_for_test()) << "bailing out must leave the cache
untouched";
+}
+
+} // end of namespace doris
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
