Andrey Yarovoy created HDDS-16776:
-------------------------------------
Summary: OM and SCM re-run a failing topology script on every
client request because failed rack resolutions are not cached
Key: HDDS-16776
URL: https://issues.apache.org/jira/browse/HDDS-16776
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Manager, SCM
Reporter: Andrey Yarovoy
h2. Problem
When \{{net.topology.node.switch.mapping.impl}} is left at its default of
{\{org.apache.hadoop.net.ScriptBasedMapping}} and
\{{net.topology.script.file.name}} points to a script
that the service cannot execute -- missing file, wrong path, not executable,
wrong working directory --
both OM and SCM attempt to fork that script *once per client request*, forever.
Nothing in the stack
remembers that the previous attempt failed.
On a cluster where the OM role could not reach its configured script driven by
only a handful of distinct client hosts-- the rate comes from repetition, not
from a large client population. Each failure logs a full stack trace at WARN.
Each occurrence is a ~45-line WARN with this shape (line numbers elided -- the
observed cluster ran a vendor Hadoop build):
{code}
WARN [IPC Server handler 6 on default port
9862]-org.apache.hadoop.net.ScriptBasedMapping: \
Exception running /path/to/topology.py <client-ip>
java.io.IOException: Cannot run program "/path/to/topology.py" (in directory
"<redacted>"): \
error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java)
at org.apache.hadoop.util.Shell.runCommand(Shell.java)
...
at
org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java)
at
org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java)
at
org.apache.hadoop.ozone.om.KeyManagerImpl.resolveNodeLocation(KeyManagerImpl.java)
at org.apache.hadoop.ozone.om.KeyManagerImpl.getOtherNode(KeyManagerImpl.java)
at org.apache.hadoop.ozone.om.KeyManagerImpl.getClientNode(KeyManagerImpl.java)
at org.apache.hadoop.ozone.om.KeyManagerImpl.sortDatanodes(KeyManagerImpl.java)
...
{code}
h2. Root cause
{\{ScriptBasedMapping}} extends Hadoop's \{{CachedDNSToSwitchMapping}}, so
successful resolutions *are*
cached -- but failures are not. The raw mapping returns \{{null}} for the whole
batch when the script
cannot be run (\{{ScriptBasedMapping.RawScriptBasedMapping.resolve}}), and the
caching layer skips nulls:
{code:java}
// CachedDNSToSwitchMapping
private void cacheResolvedHosts(List<String> uncachedHosts, List<String>
resolvedHosts) {
if (resolvedHosts != null) { // <-- a failure is never cached
for (int i = 0; i < uncachedHosts.size(); i++) {
cache.put(uncachedHosts.get(i), resolvedHosts.get(i));
}
}
}
{code}
Ozone adds no negative caching on top, so every request repeats the fork. Both
services resolve the
*client* host on the request thread:
* OM read path -- \{{KeyManagerImpl.sortDatanodes}} -> \{{getClientNode}} ->
\{{getOtherNode}} ->
{\{resolveNodeLocation}} (KeyManagerImpl.java:2311, 2382, 2388, 2404), reached
by \{{lookupKey}},
{\{getKeyInfo}}, \{{lookupFile}}, \{{getFileStatus}} and \{{listStatus}}.
* OM write path -- \{{OMKeyRequest}} ->
\{{KeyManagerImpl.sortDatanodesForWrite}} -> \{{getClientNode}}
(OMKeyRequest.java:253, KeyManagerImpl.java:2323).
* SCM -- \{{SCMBlockProtocolServer.getClientNode}} -> \{{getOtherNode}} ->
{\{StorageContainerManager.resolveNodeLocation}}
(SCMBlockProtocolServer.java:412, 422, 424), from both
the \{{SortDatanodes}} RPC (:387) and \{{allocateBlock}}'s per-block loop
(:212).
Two things amplify the rate:
# \{{getClientNode}} is invoked once per *distinct pipeline* within a single
request. The sorted-order memoisation in \{{KeyManagerImpl.sortDatanodes}} is
keyed by the pipeline's node set and does not cover client resolution, so one
\{{listStatus}} spanning M pipelines performs M resolutions for the same client
host.
# The OM only matches the client against the 3-5 nodes of the pipeline being
sorted. SCM, by contrast, consults \{{SCMNodeManager.getNodesByAddress}} first,
so a client running on a datanode host never reaches the script there. OM has
no such index, so co-located compute clients fall through to the script on
nearly every request.
* *Log flood.* ~45 lines per occurrence at ~1,500 occurrences/min is on the
order of 65,000 WARN lines and ~6.7 MB of OM log *per minute*. Real OM events
are effectively
unfindable, and log retention is destroyed.
* *A process fork per request, on the RPC handler thread.*
\{{ProcessBuilder.start}} plus
{\{Shell.runCommand}} on an IPC handler is a handler-occupancy and CPU cost
paid on every read and every block allocation.
* *Topology awareness silently stops working.* With the client unresolved,
{\{NetworkTopologyImpl.sortByDistanceCost}} takes its \{{reader == null}}
branch and *shuffles* the datanode list, so reads get a random order rather
than a locality-ordered one. Writes keep pipeline order. Nothing reports that
locality has been lost.
* *No actionable signal from Ozone.* Ozone's own handler never fires --
\{{ScriptBasedMapping}} swallows
the \{{IOException}} and returns \{{null}}, so \{{getOtherNode}}'s \{{catch}}
block is never entered ("Could not resolve client" appeared *zero* times in the
log). The only evidence is a third-party WARN that Ozone cannot throttle or
suppress.
h2. Proposed fix
Cache the *failure*. Add a short-lived negative cache in front of the
configured \{{DNSToSwitchMapping}}: when resolution yields no location for a
host, remember that for a configurable period and return no location for that
host without invoking the mapping again.
Default *15 minutes*, configurable, with 0 disabling the behaviour. This bounds
the call rate to (distinct client hosts / expiry), which is independent of
request rate -- in the observed case roughly
*one call per minute* instead of ~N, three orders of magnitude lower, while
still recovering
automatically once the script is fixed, within one expiry period and without an
OM or SCM restart.
Keep the entries *per host*, not global. The raw mapping returns \{{null}} both
for "the script is broken" and for "the script does not know this host", and
these calls are single-host, so the two cases are indistinguishable at this
layer; per-host entries bound the rate without letting one unknown host
suppress resolution for every other client.
Both services should share the implementation: \{{createDNSToSwitchMapping}}
and \{{resolveNodeLocation}}
are currently duplicated byte-for-byte between \{{StorageContainerManager}}
(lines 2351-2370) and
{\{KeyManagerImpl}} (lines 383-389, 2404-2414). Folding both into one helper in
\{{hdds-server-framework}} removes the duplication and fixes OM and SCM
together.
h2. Out of scope
* Resolving the client host from the topology the OM already holds (an address
index equivalent to SCM's \{{getNodesByAddress}}). This would both avoid the
script *and* restore correct locality for co-located clients, but only for
clients that are datanodes -- a separate improvement.
* Validating the configured script at service startup so a misconfiguration
surfaces once, loudly, instead of per request.
* Resolving the client once per request instead of once per pipeline.
h2. Acceptance criteria
# With a mapping that fails, a service resolves a given client host at most
once per expiry period, and the WARN volume falls accordingly.
# The failure is not cached when the expiry is configured to 0, preserving
today's behaviour.
# Once resolution starts succeeding, locations are resolved again within one
expiry period with no restart.
# Successful resolutions are unaffected -- they continue to be cached by the
existing mechanism.
# OM and SCM share one code path for creating the mapping and resolving a node
location.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
