Aleksei Ieshin created HDDS-17555:
-------------------------------------
Summary: Topology-aware read in OM ignores racks: pipeline
datanodes are not mapped onto OM's cluster map before sorting
Key: HDDS-17555
URL: https://issues.apache.org/jira/browse/HDDS-17555
Project: Apache Ozone
Issue Type: Bug
Components: OM
Affects Versions: 2.2.1, 2.2.0, 2.1.0, 2.0.0
Reporter: Aleksei Ieshin
h2. Problem
On the read path, OM sorts each block's datanodes by distance to the client
({{{}KeyManagerImpl.sortDatanodes(String, List<OmKeyInfo>){}}} →
{{sortDatanodes(nodes, clientMachine)}} →
{{{}NetworkTopology.sortByDistanceCost{}}}). The nodes come straight from
{{{}pipeline.getNodes(){}}}. Those pipelines reach OM from SCM over RPC, so
each {{DatanodeDetails}} is built by {{{}DatanodeDetails.getFromProtoBuf{}}}.
It carries {{networkLocation}} and {{{}level{}}}, but {*}no parent{*}: it isn't
linked into OM's cluster map.
{{NetworkTopologyImpl.getDistanceCost(client, node)}} walks
{{node.getAncestor(level - 1)}} to the root. For an unlinked node that returns
{{{}null{}}}, so the method returns {{{}Integer.MAX_VALUE{}}}. Every pipeline
node therefore gets the same cost unless it _equals_ the client. The sort
reduces to "client's own node first (when the client is a datanode in the
pipeline), then random". Rack locality is never applied to reads.
HDDS-15059 (2.3.0) describes exactly this in a code comment for the
streaming-write path ("The nodes come from SCM over RPC, so they are
deserialized DatanodeDetails with no parent/level: the topology treats them as
unknown (distance Integer.MAX_VALUE) and the order comes out random").
In the HDDS-15059 review ([PR
#10633|https://github.com/apache/ozone/pull/10633]) the read path was
explicitly left untouched.
h2. Reproduction
A unit test in {{hadoop-hdds/framework}}
({{{}org.apache.hadoop.hdds.scm.net{}}}) on 2.2.1:
# Build a {{NetworkTopologyImpl}} with ROOT/RACK/LEAF schemas and 6 datanodes
on {{{}/r1{}}}, {{{}/r2{}}}, {{{}/r3{}}}, two per rack.
# Make "pipeline" copies the way OM receives them:
{{{}DatanodeDetails.getFromProtoBuf(dn.getProtoBufMessage()){}}}.
# Use a non-datanode client on {{{}/r1{}}}, built the way
{{KeyManagerImpl.getOtherNode}} does it: {{{}new NodeImpl(host, "/r1", rack,
rack.getLevel() + 1, NODE_COST_DEFAULT){}}}.
||Measure||Topology-linked nodes||Deserialized (as on OM's read path)||
|{{level}} / {{parent}}|3 / {{{}/r1{}}}...|3 / {{null}}|
|{{getDistanceCost(client, node)}}|2 same rack, 4 other rack|{{2147483647}} for
all|
|Both {{/r1}} nodes sorted first, 1000 sorts|1000|76 (random: 2/6 × 1/5 ≈ 6.7%)|
|Client = datanode dn0 on {{{}/r1{}}}: other {{/r1}} node sorted
second|n/a|179/1000 (random ≈ 20%)|
Test source attached: {{{}TestReadPathTopologySort.java{}}}.
h2. Impact
* {{ozone.network.topology.aware.read=true}} (the default) gives no rack
locality for RATIS reads. Only same-host preference survives.
* It still costs the full sort on every read. Each node's {{getDistanceCost}}
takes {{{}NetworkTopologyImpl{}}}'s global read lock before returning
{{{}MAX_VALUE{}}}. Under heavy read load that lock is a CPU hot spot (see
HDDS-17553).
* EC reads aren't affected by the ordering ({{{}ECBlockInputStream{}}} selects
nodes by replica index from {{{}getNodes(){}}}).
h2. Proposed fix
Give the read path real rack distances for pipeline nodes that arrive from SCM
without a parent, {*}without adding per-node lock acquisitions{*}:
* Preferred: compute the sort cost from the node's
{{{}networkLocation{}}}/{{{}level{}}} (lock-free, like HDFS
{{{}NetworkTopology.getWeightUsingNetworkLocation{}}}, using the schema's
per-level costs), so unlinked nodes need no tree lookup.
Independent of the lock-contention fix (HDDS-17553). Unlinked nodes take the
path branch before any lock, so this change alone also removes the per-node
{{netlock}} acquisition from OM's read sort. Both touch
{{{}sortByDistanceCost{}}}, so whichever lands second needs a small rebase.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
