[
https://issues.apache.org/jira/browse/HDDS-17555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksei Ieshin updated HDDS-17555:
----------------------------------
Attachment: TestReadPathTopologySort.java
> Topology-aware read in OM ignores racks: pipeline datanodes are not mapped
> onto OM's cluster map before sorting
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17555
> URL: https://issues.apache.org/jira/browse/HDDS-17555
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.2.1
> Reporter: Aleksei Ieshin
> Priority: Major
> Attachments: TestReadPathTopologySort.java
>
>
> h2. Problem
> On the read path, OM sorts each block's datanodes by distance to the client
> ({{{}KeyManagerImpl.sortDatanodes(String, List<OmKeyInfo>){}}} →
> {{sortDatanodes(nodes, clientMachine)}} →
> {{{}NetworkTopology.sortByDistanceCost{}}}). The nodes come straight from
> {{{}pipeline.getNodes(){}}}. Those pipelines reach OM from SCM over RPC, so
> each {{DatanodeDetails}} is built by {{{}DatanodeDetails.getFromProtoBuf{}}}.
> It carries {{networkLocation}} and {{{}level{}}}, but {*}no parent{*}: it
> isn't linked into OM's cluster map.
> {{NetworkTopologyImpl.getDistanceCost(client, node)}} walks
> {{node.getAncestor(level - 1)}} to the root. For an unlinked node that
> returns {{{}null{}}}, so the method returns {{{}Integer.MAX_VALUE{}}}. Every
> pipeline node therefore gets the same cost unless it _equals_ the client. The
> sort reduces to "client's own node first (when the client is a datanode in
> the pipeline), then random". Rack locality is never applied to reads.
> HDDS-15059 (2.3.0) describes exactly this in a code comment for the
> streaming-write path ("The nodes come from SCM over RPC, so they are
> deserialized DatanodeDetails with no parent/level: the topology treats them
> as unknown (distance Integer.MAX_VALUE) and the order comes out random").
> In the HDDS-15059 review ([PR
> #10633|https://github.com/apache/ozone/pull/10633]) the read path was
> explicitly left untouched.
> h2. Reproduction
> A unit test in {{hadoop-hdds/framework}}
> ({{{}org.apache.hadoop.hdds.scm.net{}}}) on 2.2.1:
> # Build a {{NetworkTopologyImpl}} with ROOT/RACK/LEAF schemas and 6
> datanodes on {{{}/r1{}}}, {{{}/r2{}}}, {{{}/r3{}}}, two per rack.
> # Make "pipeline" copies the way OM receives them:
> {{{}DatanodeDetails.getFromProtoBuf(dn.getProtoBufMessage()){}}}.
> # Use a non-datanode client on {{{}/r1{}}}, built the way
> {{KeyManagerImpl.getOtherNode}} does it: {{{}new NodeImpl(host, "/r1", rack,
> rack.getLevel() + 1, NODE_COST_DEFAULT){}}}.
> ||Measure||Topology-linked nodes||Deserialized (as on OM's read path)||
> |{{level}} / {{parent}}|3 / {{{}/r1{}}}...|3 / {{null}}|
> |{{getDistanceCost(client, node)}}|2 same rack, 4 other rack|{{2147483647}}
> for all|
> |Both {{/r1}} nodes sorted first, 1000 sorts|1000|76 (random: 2/6 × 1/5 ≈
> 6.7%)|
> |Client = datanode dn0 on {{{}/r1{}}}: other {{/r1}} node sorted
> second|n/a|179/1000 (random ≈ 20%)|
> Test source attached: {{{}TestReadPathTopologySort.java{}}}.
> h2. Impact
> * {{ozone.network.topology.aware.read=true}} (the default) gives no rack
> locality for RATIS reads. Only same-host preference survives.
> * It still costs the full sort on every read. Each node's
> {{getDistanceCost}} takes {{{}NetworkTopologyImpl{}}}'s global read lock
> before returning {{{}MAX_VALUE{}}}. Under heavy read load that lock is a CPU
> hot spot (see HDDS-17553).
> * EC reads aren't affected by the ordering ({{{}ECBlockInputStream{}}}
> selects nodes by replica index from {{{}getNodes(){}}}).
> h2. Proposed fix
> Give the read path real rack distances for pipeline nodes that arrive from
> SCM without a parent, {*}without adding per-node lock acquisitions{*}:
> * Preferred: compute the sort cost from the node's
> {{{}networkLocation{}}}/{{{}level{}}} (lock-free, like HDFS
> {{{}NetworkTopology.getWeightUsingNetworkLocation{}}}, using the schema's
> per-level costs), so unlinked nodes need no tree lookup.
> Independent of the lock-contention fix (HDDS-17553). Unlinked nodes take the
> path branch before any lock, so this change alone also removes the per-node
> {{netlock}} acquisition from OM's read sort. Both touch
> {{{}sortByDistanceCost{}}}, so whichever lands second needs a small rebase.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
