[
https://issues.apache.org/jira/browse/HDDS-17553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksei Ieshin updated HDDS-17553:
----------------------------------
Description:
h2. Problem
On every read that returns block locations ({{{}GetKeyInfo{}}},
{{{}lookupKey{}}}, {{{}getFileStatus{}}}, {{listStatus}} with locations), OM
sorts each block's datanodes by distance to the client:
{{KeyManagerImpl.sortDatanodes}} → {{NetworkTopologyImpl.sortByDistanceCost}} →
{{getDistanceCost(reader, node)}} *once per node*
{{getDistanceCost}} takes the read lock of {{{}NetworkTopologyImpl.netlock{}}},
a single global {{new ReentrantReadWriteLock(true)}}
({{{}NetworkTopologyImpl.java:65{}}}; the call is at {{:799}} in
{{{}sortByDistanceCost{}}}).
Under read-heavy load this means one lock round-trip per node per block per
request, on a single fair lock shared by every RPC handler. Reads still have to
CAS one shared state word and look up a per-thread hold count in a ThreadLocal.
With 100+ handlers on a large host, that turns into CAS retry loops and long
ThreadLocal probes, and OM CPU climbs while throughput falls. It feeds on
itself: slower calls mean more handlers in flight and more contention.
HDFS takes no such lock on this path. {{org.apache.hadoop.net.NetworkTopology}}
uses the same fair {{{}netlock{}}}, but {{sortByDistance}} computes weights
with {{getWeight}} / {{{}getWeightUsingNetworkLocation{}}}, a *lock-free* walk
of parent pointers or path strings. The lock is taken only for topology changes.
h2. Observed (production, 2.2.1, 3-node OM HA, ~32 DNs, OM host with 168 cores)
* The OM leader repeatedly bursts from a 3–5 core baseline to {*}80–135 cores
for 1–5 minutes{*}, 20+ times a day. The trigger is a rise in concurrent S3
GETs (~15–30k/s), not a change in request mix.
* During a burst, {{GetKeyInfo}} takes *~6 ms per call* against ~1.5–2.5 ms at
baseline, at a _lower_ call rate.
*
{{{}om_performance_metrics_get_key_info_sort_datanodes_latency_ns_avg_time{}}}:
*0.01 ms* quiet, *4.2 ms* at the burst peak. The sort step alone is ~70% of the
call.
* JFR ({{{}profile{}}} settings), IPC handler threads, burst against the
preceding 10 minutes:
||Samples||Baseline||Burst||
|CPU samples in {{ThreadLocal$ThreadLocalMap.getEntryAfterMiss}}|9%|*68%*|
|... of which under {{ReentrantReadWriteLock.ReadLock.lock}} from
{{NetworkTopologyImpl.getDistanceCost}} ← {{sortByDistanceCost}} ←
{{KeyManagerImpl.sortDatanodes}}| |*85%*|
|... from {{NetworkTopologyImpl.getNode}} ← {{KeyManagerImpl.getOtherNode}}|
|*15%*|
Top of a burst stack:
{noformat}
java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.fullTryAcquireShared
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock
org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl.getDistanceCost
org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl.sortByDistanceCost
org.apache.hadoop.ozone.om.KeyManagerImpl.sortDatanodes
...
org.apache.hadoop.ozone.om.KeyManagerImpl.getKeyInfo
{noformat}
The only workaround is {{ozone.network.topology.aware.read=false}} on clients,
which gives up topology-aware reads.
h2. Proposed fix
# In {{{}NetworkTopologyImpl.sortByDistanceCost{}}}, compute costs with a
lock-free walk of the parent pointers, as HDFS {{NetworkTopology.getWeight}}
does. {{getDistanceCost}} stays as it is for its other callers. Alternatively,
take {{netlock}} once for the whole loop instead of once per node; that's
simpler, but still one acquisition per block.
# In {{{}KeyManagerImpl{}}}, avoid a {{getNode}} lock acquisition and
topology-script resolution per request for non-datanode clients
({{{}getOtherNode{}}}), e.g. a small cache of resolved client nodes,
invalidated on topology refresh. This also overlaps with HDDS-16776.
Both changes keep the sort result *identical* (same cost function, same
per-cost-group shuffle; nodes without a parent still score
{{{}Integer.MAX_VALUE{}}}). This is a pure performance change. The read-path
rack-ordering bug below is tracked separately; the two fixes are independent.
h2. Related correctness issue
([https://issues.apache.org/jira/browse/HDDS-17555])
Pipeline {{DatanodeDetails}} on the read path are deserialized from SCM's
protobuf. They carry {{level}} but no {{{}parent{}}}, and the read path doesn't
map them onto OM's cluster map. A unit test on 2.2.1 confirms that
{{getDistanceCost}} then returns {{Integer.MAX_VALUE}} for every node except
the client, _after_ taking the lock. Read ordering is therefore "same host
first, then random". A non-datanode client on rack r1 got its same-rack nodes
first in 76/1000 sorts (random), against 1000/1000 with linked nodes. So the
lock contention reported here buys no rack locality today.
Please see the patch link attached.
was:
h2. Problem
On every read that returns block locations ({{{}GetKeyInfo{}}},
{{{}lookupKey{}}}, {{{}getFileStatus{}}}, {{listStatus}} with locations), OM
sorts each block's datanodes by distance to the client:
{{KeyManagerImpl.sortDatanodes}} → {{NetworkTopologyImpl.sortByDistanceCost}} →
{{getDistanceCost(reader, node)}} *once per node*
{{getDistanceCost}} takes the read lock of {{{}NetworkTopologyImpl.netlock{}}},
a single global {{new ReentrantReadWriteLock(true)}}
({{{}NetworkTopologyImpl.java:65{}}}; the call is at {{:799}} in
{{{}sortByDistanceCost{}}}).
Under read-heavy load this means one lock round-trip per node per block per
request, on a single fair lock shared by every RPC handler. Reads still have to
CAS one shared state word and look up a per-thread hold count in a ThreadLocal.
With 100+ handlers on a large host, that turns into CAS retry loops and long
ThreadLocal probes, and OM CPU climbs while throughput falls. It feeds on
itself: slower calls mean more handlers in flight and more contention.
HDFS takes no such lock on this path. {{org.apache.hadoop.net.NetworkTopology}}
uses the same fair {{{}netlock{}}}, but {{sortByDistance}} computes weights
with {{getWeight}} / {{{}getWeightUsingNetworkLocation{}}}, a *lock-free* walk
of parent pointers or path strings. The lock is taken only for topology changes.
h2. Observed (production, 2.2.1, 3-node OM HA, ~32 DNs, OM host with 168 cores)
* The OM leader repeatedly bursts from a 3–5 core baseline to {*}80–135 cores
for 1–5 minutes{*}, 20+ times a day. The trigger is a rise in concurrent S3
GETs (~15–30k/s), not a change in request mix.
* During a burst, {{GetKeyInfo}} takes *~6 ms per call* against ~1.5–2.5 ms at
baseline, at a _lower_ call rate.
*
{{{}om_performance_metrics_get_key_info_sort_datanodes_latency_ns_avg_time{}}}:
*0.01 ms* quiet, *4.2 ms* at the burst peak. The sort step alone is ~70% of the
call.
* JFR ({{{}profile{}}} settings), IPC handler threads, burst against the
preceding 10 minutes:
||Samples||Baseline||Burst||
|CPU samples in {{ThreadLocal$ThreadLocalMap.getEntryAfterMiss}}|9%|*68%*|
|... of which under {{ReentrantReadWriteLock.ReadLock.lock}} from
{{NetworkTopologyImpl.getDistanceCost}} ← {{sortByDistanceCost}} ←
{{KeyManagerImpl.sortDatanodes}}| |*85%*|
|... from {{NetworkTopologyImpl.getNode}} ← {{KeyManagerImpl.getOtherNode}}|
|*15%*|
Top of a burst stack:
{noformat}
java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.fullTryAcquireShared
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock
org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl.getDistanceCost
org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl.sortByDistanceCost
org.apache.hadoop.ozone.om.KeyManagerImpl.sortDatanodes
...
org.apache.hadoop.ozone.om.KeyManagerImpl.getKeyInfo
{noformat}
The only workaround is {{ozone.network.topology.aware.read=false}} on clients,
which gives up topology-aware reads.
h2. Proposed fix
# In {{{}NetworkTopologyImpl.sortByDistanceCost{}}}, compute costs with a
lock-free walk of the parent pointers, as HDFS {{NetworkTopology.getWeight}}
does. {{getDistanceCost}} stays as it is for its other callers. Alternatively,
take {{netlock}} once for the whole loop instead of once per node; that's
simpler, but still one acquisition per block.
# In {{{}KeyManagerImpl{}}}, avoid a {{getNode}} lock acquisition and
topology-script resolution per request for non-datanode clients
({{{}getOtherNode{}}}), e.g. a small cache of resolved client nodes,
invalidated on topology refresh. This also overlaps with HDDS-16776.
Both changes keep the sort result *identical* (same cost function, same
per-cost-group shuffle; nodes without a parent still score
{{{}Integer.MAX_VALUE{}}}). This is a pure performance change. The read-path
rack-ordering bug below is tracked separately; the two fixes are independent.
h2. Related correctness issue (https://issues.apache.org/jira/browse/HDDS-17555)
Pipeline {{DatanodeDetails}} on the read path are deserialized from SCM's
protobuf. They carry {{level}} but no {{{}parent{}}}, and the read path doesn't
map them onto OM's cluster map. A unit test on 2.2.1 confirms that
{{getDistanceCost}} then returns {{Integer.MAX_VALUE}} for every node except
the client, _after_ taking the lock. Read ordering is therefore "same host
first, then random". A non-datanode client on rack r1 got its same-rack nodes
first in 76/1000 sorts (random), against 1000/1000 with linked nodes. So the
lock contention reported here buys no rack locality today.
> OM read path saturates CPU on NetworkTopologyImpl's global read lock in
> sortDatanodes
> -------------------------------------------------------------------------------------
>
> Key: HDDS-17553
> URL: https://issues.apache.org/jira/browse/HDDS-17553
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM, SCM
> Affects Versions: 2.2.1
> Reporter: Aleksei Ieshin
> Priority: Major
> Labels: pull-request-available
>
> h2. Problem
> On every read that returns block locations ({{{}GetKeyInfo{}}},
> {{{}lookupKey{}}}, {{{}getFileStatus{}}}, {{listStatus}} with locations), OM
> sorts each block's datanodes by distance to the client:
> {{KeyManagerImpl.sortDatanodes}} → {{NetworkTopologyImpl.sortByDistanceCost}}
> → {{getDistanceCost(reader, node)}} *once per node*
> {{getDistanceCost}} takes the read lock of
> {{{}NetworkTopologyImpl.netlock{}}}, a single global {{new
> ReentrantReadWriteLock(true)}} ({{{}NetworkTopologyImpl.java:65{}}}; the call
> is at {{:799}} in {{{}sortByDistanceCost{}}}).
> Under read-heavy load this means one lock round-trip per node per block per
> request, on a single fair lock shared by every RPC handler. Reads still have
> to CAS one shared state word and look up a per-thread hold count in a
> ThreadLocal. With 100+ handlers on a large host, that turns into CAS retry
> loops and long ThreadLocal probes, and OM CPU climbs while throughput falls.
> It feeds on itself: slower calls mean more handlers in flight and more
> contention.
> HDFS takes no such lock on this path.
> {{org.apache.hadoop.net.NetworkTopology}} uses the same fair {{{}netlock{}}},
> but {{sortByDistance}} computes weights with {{getWeight}} /
> {{{}getWeightUsingNetworkLocation{}}}, a *lock-free* walk of parent pointers
> or path strings. The lock is taken only for topology changes.
> h2. Observed (production, 2.2.1, 3-node OM HA, ~32 DNs, OM host with 168
> cores)
> * The OM leader repeatedly bursts from a 3–5 core baseline to {*}80–135
> cores for 1–5 minutes{*}, 20+ times a day. The trigger is a rise in
> concurrent S3 GETs (~15–30k/s), not a change in request mix.
> * During a burst, {{GetKeyInfo}} takes *~6 ms per call* against ~1.5–2.5 ms
> at baseline, at a _lower_ call rate.
> *
> {{{}om_performance_metrics_get_key_info_sort_datanodes_latency_ns_avg_time{}}}:
> *0.01 ms* quiet, *4.2 ms* at the burst peak. The sort step alone is ~70% of
> the call.
> * JFR ({{{}profile{}}} settings), IPC handler threads, burst against the
> preceding 10 minutes:
> ||Samples||Baseline||Burst||
> |CPU samples in {{ThreadLocal$ThreadLocalMap.getEntryAfterMiss}}|9%|*68%*|
> |... of which under {{ReentrantReadWriteLock.ReadLock.lock}} from
> {{NetworkTopologyImpl.getDistanceCost}} ← {{sortByDistanceCost}} ←
> {{KeyManagerImpl.sortDatanodes}}| |*85%*|
> |... from {{NetworkTopologyImpl.getNode}} ← {{KeyManagerImpl.getOtherNode}}|
> |*15%*|
> Top of a burst stack:
> {noformat}
> java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.fullTryAcquireShared
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock
> org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl.getDistanceCost
> org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl.sortByDistanceCost
> org.apache.hadoop.ozone.om.KeyManagerImpl.sortDatanodes
> ...
> org.apache.hadoop.ozone.om.KeyManagerImpl.getKeyInfo
> {noformat}
> The only workaround is {{ozone.network.topology.aware.read=false}} on
> clients, which gives up topology-aware reads.
> h2. Proposed fix
> # In {{{}NetworkTopologyImpl.sortByDistanceCost{}}}, compute costs with a
> lock-free walk of the parent pointers, as HDFS {{NetworkTopology.getWeight}}
> does. {{getDistanceCost}} stays as it is for its other callers.
> Alternatively, take {{netlock}} once for the whole loop instead of once per
> node; that's simpler, but still one acquisition per block.
> # In {{{}KeyManagerImpl{}}}, avoid a {{getNode}} lock acquisition and
> topology-script resolution per request for non-datanode clients
> ({{{}getOtherNode{}}}), e.g. a small cache of resolved client nodes,
> invalidated on topology refresh. This also overlaps with HDDS-16776.
> Both changes keep the sort result *identical* (same cost function, same
> per-cost-group shuffle; nodes without a parent still score
> {{{}Integer.MAX_VALUE{}}}). This is a pure performance change. The read-path
> rack-ordering bug below is tracked separately; the two fixes are independent.
> h2. Related correctness issue
> ([https://issues.apache.org/jira/browse/HDDS-17555])
> Pipeline {{DatanodeDetails}} on the read path are deserialized from SCM's
> protobuf. They carry {{level}} but no {{{}parent{}}}, and the read path
> doesn't map them onto OM's cluster map. A unit test on 2.2.1 confirms that
> {{getDistanceCost}} then returns {{Integer.MAX_VALUE}} for every node except
> the client, _after_ taking the lock. Read ordering is therefore "same host
> first, then random". A non-datanode client on rack r1 got its same-rack nodes
> first in 76/1000 sorts (random), against 1000/1000 with linked nodes. So the
> lock contention reported here buys no rack locality today.
> Please see the patch link attached.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
