[
https://issues.apache.org/jira/browse/HDDS-9272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duong updated HDDS-9272:
------------------------
Description:
h2. Problem
After HDDS-8300, get key metadata API in OM sorts datanodes to allow clients to
optimize data I/O based on locality. However, today this requires OM to make
additional calls to SCM to sort the datanodes. These calls contribute more than
80% of the getKeyInfo latency and reduce the peak pure read OPPS from *100K* to
{*}25K{*}.
!Screenshot 2023-09-12 at 9.52.33 AM.png|width=754,height=557!
h2. Optimization
Today, the get key metadata API makes 2 separate calls to SCM, one to get
container pipelines and the second to sort datanodes in the container pipelines
(see
[KeyManagerImpl#getKeyInfo|https://github.com/apache/ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java#L1946-L1959]):
{code:java}
if (!args.isHeadOp()) {
...
// get container pipeline info from cache.
captureLatencyNs(metrics.getGetKeyInfoRefreshLocationLatencyNs(),
() -> refreshPipelineFromCache(value,
args.isForceUpdateContainerCacheFromSCM()));
if (args.getSortDatanodes()) {
sortDatanodes(clientAddress, value);
}
{code}
The two calls can be combined into one and cached in the existing
containerLocationCache in ScmClient.
Some details of the implementation:
# A new SCM API, called `Pipeline getContainerPipeline(long containerId,
boolean sortDatanodes)` is added to `SCMClientProtocolServer`. When the
second argument is true, the datanodes in the result pipeline are sorted by the
logic in `SCMBlockProtocolServer#sortDatanodes`.
was:
h2. Problem
After HDDS-8300, get key metadata API in OM sorts datanodes to allow clients to
optimize data I/O based on locality. However, today this requires OM to make
additional calls to SCM to sort the datanodes. These calls contribute more than
80% of the getKeyInfo latency and reduce the peak pure read OPPS from *100K* to
{*}25K{*}.
!Screenshot 2023-09-12 at 9.52.33 AM.png|width=754,height=557!
h2. Optimization
Today, the get key metadata API makes 2 separate calls to SCM, one to get
container pipelines and the second to sort datanodes in the container pipelines
(see
[KeyManagerImpl#getKeyInfo|https://github.com/apache/ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java#L1946-L1959]):
{code:java}
// If operation is head, do not perform any additional steps based on flags.
// As head operation does not need any of those details.
if (!args.isHeadOp()) {
// add block token for read.
captureLatencyNs(metrics.getGetKeyInfoGenerateBlockTokenLatencyNs(),
() -> addBlockToken4Read(value));
// get container pipeline info from cache.
captureLatencyNs(metrics.getGetKeyInfoRefreshLocationLatencyNs(),
() -> refreshPipelineFromCache(value,
args.isForceUpdateContainerCacheFromSCM()));
if (args.getSortDatanodes()) {
sortDatanodes(clientAddress, value);
}
{code}
> Performance improvement for OM's sort datanodes
> -----------------------------------------------
>
> Key: HDDS-9272
> URL: https://issues.apache.org/jira/browse/HDDS-9272
> Project: Apache Ozone
> Issue Type: Improvement
> Affects Versions: 1.4.0
> Reporter: Duong
> Assignee: Tanvi Penumudy
> Priority: Critical
> Labels: ozone-performance
> Attachments: Screenshot 2023-09-12 at 9.52.33 AM.png,
> sortDatanode_flameGraph.html
>
>
> h2. Problem
> After HDDS-8300, get key metadata API in OM sorts datanodes to allow clients
> to optimize data I/O based on locality. However, today this requires OM to
> make additional calls to SCM to sort the datanodes. These calls contribute
> more than 80% of the getKeyInfo latency and reduce the peak pure read OPPS
> from *100K* to {*}25K{*}.
> !Screenshot 2023-09-12 at 9.52.33 AM.png|width=754,height=557!
> h2. Optimization
> Today, the get key metadata API makes 2 separate calls to SCM, one to get
> container pipelines and the second to sort datanodes in the container
> pipelines (see
> [KeyManagerImpl#getKeyInfo|https://github.com/apache/ozone/blob/master/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java#L1946-L1959]):
>
> {code:java}
> if (!args.isHeadOp()) {
> ...
> // get container pipeline info from cache.
> captureLatencyNs(metrics.getGetKeyInfoRefreshLocationLatencyNs(),
> () -> refreshPipelineFromCache(value,
> args.isForceUpdateContainerCacheFromSCM()));
> if (args.getSortDatanodes()) {
> sortDatanodes(clientAddress, value);
> }
> {code}
> The two calls can be combined into one and cached in the existing
> containerLocationCache in ScmClient.
> Some details of the implementation:
> # A new SCM API, called `Pipeline getContainerPipeline(long containerId,
> boolean sortDatanodes)` is added to `SCMClientProtocolServer`. When the
> second argument is true, the datanodes in the result pipeline are sorted by
> the logic in `SCMBlockProtocolServer#sortDatanodes`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]