[
https://issues.apache.org/jira/browse/HDDS-10586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai resolved HDDS-10586.
-------------------------------------
Fix Version/s: 1.5.0
Resolution: Fixed
> Avoid loading network topology layer schema file for every read
> ---------------------------------------------------------------
>
> Key: HDDS-10586
> URL: https://issues.apache.org/jira/browse/HDDS-10586
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: OM, SCM
> Reporter: Tanvi Penumudy
> Assignee: Tanvi Penumudy
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.5.0
>
>
> * After the change implemented in:
> [https://github.com/apache/ozone/pull/5391], the network topology layer
> schema file is now loaded for every key-level read operation, flooding the OM
> logs (please find the below logs for reference).
> * This is because a new NetworkTopologyImpl object is being instantiated for
> each read. As part of this instantiation, NodeSchemaManager is initialized
> each time resulting in the loading of the schema file each time.
> * To optimize this process, a new NetworkTopologyImpl object is now
> generated only when the fetched network topology cluster tree differs from
> the current cluster tree.
> * This object generation is now bound by the rate specified by the
> ozone.om.network.topology.refresh.duration configuration which defaults to 1
> hour.
> * This ensures that the schema file is also loaded at the same rate, thereby
> reducing the overhead during read operations.
> *Synthetic read workload:* ozone freon ockrw -n 10 --percentage-read 100
> --percentage-list 0 -v vol1 -b buck1 -p sample
> *OM logs:*
> {code:java}
> 2024-03-25 11:25:57 2024-03-25 05:55:57,738 [IPC Server handler 97 on default
> port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema
> file
> 2024-03-25 11:25:57 2024-03-25 05:55:57,739 [IPC Server handler 46 on default
> port 9862] INFO net.NodeSchemaLoader: Loading schema from
> [file:/etc/hadoop/network-topology-default.xml,
> jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> 2024-03-25 11:25:57 2024-03-25 05:55:57,739 [IPC Server handler 46 on default
> port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema
> file
> 2024-03-25 11:25:57 2024-03-25 05:55:57,740 [IPC Server handler 35 on default
> port 9862] INFO net.NodeSchemaLoader: Loading schema from
> [file:/etc/hadoop/network-topology-default.xml,
> jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> 2024-03-25 11:25:57 2024-03-25 05:55:57,740 [IPC Server handler 35 on default
> port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema
> file
> 2024-03-25 11:25:57 2024-03-25 05:55:57,741 [IPC Server handler 36 on default
> port 9862] INFO net.NodeSchemaLoader: Loading schema from
> [file:/etc/hadoop/network-topology-default.xml,
> jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> 2024-03-25 11:25:57 2024-03-25 05:55:57,741 [IPC Server handler 36 on default
> port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema
> file
> 2024-03-25 12:33:36 2024-03-25 07:03:36,016 [IPC Server handler 95 on default
> port 9862] INFO net.NodeSchemaLoader: Loading schema from
> [file:/etc/hadoop/network-topology-default.xml,
> jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> 2024-03-25 12:33:36 2024-03-25 07:03:36,016 [IPC Server handler 40 on default
> port 9862] INFO net.NodeSchemaLoader: Loading schema from
> [file:/etc/hadoop/network-topology-default.xml,
> jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> 2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 40 on default
> port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema
> file
> 2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 5 on default
> port 9862] INFO net.NodeSchemaLoader: Loading schema from
> [file:/etc/hadoop/network-topology-default.xml,
> jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> 2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 5 on default
> port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema
> file
> 2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 95 on default
> port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema
> file
> 2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 71 on default
> port 9862] INFO net.NodeSchemaLoader: Loading schema from
> [file:/etc/hadoop/network-topology-default.xml,
> jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
