Tanvi Penumudy created HDDS-10586:
-------------------------------------
Summary: Avoid loading network topology layer schema file for
every read
Key: HDDS-10586
URL: https://issues.apache.org/jira/browse/HDDS-10586
Project: Apache Ozone
Issue Type: Sub-task
Components: OM, SCM
Reporter: Tanvi Penumudy
Assignee: Tanvi Penumudy
* After the change implemented in: [https://github.com/apache/ozone/pull/5391],
the network topology layer schema file is now loaded for every key-level read
operation.
* This is because a new NetworkTopologyImpl object is being instantiated for
each read. As part of this instantiation, NodeSchemaManager is initialized each
time resulting in the loading of the schema file each time.
* To optimize this process, a new NetworkTopologyImpl object is now generated
only when the fetched network topology cluster tree differs from the current
cluster tree.
* This object generation is now bound by the rate specified by the
ozone.om.network.topology.refresh.duration configuration which defaults to 1
hour.
* This ensures that the schema file is also loaded at the same rate, thereby
reducing the overhead during read operations.
{code:java}
2024-03-21 23:17:30,064 INFO [IPC Server handler 34 on
9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading network topology
layer schema file
2024-03-21 23:17:30,111 INFO [IPC Server handler 48 on
9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading schema from
[jar:file:/path/to/hadoop-ozone/share/ozone/lib/hdds-common-1.4.0.1.5.0-b4.jar!/network-topology-default.xml]
2024-03-21 23:17:30,111 INFO [IPC Server handler 48 on
9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading network topology
layer schema file
2024-03-21 23:17:30,112 INFO [IPC Server handler 48 on
9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading schema from
[jar:file:/path/to/hadoop-ozone/share/ozone/lib/hdds-common-1.4.0.1.5.0-b4.jar!/network-topology-default.xml]
2024-03-21 23:17:30,112 INFO [IPC Server handler 48 on
9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading network topology
layer schema file
2024-03-21 23:17:30,135 INFO [IPC Server handler 60 on
9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading schema from
[jar:file:/path/to/hadoop-ozone/share/ozone/lib/hdds-common-1.4.0.1.5.0-b4.jar!/network-topology-default.xml]
2024-03-21 23:17:30,135 INFO [IPC Server handler 60 on
9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading network topology
layer schema file
2024-03-21 23:17:30,136 INFO [IPC Server handler 60 on
9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading schema from
[jar:file:/path/to/hadoop-ozone/share/ozone/lib/hdds-common-1.4.0.1.5.0-b4.jar!/network-topology-default.xml]
2024-03-21 23:17:30,136 INFO [IPC Server handler 60 on
9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading network topology
layer schema file
2024-03-21 23:17:32,716 INFO [IPC Server handler 36 on
9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading schema from
[jar:file:/path/to/hadoop-ozone/share/ozone/lib/hdds-common-1.4.0.1.5.0-b4.jar!/network-topology-default.xml]
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
