[
https://issues.apache.org/jira/browse/HDDS-10586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tanvi Penumudy updated HDDS-10586:
----------------------------------
Description:
* After the change implemented in: [https://github.com/apache/ozone/pull/5391],
the network topology layer schema file is now loaded for every key-level read
operation, flooding the OM logs (please find the below logs for reference).
* This is because a new NetworkTopologyImpl object is being instantiated for
each read. As part of this instantiation, NodeSchemaManager is initialized each
time resulting in the loading of the schema file each time.
* To optimize this process, a new NetworkTopologyImpl object is now generated
only when the fetched network topology cluster tree differs from the current
cluster tree.
* This object generation is now bound by the rate specified by the
ozone.om.network.topology.refresh.duration configuration which defaults to 1
hour.
* This ensures that the schema file is also loaded at the same rate, thereby
reducing the overhead during read operations.
*Synthetic read workload:* ozone freon ockrw -n 10 --percentage-read 100
--percentage-list 0 -v vol1 -b buck1 -p sample
*OM logs:*
{code:java}
2024-03-25 11:25:57 2024-03-25 05:55:57,738 [IPC Server handler 97 on default
port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 11:25:57 2024-03-25 05:55:57,739 [IPC Server handler 46 on default
port 9862] INFO net.NodeSchemaLoader: Loading schema from
[file:/etc/hadoop/network-topology-default.xml,
jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
2024-03-25 11:25:57 2024-03-25 05:55:57,739 [IPC Server handler 46 on default
port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 11:25:57 2024-03-25 05:55:57,740 [IPC Server handler 35 on default
port 9862] INFO net.NodeSchemaLoader: Loading schema from
[file:/etc/hadoop/network-topology-default.xml,
jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
2024-03-25 11:25:57 2024-03-25 05:55:57,740 [IPC Server handler 35 on default
port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 11:25:57 2024-03-25 05:55:57,741 [IPC Server handler 36 on default
port 9862] INFO net.NodeSchemaLoader: Loading schema from
[file:/etc/hadoop/network-topology-default.xml,
jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
2024-03-25 11:25:57 2024-03-25 05:55:57,741 [IPC Server handler 36 on default
port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 12:33:36 2024-03-25 07:03:36,016 [IPC Server handler 95 on default
port 9862] INFO net.NodeSchemaLoader: Loading schema from
[file:/etc/hadoop/network-topology-default.xml,
jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
2024-03-25 12:33:36 2024-03-25 07:03:36,016 [IPC Server handler 40 on default
port 9862] INFO net.NodeSchemaLoader: Loading schema from
[file:/etc/hadoop/network-topology-default.xml,
jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 40 on default
port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 5 on default
port 9862] INFO net.NodeSchemaLoader: Loading schema from
[file:/etc/hadoop/network-topology-default.xml,
jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 5 on default
port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 95 on default
port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file
2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 71 on default
port 9862] INFO net.NodeSchemaLoader: Loading schema from
[file:/etc/hadoop/network-topology-default.xml,
jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
{code}
was:
* After the change implemented in: [https://github.com/apache/ozone/pull/5391],
the network topology layer schema file is now loaded for every key-level read
operation.
* This is because a new NetworkTopologyImpl object is being instantiated for
each read. As part of this instantiation, NodeSchemaManager is initialized each
time resulting in the loading of the schema file each time.
* To optimize this process, a new NetworkTopologyImpl object is now generated
only when the fetched network topology cluster tree differs from the current
cluster tree.
* This object generation is now bound by the rate specified by the
ozone.om.network.topology.refresh.duration configuration which defaults to 1
hour.
* This ensures that the schema file is also loaded at the same rate, thereby
reducing the overhead during read operations.
{code:java}
2024-03-21 23:17:30,064 INFO [IPC Server handler 34 on
9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading network topology
layer schema file
2024-03-21 23:17:30,111 INFO [IPC Server handler 48 on
9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading schema from
[jar:file:/path/to/hadoop-ozone/share/ozone/lib/hdds-common-1.4.0.1.5.0-b4.jar!/network-topology-default.xml]
2024-03-21 23:17:30,111 INFO [IPC Server handler 48 on
9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading network topology
layer schema file
2024-03-21 23:17:30,112 INFO [IPC Server handler 48 on
9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading schema from
[jar:file:/path/to/hadoop-ozone/share/ozone/lib/hdds-common-1.4.0.1.5.0-b4.jar!/network-topology-default.xml]
2024-03-21 23:17:30,112 INFO [IPC Server handler 48 on
9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading network topology
layer schema file
2024-03-21 23:17:30,135 INFO [IPC Server handler 60 on
9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading schema from
[jar:file:/path/to/hadoop-ozone/share/ozone/lib/hdds-common-1.4.0.1.5.0-b4.jar!/network-topology-default.xml]
2024-03-21 23:17:30,135 INFO [IPC Server handler 60 on
9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading network topology
layer schema file
2024-03-21 23:17:30,136 INFO [IPC Server handler 60 on
9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading schema from
[jar:file:/path/to/hadoop-ozone/share/ozone/lib/hdds-common-1.4.0.1.5.0-b4.jar!/network-topology-default.xml]
2024-03-21 23:17:30,136 INFO [IPC Server handler 60 on
9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading network topology
layer schema file
2024-03-21 23:17:32,716 INFO [IPC Server handler 36 on
9862]-org.apache.hadoop.hdds.scm.net.NodeSchemaLoader: Loading schema from
[jar:file:/path/to/hadoop-ozone/share/ozone/lib/hdds-common-1.4.0.1.5.0-b4.jar!/network-topology-default.xml]
{code}
> Avoid loading network topology layer schema file for every read
> ---------------------------------------------------------------
>
> Key: HDDS-10586
> URL: https://issues.apache.org/jira/browse/HDDS-10586
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: OM, SCM
> Reporter: Tanvi Penumudy
> Assignee: Tanvi Penumudy
> Priority: Major
>
> * After the change implemented in:
> [https://github.com/apache/ozone/pull/5391], the network topology layer
> schema file is now loaded for every key-level read operation, flooding the OM
> logs (please find the below logs for reference).
> * This is because a new NetworkTopologyImpl object is being instantiated for
> each read. As part of this instantiation, NodeSchemaManager is initialized
> each time resulting in the loading of the schema file each time.
> * To optimize this process, a new NetworkTopologyImpl object is now
> generated only when the fetched network topology cluster tree differs from
> the current cluster tree.
> * This object generation is now bound by the rate specified by the
> ozone.om.network.topology.refresh.duration configuration which defaults to 1
> hour.
> * This ensures that the schema file is also loaded at the same rate, thereby
> reducing the overhead during read operations.
> *Synthetic read workload:* ozone freon ockrw -n 10 --percentage-read 100
> --percentage-list 0 -v vol1 -b buck1 -p sample
> *OM logs:*
> {code:java}
> 2024-03-25 11:25:57 2024-03-25 05:55:57,738 [IPC Server handler 97 on default
> port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema
> file
> 2024-03-25 11:25:57 2024-03-25 05:55:57,739 [IPC Server handler 46 on default
> port 9862] INFO net.NodeSchemaLoader: Loading schema from
> [file:/etc/hadoop/network-topology-default.xml,
> jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> 2024-03-25 11:25:57 2024-03-25 05:55:57,739 [IPC Server handler 46 on default
> port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema
> file
> 2024-03-25 11:25:57 2024-03-25 05:55:57,740 [IPC Server handler 35 on default
> port 9862] INFO net.NodeSchemaLoader: Loading schema from
> [file:/etc/hadoop/network-topology-default.xml,
> jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> 2024-03-25 11:25:57 2024-03-25 05:55:57,740 [IPC Server handler 35 on default
> port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema
> file
> 2024-03-25 11:25:57 2024-03-25 05:55:57,741 [IPC Server handler 36 on default
> port 9862] INFO net.NodeSchemaLoader: Loading schema from
> [file:/etc/hadoop/network-topology-default.xml,
> jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> 2024-03-25 11:25:57 2024-03-25 05:55:57,741 [IPC Server handler 36 on default
> port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema
> file
> 2024-03-25 12:33:36 2024-03-25 07:03:36,016 [IPC Server handler 95 on default
> port 9862] INFO net.NodeSchemaLoader: Loading schema from
> [file:/etc/hadoop/network-topology-default.xml,
> jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> 2024-03-25 12:33:36 2024-03-25 07:03:36,016 [IPC Server handler 40 on default
> port 9862] INFO net.NodeSchemaLoader: Loading schema from
> [file:/etc/hadoop/network-topology-default.xml,
> jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> 2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 40 on default
> port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema
> file
> 2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 5 on default
> port 9862] INFO net.NodeSchemaLoader: Loading schema from
> [file:/etc/hadoop/network-topology-default.xml,
> jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> 2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 5 on default
> port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema
> file
> 2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 95 on default
> port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema
> file
> 2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 71 on default
> port 9862] INFO net.NodeSchemaLoader: Loading schema from
> [file:/etc/hadoop/network-topology-default.xml,
> jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml]
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
