tanvipenumudy opened a new pull request, #6434: URL: https://github.com/apache/ozone/pull/6434
## What changes were proposed in this pull request? - After the change implemented in: [#5391](https://github.com/apache/ozone/pull/5391), the network topology layer schema file is now loaded for every key-level read operation, flooding the OM logs (please find the below logs for reference). - This is because a new `NetworkTopologyImpl` object is being instantiated for each read. As part of this instantiation, `NodeSchemaManager` is initialized each time resulting in the loading of the schema file each time. - To optimize this process, a new `NetworkTopologyImpl` object is now generated only when the fetched network topology cluster tree differs from the current cluster tree. - This object generation is now bound by the rate specified by the `ozone.om.network.topology.refresh.duration` configuration which defaults to 1 hour. - This ensures that the schema file is also loaded at the same rate, thereby reducing the overhead during read operations. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-10586 ## How was this patch tested? Tested with a synthetic read workload on a docker-compose cluster (using freon `ockrw` tool): ``` ozone freon ockrw -n 10 --percentage-read 100 --percentage-list 0 -v vol1 -b buck1 -p sample ``` ozone-om logs: ``` ... 2024-03-25 11:25:57 2024-03-25 05:55:57,738 [IPC Server handler 97 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file 2024-03-25 11:25:57 2024-03-25 05:55:57,739 [IPC Server handler 46 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml] 2024-03-25 11:25:57 2024-03-25 05:55:57,739 [IPC Server handler 46 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file 2024-03-25 11:25:57 2024-03-25 05:55:57,740 [IPC Server handler 35 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml] 2024-03-25 11:25:57 2024-03-25 05:55:57,740 [IPC Server handler 35 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file 2024-03-25 11:25:57 2024-03-25 05:55:57,741 [IPC Server handler 36 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml] 2024-03-25 11:25:57 2024-03-25 05:55:57,741 [IPC Server handler 36 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file 2024-03-25 12:33:36 2024-03-25 07:03:36,016 [IPC Server handler 95 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml] 2024-03-25 12:33:36 2024-03-25 07:03:36,016 [IPC Server handler 40 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml] 2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 40 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file 2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 5 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml] 2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 5 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file 2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 95 on default port 9862] INFO net.NodeSchemaLoader: Loading network topology layer schema file 2024-03-25 12:33:36 2024-03-25 07:03:36,017 [IPC Server handler 71 on default port 9862] INFO net.NodeSchemaLoader: Loading schema from [file:/etc/hadoop/network-topology-default.xml, jar:file:/opt/hadoop/share/ozone/lib/hdds-common-1.5.0-SNAPSHOT.jar!/network-topology-default.xml] ... ``` Before changes: ``` docker-compose logs | grep 'Loading network topology layer schema file' | grep 'ozone-om-1' | wc -l 20 ``` After changes: ``` docker-compose logs | grep 'Loading network topology layer schema file' | grep 'ozone-om-1' | wc -l 1 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
