devmadhuu commented on PR #7600:
URL: https://github.com/apache/ozone/pull/7600#issuecomment-2556670546
> Thanks for adding this @devmadhuu. Could you share the benchmarks that
were used to arrive at these numbers with the community?
yes sure, thanks @errose28 . We did following performance benchmarking
testing for Recon OM sync process flow.
**Workload test ran for 5K TPS (create/commit operations) on NvMe Cisco
cluster:**
`ozone freon ockrw -n 10000000 -t 100 --percentage-read 0 --size 0 -r
1000000 -v voltest -b buckettest -p performanceTest`
**Following configs:**
Recon heap allocation - 31 GB
ozone.recon.om.snapshot.task.interval.delay - 5s
recon.om.delta.update.limit - 50k
recon.om.delta.update.loop.limit - 50
Test ran for 39 mins and approx 10M OM DB events got generated having a mix
of following events due to 5K create/commit key operations per sec:
create/commit
```
-
- insert in open key (PUT)
- update bucket info (UPDATE)
- delete from open key (DELETE)
- update bucket info (UPDATE)
- insert to key (PUT)
- insert to delete key (PUT)
- removal from deleted key table (DELETE)
```
Further observations:
- Approx 2.1M OM DB events per min got generated by workload till the
whole test run duration.
- No JVM pause detection and GC pauses.
- Recon OM data was lagging by approx 330k OM DB events in one sync
interval and it was near real time sync while test workload was in progress.
- Recon OM sync is divided among following sub tasks:
-
- Get from OM
- Perform DB update in batch
- Prepare events based on DB update in batch.- These 3 tasks together
took 16 secs
- Process those DB events by each of the 4 background task
concurrently.- 30 secs
So based on above perf stats, Recon was actually processing end to end
1.4M per min and OM was generating at a pace of 2.1M per min. This data also
confirmed by Grafana metrics. Based on this data, if we increase delta update
limit further, it will not help much because processing time will increase and
after each run, there is an delay of 1 min. so we need to reduce the delay
further to 5s, so that lag between Recon and OM is kept to min in the range of
just 330k (1 sync cycle will match up this as well after test workload
finishes).
Our next task would be to think, how we can optimize the processing speed of
background tasks, though there is limited possibility due to the nature of data
and Recon's background tasks must process all the events in sequence and cannot
process concurrently, we need to think and see the optimization possibility or
possibility of processing concurrently in processing logic of single event by
each background task.
Raised [HDDS-11688](https://issues.apache.org/jira/browse/HDDS-11688) and
[HDDS-11953](https://issues.apache.org/jira/browse/HDDS-11953) to handle
further optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
