adoroszlai opened a new pull request #1085: HDDS-1785. OOM error in Freon due to the concurrency handling URL: https://github.com/apache/hadoop/pull/1085 ## What changes were proposed in this pull request? Change concurrency in Freon `RandomKeyGenerator`: * create a worker for each thread * let each worker create volumes, buckets and keys, without limiting "inner" objects to specific "outer" ones (eg. create key for any bucket) Workers coordinate the items they create using "global" counters. https://issues.apache.org/jira/browse/HDDS-1785 ## How was this patch tested? Tested with various number of volumes/buckets/threads. ``` $ ozone freon rk --numOfVolumes 1 --numOfBuckets 100 --numOfKeys 5 --numOfThreads 1 --replicationType=RATIS --factor=THREE ... Number of Volumes created: 1 Number of Buckets created: 100 Number of Keys added: 500 Ratis replication factor: THREE Ratis replication type: RATIS Average Time spent in volume creation: 00:00:00,100 Average Time spent in bucket creation: 00:00:00,304 Average Time spent in key creation: 00:00:01,556 Average Time spent in key write: 00:00:53,509 Total bytes written: 5120000 Total Execution time: 00:01:01,537 ``` ``` $ ozone freon rk --numOfVolumes 1 --numOfBuckets 100 --numOfKeys 5 --numOfThreads 50 --replicationType=RATIS --factor=THREE ... Number of Volumes created: 1 Number of Buckets created: 100 Number of Keys added: 500 Ratis replication factor: THREE Ratis replication type: RATIS Average Time spent in volume creation: 00:00:00,003 Average Time spent in bucket creation: 00:00:00,229 Average Time spent in key creation: 00:00:00,273 Average Time spent in key write: 00:00:10,375 Total bytes written: 5120000 Total Execution time: 00:00:16,872 ``` ``` $ ozone freon rk --numOfVolumes 10 --numOfBuckets 10 --numOfKeys 500 --numOfThreads 50 --replicationType=RATIS --factor=THREE ... Number of Volumes created: 10 Number of Buckets created: 100 Number of Keys added: 50000 Ratis replication factor: THREE Ratis replication type: RATIS Average Time spent in volume creation: 00:00:00,052 Average Time spent in bucket creation: 00:00:00,240 Average Time spent in key creation: 00:00:30,742 Average Time spent in key write: 00:10:04,146 Total bytes written: 512000000 Total Execution time: 00:10:42,463 ``` ``` $ ozone freon rk --numOfVolumes 100 --numOfBuckets 100 --numOfKeys 2 --numOfThreads 50 --replicationType=RATIS --factor=THREE ... Number of Volumes created: 100 Number of Buckets created: 10000 Number of Keys added: 20000 Ratis replication factor: THREE Ratis replication type: RATIS Average Time spent in volume creation: 00:00:00,266 Average Time spent in bucket creation: 00:00:06,388 Average Time spent in key creation: 00:00:09,324 Average Time spent in key write: 00:03:44,925 Total bytes written: 204800000 Total Execution time: 00:04:11,735 ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
