guihecheng opened a new pull request #2869: URL: https://github.com/apache/ozone/pull/2869
## What changes were proposed in this pull request? Multi-threaded validator for freon RandomKeyGenerator. The thread pool size defaults to 1, so it does not change the default behavior, but we could have multiple validator threads if needed. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-6046 ## How was this patch tested? manual test We could see that multi-threaded version saves time in validation and do little harm to writes when we have enough system resources. Single-threaded: ./bin/ozone freon rk --keySize=20971520 --numOfBuckets=1 --numOfVolumes=1 --numOfKeys=10000 --replicationType=RATIS --factor=THREE --numOfThreads=20 --validateWrites *************************************************** Status: Failed Git Base Revision: a3b9c37a397ad4188041dd80621bdeefc46885f2 Number of Volumes created: 1 Number of Buckets created: 1 Number of Keys added: 9995 Ratis replication factor: THREE Ratis replication type: RATIS Average Time spent in volume creation: 00:00:00,001 Average Time spent in bucket creation: 00:00:00,000 Average Time spent in key creation: 00:00:09,233 Average Time spent in key write: 00:12:28,440 <-- the same time spent in write Total bytes written: 209610342400 Total number of writes validated: 9995 Writes validated: 100.0 % Successful validation: 9995 Unsuccessful validation: 0 Total Execution time: 00:33:43,738 <-- longer time with single thread *************************************************** Multi-threaded(20): ./bin/ozone freon rk --keySize=20971520 --numOfBuckets=1 --numOfVolumes=1 --numOfKeys=10000 --replicationType=RATIS --factor=THREE --numOfThreads=20 --validateWrites --numOfValidateThreads=20 *************************************************** Status: Success Git Base Revision: a3b9c37a397ad4188041dd80621bdeefc46885f2 Number of Volumes created: 1 Number of Buckets created: 1 Number of Keys added: 10000 Ratis replication factor: THREE Ratis replication type: RATIS Average Time spent in volume creation: 00:00:00,005 Average Time spent in bucket creation: 00:00:00,000 Average Time spent in key creation: 00:00:09,485 Average Time spent in key write: 00:12:26,249 <-- the same time spent in write Total bytes written: 209715200000 Total number of writes validated: 10000 Writes validated: 100.0 % Successful validation: 10000 Unsuccessful validation: 0 Total Execution time: 00:12:58,046 <-- shorter time with multiple threads *************************************************** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
