[
https://issues.apache.org/jira/browse/HDDS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16862046#comment-16862046
]
Xudong Cao edited comment on HDDS-1532 at 6/12/19 12:34 PM:
------------------------------------------------------------
Test performance comparison before and after this jira：
1. Test Environment
Test under a 3 nodes ozone cluster, each node is equiped as below:
||Hardware ||Configuration||
|CPU, Memory|16 cores Xeon 2.10GHZ + 64GB RAM|
|Disk|3.6TB HDD * 11pcs|
|Network|1000M Network Card|
h3. 2. Test result before jira
*freon command (since there is only 1 volume, so only 1 thread will be
created):*
bin/ozone freon randomkeys --numOfVolumes=1 --numOfBuckets 10 --numOfKeys 1000
--keySize 1048576 --replicationType=RATIS --factor=THREE
*freon result:*
Number of Volumes created: 1
Number of Buckets created: 10
Number of Keys added: 10000
Ratis replication factor: THREE
Ratis replication type: RATIS
Average Time spent in volume creation: 00:00:00,040
Average Time spent in bucket creation: 00:00:00,020
Average Time spent in key creation: 00:00:14,493
Average Time spent in key write: 00:23:14,749
Total bytes written: 10485760000
Total Execution time: 00:23:31,540
h3. 3. Test result after jira
*freon command (note we have set --numOfThreads 50, so although there is only 1
volume, we create 50 threads):*
ozone freon randomkeys --numOfVolumes=1 --numOfBuckets 10 --numOfKeys 1000
--numOfThreads 50 --keySize 1048576 --replicationType=RATIS --factor=THREE
*freon result:*
Number of Volumes created: 1
Number of Buckets created: 10
Number of Keys added: 10000
Ratis replication factor: THREE
Ratis replication type: RATIS
Average Time spent in volume creation: 00:00:00,000
Average Time spent in bucket creation: 00:00:00,002
Average Time spent in key creation: 00:00:02,430
Average Time spent in key write: 00:03:14,370
Total bytes written: 10485760000
Total Execution time: 00:03:21,064
h3. 4. Conclusion
In this environment, writing the same amount of data, by improving the
concurrency framework, test performance is 8 times faster.
was (Author: xudongcao):
Test performance comparison before and after jira：
1. Test Environment
Test under a 3 nodes ozone cluster, each node is equiped as below:
||Hardware ||Configuration||
|CPU, Memory|16 cores Xeon 2.10GHZ + 64GB RAM|
|Disk|3.6TB HDD * 11pcs|
|Network|1000M Network Card|
h3. 2. Test result before jira
*freon command (since there is only 1 volume, so only 1 thread will be
created):*
bin/ozone freon randomkeys --numOfVolumes=1 --numOfBuckets 10 --numOfKeys 1000
--keySize 1048576 --replicationType=RATIS --factor=THREE
*freon result:*
Number of Volumes created: 1
Number of Buckets created: 10
Number of Keys added: 10000
Ratis replication factor: THREE
Ratis replication type: RATIS
Average Time spent in volume creation: 00:00:00,040
Average Time spent in bucket creation: 00:00:00,020
Average Time spent in key creation: 00:00:14,493
Average Time spent in key write: 00:23:14,749
Total bytes written: 10485760000
Total Execution time: 00:23:31,540
h3. 3. Test result after jira
*freon command (note we have set --numOfThreads 50, so although there is only 1
volume, we create 50 threads):*
ozone freon randomkeys --numOfVolumes=1 --numOfBuckets 10 --numOfKeys 1000
--numOfThreads 50 --keySize 1048576 --replicationType=RATIS --factor=THREE
*freon result:*
Number of Volumes created: 1
Number of Buckets created: 10
Number of Keys added: 10000
Ratis replication factor: THREE
Ratis replication type: RATIS
Average Time spent in volume creation: 00:00:00,000
Average Time spent in bucket creation: 00:00:00,002
Average Time spent in key creation: 00:00:02,430
Average Time spent in key write: 00:03:14,370
Total bytes written: 10485760000
Total Execution time: 00:03:21,064
h3. 4. Conclusion
In this environment, writing the same amount of data, by improving the
concurrency framework, test performance is 8 times faster.
> Freon: Improve the concurrency testing framework.
> -------------------------------------------------
>
> Key: HDDS-1532
> URL: https://issues.apache.org/jira/browse/HDDS-1532
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
> Reporter: Xudong Cao
> Assignee: Xudong Cao
> Priority: Major
>
> Currently, Freon's concurrency framework is just on volume-level, but in
> actual testing, users are likely to provide a smaller volume number（typically
> 1）, and a larger bucket number and key number, in which case the existing
> concurrency framework can not make good use of the thread pool.
> We need to improve the concurrency policy, make the volume creation task,
> bucket creation task, and key creation task all can be equally submitted to
> the thread pool as a general task.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
