[ 
https://issues.apache.org/jira/browse/HDDS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16862046#comment-16862046
 ] 

Xudong Cao edited comment on HDDS-1532 at 6/12/19 12:34 PM:
------------------------------------------------------------

Test performance comparison before and after this jira:

1. Test Environment

 Test under a 3 nodes ozone cluster, each node is equiped as below:
||Hardware ||Configuration||
|CPU, Memory|16 cores Xeon 2.10GHZ + 64GB RAM|
|Disk|3.6TB HDD * 11pcs|
|Network|1000M Network Card|

 
h3. 2. Test result before jira

*freon command (since there is only 1 volume, so only 1 thread will be 
created):*

bin/ozone freon randomkeys --numOfVolumes=1 --numOfBuckets 10 --numOfKeys 1000 
--keySize 1048576 --replicationType=RATIS --factor=THREE

*freon result:*

Number of Volumes created: 1
 Number of Buckets created: 10
 Number of Keys added: 10000
 Ratis replication factor: THREE
 Ratis replication type: RATIS
 Average Time spent in volume creation: 00:00:00,040
 Average Time spent in bucket creation: 00:00:00,020
 Average Time spent in key creation: 00:00:14,493
 Average Time spent in key write: 00:23:14,749
 Total bytes written: 10485760000
 Total Execution time: 00:23:31,540

 
h3. 3. Test result after jira

*freon command (note we have set --numOfThreads 50, so although there is only 1 
volume, we create 50 threads):*

ozone freon randomkeys --numOfVolumes=1 --numOfBuckets 10 --numOfKeys 1000 
--numOfThreads 50 --keySize 1048576 --replicationType=RATIS --factor=THREE

*freon result:*

Number of Volumes created: 1
 Number of Buckets created: 10
 Number of Keys added: 10000
 Ratis replication factor: THREE
 Ratis replication type: RATIS
 Average Time spent in volume creation: 00:00:00,000
 Average Time spent in bucket creation: 00:00:00,002
 Average Time spent in key creation: 00:00:02,430
 Average Time spent in key write: 00:03:14,370
 Total bytes written: 10485760000
 Total Execution time: 00:03:21,064

 
h3. 4. Conclusion

In this environment, writing the same amount of data, by improving the 
concurrency framework, test performance is 8 times faster.

 


was (Author: xudongcao):
Test performance comparison before and after jira:

1. Test Environment

 Test under a 3 nodes ozone cluster, each node is equiped as below:
||Hardware ||Configuration||
|CPU, Memory|16 cores Xeon 2.10GHZ + 64GB RAM|
|Disk|3.6TB HDD * 11pcs|
|Network|1000M Network Card|

 
h3. 2. Test result before jira

*freon command (since there is only 1 volume, so only 1 thread will be 
created):*

bin/ozone freon randomkeys --numOfVolumes=1 --numOfBuckets 10 --numOfKeys 1000 
--keySize 1048576 --replicationType=RATIS --factor=THREE

*freon result:*

Number of Volumes created: 1
Number of Buckets created: 10
Number of Keys added: 10000
Ratis replication factor: THREE
Ratis replication type: RATIS
Average Time spent in volume creation: 00:00:00,040
Average Time spent in bucket creation: 00:00:00,020
Average Time spent in key creation: 00:00:14,493
Average Time spent in key write: 00:23:14,749
Total bytes written: 10485760000
Total Execution time: 00:23:31,540

 
h3. 3. Test result after jira

*freon command (note we have set --numOfThreads 50, so although there is only 1 
volume, we create 50 threads):*

ozone freon randomkeys --numOfVolumes=1 --numOfBuckets 10 --numOfKeys 1000 
--numOfThreads 50 --keySize 1048576 --replicationType=RATIS --factor=THREE

*freon result:*

Number of Volumes created: 1
Number of Buckets created: 10
Number of Keys added: 10000
Ratis replication factor: THREE
Ratis replication type: RATIS
Average Time spent in volume creation: 00:00:00,000
Average Time spent in bucket creation: 00:00:00,002
Average Time spent in key creation: 00:00:02,430
Average Time spent in key write: 00:03:14,370
Total bytes written: 10485760000
Total Execution time: 00:03:21,064

 
h3. 4. Conclusion

In this environment, writing the same amount of data, by improving the 
concurrency framework, test performance is 8 times faster.

 

> Freon: Improve the concurrency testing framework.
> -------------------------------------------------
>
>                 Key: HDDS-1532
>                 URL: https://issues.apache.org/jira/browse/HDDS-1532
>             Project: Hadoop Distributed Data Store
>          Issue Type: Bug
>          Components: test
>            Reporter: Xudong Cao
>            Assignee: Xudong Cao
>            Priority: Major
>
> Currently, Freon's concurrency framework is just on volume-level, but in 
> actual testing, users are likely to provide a smaller volume number(typically 
> 1), and a larger bucket number and key number, in which case the existing 
> concurrency framework can not make good use of the thread pool.
> We need to improve the concurrency policy, make the volume creation task, 
> bucket creation task, and key creation task all can be equally submitted to 
> the thread pool as a general task. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Reply via email to