[
https://issues.apache.org/jira/browse/RATIS-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298504#comment-17298504
]
runzhiwang edited comment on RATIS-1312 at 3/10/21, 6:56 AM:
-------------------------------------------------------------
[~szetszwo] Hi, I have follow your suggestions:
1. FileStore max is 1000
https://github.com/runzhiwang/incubator-ratis/commit/f98e0ac3265a8e86d07c9df13dfd827d4d45be50
2. print out also the HDFS pipeline setup time to see if it makes big
difference.
https://github.com/runzhiwang/HdfsPerformance/commit/e8067bf0c5e5c73bda5b127a043545ea4d17e1ae
3. We may change the code to use one RaftClient per file
https://github.com/runzhiwang/incubator-ratis/commit/cfc4ab07645963e5affd5be3fe962218d2e9e4c0
4. We may try using different primary server in FileStore.
https://github.com/runzhiwang/incubator-ratis/commit/cfc4ab07645963e5affd5be3fe962218d2e9e4c0#diff-2f55b87ccdd966b0a516cd6e736fa646c29513dfbd8a075b31a0a06f496a722dR136
increase client number can improve the performance, but only improve 16%, hdfs
cost 24 seconds, streaming with one client cost 50 seconds, streaming with 20
clients cost 43 seconds. And if there are about 100 clients, there is
OutOfDirectMemoryError.
Besides, when I write 300 files, without sync, there is no performance
difference between 20 clients and 40 clients.
!image-2021-03-10-11-23-42-092.png!
was (Author: yjxxtd):
[~szetszwo] Hi, I have follow your suggestions:
1. FileStore max is 1000
https://github.com/runzhiwang/incubator-ratis/commit/f98e0ac3265a8e86d07c9df13dfd827d4d45be50
2. print out also the HDFS pipeline setup time to see if it makes big
difference.
https://github.com/runzhiwang/HdfsPerformance/commit/e8067bf0c5e5c73bda5b127a043545ea4d17e1ae
3. We may change the code to use one RaftClient per file
https://github.com/runzhiwang/incubator-ratis/commit/cfc4ab07645963e5affd5be3fe962218d2e9e4c0
4. We may try using different primary server in FileStore.
https://github.com/runzhiwang/incubator-ratis/commit/cfc4ab07645963e5affd5be3fe962218d2e9e4c0#diff-2f55b87ccdd966b0a516cd6e736fa646c29513dfbd8a075b31a0a06f496a722dR136
increase client number can improve the performance, but only improve 16%, and
if there are about 100 clients, there is OutOfDirectMemoryError.
Besides, when I write 300 files, without sync, there is no performance
difference between 20 clients and 40 clients.
!image-2021-03-10-11-23-42-092.png!
> Compare the performance between HDFS and DataStreamApi
> ------------------------------------------------------
>
> Key: RATIS-1312
> URL: https://issues.apache.org/jira/browse/RATIS-1312
> Project: Ratis
> Issue Type: Sub-task
> Reporter: runzhiwang
> Priority: Major
> Attachments: hdfs.svg, image-2021-03-10-11-23-19-681.png,
> image-2021-03-10-11-23-42-092.png, screenshot-1.png, screenshot-2.png,
> streaming.svg
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
