[
https://issues.apache.org/jira/browse/RATIS-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17297901#comment-17297901
]
Tsz-wo Sze commented on RATIS-1312:
-----------------------------------
[~yjxxtd] thanks a lot for the update. Some ideas below.
- In the new code, it has maxThreadNum = 1000 (FileStore max is 100).
[https://github.com/runzhiwang/HdfsPerformance/blob/9d3093d2a708aab4a5ca1e04fb7068ad8ce31988/src/main/java/org/apache/HdfsPerformance.java#L34]
- BTW, the HDSF output streams are already created before start. The pipelines
setup time is not counted in HDFS. We do count the setup time in FileStore. I
suggest to print out also the HDFS pipeline setup time to see if it makes big
difference.
{code}
List<FSDataOutputStream> outs = new ArrayList<>();
for (int i = 0; i < paths.size(); i ++) {
Path dstPath = new Path(paths.get(i)); //目标路径
//打开一个输出流
FSDataOutputStream outputStream = fs.create(dstPath);
outs.add(outputStream);
}
System.out.println("Start write now");
long start = System.currentTimeMillis();
{code}
- FileStore uses only one RaftClient (i.e. one connection). We may change the
code to use one RaftClient per file. The slowness may be due to the client.
(It is okay since we want to test the performance of the server but not the
client.)
> Compare the performance between HDFS and DataStreamApi
> ------------------------------------------------------
>
> Key: RATIS-1312
> URL: https://issues.apache.org/jira/browse/RATIS-1312
> Project: Ratis
> Issue Type: Sub-task
> Reporter: runzhiwang
> Priority: Major
> Attachments: hdfs.svg, screenshot-1.png, streaming.svg
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
