[
https://issues.apache.org/jira/browse/RATIS-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17280482#comment-17280482
]
runzhiwang edited comment on RATIS-1312 at 2/7/21, 12:31 PM:
-------------------------------------------------------------
[~szetszwo] Hi, let's discuss here.
I have test the performance of HDFS and Ratis streaming. But Ratis streaming is
about 30% slower than HDFS. When 2 clients * 128MB * 200 files, Ratis streaming
cost about 80 seconds, HDFS cost about 60 seconds. Not sure why.
In my test, as the following code shows, I did not call startTransaction, to
avoid submitting raft request, so that we can only compare the performance of
network and write disk.
I have attached the flame graph of hdfs and ratis streaming. I perf two
datanodes, and there are total 3 datanodes.
{code:java}
composeAsync(info.getPrevious(), requestExecutor, n ->
JavaUtils.allOf(remoteWrites)
.thenCombineAsync(localWrite, (v, bytesWritten) -> {
if (request.getType() == Type.STREAM_HEADER
|| (request.getType() == Type.STREAM_DATA && !close)) {
sendReply(remoteWrites, request, bytesWritten, ctx);
} else if (close) {
sendReply(remoteWrites, request, bytesWritten, ctx);
// if (info.isPrimary()) {
// // after all server close stream, primary server start
transaction
// startTransaction(info, request, bytesWritten, ctx);
// } else {
// sendReply(remoteWrites, request, bytesWritten, ctx);
// }
} else {
throw new IllegalStateException(this + ": Unexpected type " +
request.getType() + ", request=" + request);
}
return null;
}, requestExecutor)).whenComplete((v, exception) -> {
{code}
was (Author: yjxxtd):
[~szetszwo] Hi, let's discuss here.
I have test the performance of HDFS and Ratis streaming. But Ratis streaming is
about 30% slower than HDFS. When 2 clients * 128MB * 200 files, Ratis streaming
cost about 80 seconds, HDFS cost about 60 seconds. Not sure why.
In my test, as the following code shows, I did not call startTransaction, to
avoid submitting raft request, so that we can only compare the performance of
network and write disk.
I have attached the flame graph of hdfs and ratis streaming.
{code:java}
composeAsync(info.getPrevious(), requestExecutor, n ->
JavaUtils.allOf(remoteWrites)
.thenCombineAsync(localWrite, (v, bytesWritten) -> {
if (request.getType() == Type.STREAM_HEADER
|| (request.getType() == Type.STREAM_DATA && !close)) {
sendReply(remoteWrites, request, bytesWritten, ctx);
} else if (close) {
sendReply(remoteWrites, request, bytesWritten, ctx);
// if (info.isPrimary()) {
// // after all server close stream, primary server start
transaction
// startTransaction(info, request, bytesWritten, ctx);
// } else {
// sendReply(remoteWrites, request, bytesWritten, ctx);
// }
} else {
throw new IllegalStateException(this + ": Unexpected type " +
request.getType() + ", request=" + request);
}
return null;
}, requestExecutor)).whenComplete((v, exception) -> {
{code}
> Compare the performance between HDFS and DataStreamApi
> ------------------------------------------------------
>
> Key: RATIS-1312
> URL: https://issues.apache.org/jira/browse/RATIS-1312
> Project: Ratis
> Issue Type: Sub-task
> Reporter: runzhiwang
> Priority: Major
> Attachments: hdfs-1.svg, hdfs-2.svg, streaming-1.svg, streaming-2.svg
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
