[
https://issues.apache.org/jira/browse/RATIS-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17238817#comment-17238817
]
Tsz-wo Sze commented on RATIS-1176:
-----------------------------------
Just have tried DefaultFileRegion in RATIS-1179. It is much faster than
transferToArbitraryChannel.
{code}
2020-11-26 01:07:05,062 [Time-limited test] INFO
datastream.TestNettyDataStreamWithGrpcCluster
(DataStreamClusterTests.java:runTestWriteFile(101)) - 0:
writeAsyncDefaultFileRegion elapsed 107ms
2020-11-26 01:07:06,032 [Time-limited test] INFO
datastream.TestNettyDataStreamWithGrpcCluster
(DataStreamClusterTests.java:runTestWriteFile(101)) - 0:
transferToWritableByteChannel elapsed 942ms
2020-11-26 01:07:06,184 [Time-limited test] INFO
datastream.TestNettyDataStreamWithGrpcCluster
(DataStreamClusterTests.java:runTestWriteFile(101)) - 1:
writeAsyncDefaultFileRegion elapsed 109ms
2020-11-26 01:07:07,032 [Time-limited test] INFO
datastream.TestNettyDataStreamWithGrpcCluster
(DataStreamClusterTests.java:runTestWriteFile(101)) - 1:
transferToWritableByteChannel elapsed 770ms
2020-11-26 01:07:07,134 [Time-limited test] INFO
datastream.TestNettyDataStreamWithGrpcCluster
(DataStreamClusterTests.java:runTestWriteFile(101)) - 2:
writeAsyncDefaultFileRegion elapsed 69ms
2020-11-26 01:07:07,633 [Time-limited test] INFO
datastream.TestNettyDataStreamWithGrpcCluster
(DataStreamClusterTests.java:runTestWriteFile(101)) - 2:
transferToWritableByteChannel elapsed 472ms
{code}
The test sends a 10MB on disk file in DataStreamClusterTests.testStreamWrites.
> Benchmark various ways to stream data
> -------------------------------------
>
> Key: RATIS-1176
> URL: https://issues.apache.org/jira/browse/RATIS-1176
> Project: Ratis
> Issue Type: Sub-task
> Components: client, Streaming
> Reporter: Tsz-wo Sze
> Priority: Major
> Attachments: image-2020-11-25-07-40-50-383.png
>
>
> In RATIS-1175, we provided a WritableByteChannel view of DataStreamOutput in
> order to support FileChannel.transferTo. However, [~runzhiwang] pointed out
> that sun.nio.ch.FileChannelImpl.transferTo has three submethods
> - transferToDirectly (fastest)
> - transferToTrustedChannel
> - transferToArbitraryChannel (slowest, requires buffer copying)
> Unfortunately, our current implementation only able to use
> transferToArbitraryChannel.
> There are several ideas below to improve the performance. We should
> benchmark them.
> # Improve the current implementation of WritableByteChannel so that it may be
> able to use a faster transferTo method.
> # Use
> [FileChannel.map(..)|https://docs.oracle.com/javase/8/docs/api/java/nio/channels/FileChannel.html#map-java.nio.channels.FileChannel.MapMode-long-long-]
> and pass MappedByteBuffer to our DataStreamOutput.writeAsync method.
> # Add a new API
> {code}
> //DataStreamOutput
> CompletableFuture<DataStreamReply> writeAsync(File);
> {code}
> Internally, use Netty DefaultFileRegion for zero-copy file transfer:
> https://github.com/netty/netty/blob/4.1/example/src/main/java/io/netty/example/file/FileServerHandler.java#L53
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
