elek opened a new pull request #2256:
URL: https://github.com/apache/ozone/pull/2256
## What changes were proposed in this pull request?
This was discussed offline a few times. (cc @bshashikant @mukul1987)
Today we have server/client utilities/API for Hadoop RPC and GRPC based
services but we don't have any unified and tested API for data streaming.
It is suggested to create a generic API for streaming.
As I need to modify the closed container replication I already created a POC
based on Netty (instead of GRPC).
This patch shows the first proposed version.
1. It uses a file-based API (the files should be specified to stream for a
given id) because we can use Netty/native streaming without buffering in Java
memory.
2. Offsets can be supported (but not supported yet).
3. Freon test is included which shows 8x better performance compared to the
old GRPC based container replication. (220Mb / sec vs 1.6-1.7 Gb/sec, using one
thread + tmpfs)
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5142
## How was this patch tested?
```
./ozone freon strmg -n 1000 --files 10
```
It creates a directory with 10*100MB files (1G) and replicates them 1000
times. Supposed to be finished under 10 minutes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
