[
https://issues.apache.org/jira/browse/HDFS-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15837416#comment-15837416
]
Duo Zhang commented on HDFS-9924:
---------------------------------
Any updates here? Seems no commit on HDFS-9924 branch for a long time...
I can help a bit here as I still want to move the
FanOutOneBlockAsyncDFSOutputStream into HDFS rather than maintain it in HBase...
I think the problem here is that, our interface is blocking. It is really
awkward to implement async stuffs on top of an blocking interface so I do not
like the current approach. I think we can either
1. Use grpc instead of the current rpc. Add a port unification service in front
of the grpc server and the old rpc server to support both grpc client and old
client. Yeah we need to write lots of code if we choose this way, but I think
most code are just boilerplate. Another benefit is that multi language support
will be much easier if we use standard grpc.
2. Use grpc but do not use the HTTP/2 transport, implement our own transport. I
haven't tried this yet but grpc-java does support customized transport so I
think it is possible. The benefit is that we do not need port unification
service at server side and do not need to maintain two implementations at
server side.
3. Use the old protobuf rpc interface and implement a new rpc framework. The
benefit is that we also do not need port unification service at server side and
do not need to maintain two implementations at server side. And one more thing
is that we do not need to upgrade protobuf to 3.x.
4. As said in the design doc above, generate new interfaces which return a
CompletableFuture based on the old blocking interface. And add a new feature in
the current rpc implementation to support the new interface.
I'm OK with any of the approach above. Can start working on branch HDFS-9924
after we decide which one to use.
Thanks.
> [umbrella] Nonblocking HDFS Access
> ----------------------------------
>
> Key: HDFS-9924
> URL: https://issues.apache.org/jira/browse/HDFS-9924
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: fs
> Reporter: Tsz Wo Nicholas Sze
> Assignee: Xiaobing Zhou
> Attachments: AsyncHdfs20160510.pdf, Async-HDFS-Performance-Report.pdf
>
>
> This is an umbrella JIRA for supporting Nonblocking HDFS Access.
> Currently, all the API methods are blocking calls -- the caller is blocked
> until the method returns. It is very slow if a client makes a large number
> of independent calls in a single thread since each call has to wait until the
> previous call is finished. It is inefficient if a client needs to create a
> large number of threads to invoke the calls.
> We propose adding a new API to support nonblocking calls, i.e. the caller is
> not blocked. The methods in the new API immediately return a Java Future
> object. The return value can be obtained by the usual Future.get() method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
