[
https://issues.apache.org/jira/browse/HADOOP-12910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15301094#comment-15301094
]
stack commented on HADOOP-12910:
--------------------------------
The patch here converts one method only? Is the intent to do all methods (along
w/ the spec suggested by [~steve_l]?)
Is this issue for an AsyncFileSystem or for an async rename only? Are we
targeting H3 only or is there some thought that this could get pulled back into
H2?
bq. Futures are a good match for the use case where the consumer wants to kick
of a multitude of async requests and wait until they are all done to make
progress, but we've found that there are also compelling use cases where you
want a small amount of logic and further async I/O in a completion handler, so
I might recommend supporting both Future-based results as well as
callback-based results.
A few of us (mainly [~Apache9]), are looking at being able to go async against
hdfs. There is already a stripped down async subset of dfsclient that we are
using to write our WALs done by [~Apache9] that uses way less resources while
going much faster (see HBASE-14790). As Duo says, we want to push this up into
HDFS, and given our good experience with this effort, we want to convert over
more of our HDFS connection to be async. Parking a resource waiting on a Future
to complete or keeping some list/queue of Futures which we check on a period to
see if it is 'done' is much less attractive (and less performant) to our being
instead notified on completion -- a callback (as [~bobhansen] suggests above in
the comment repeated here). Ideally we'd like to move our interaction with
HDFS to be event-driven (ultimately backing this up all the ways into the guts
of the regionserver, but that is another story)
OK if we put up suggested patch that say presumes jdk8/h3 only and instead of
returning Future, returns jdk8 CompletableFuture? Chatting yesterday, we think
we could consume/feed HDFS in a non-blocking way if we got back a
CompletableFuture (or we could add a callback handler as a param on a method if
folks preferred that?). We'd put up a sketch patch, and if amenable, we could
start up a sympathetic spec doc as a subtask so code and spec arrive at the
same time?
Thanks.
> Add new FileSystem API to support asynchronous method calls
> -----------------------------------------------------------
>
> Key: HADOOP-12910
> URL: https://issues.apache.org/jira/browse/HADOOP-12910
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
> Reporter: Tsz Wo Nicholas Sze
> Assignee: Xiaobing Zhou
> Attachments: HADOOP-12910-HDFS-9924.000.patch,
> HADOOP-12910-HDFS-9924.001.patch, HADOOP-12910-HDFS-9924.002.patch
>
>
> Add a new API, namely FutureFileSystem (or AsynchronousFileSystem, if it is a
> better name). All the APIs in FutureFileSystem are the same as FileSystem
> except that the return type is wrapped by Future, e.g.
> {code}
> //FileSystem
> public boolean rename(Path src, Path dst) throws IOException;
> //FutureFileSystem
> public Future<Boolean> rename(Path src, Path dst) throws IOException;
> {code}
> Note that FutureFileSystem does not extend FileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
