[
https://issues.apache.org/jira/browse/HDFS-9020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729874#comment-14729874
]
Daryn Sharp commented on HDFS-9020:
-----------------------------------
In fairness, I didn't advocate for the above. :) I agreed that chunked writes
are probably the only "clean" design but it's complicated. I think moving from
a state-less operation with a fixed lifecycle, to an implementation that
becomes state-full and async with a non-deterministic lifecycle will be
difficult.
A big concern is the caching of dfsclients & streams. It can easily lead to
leaks and exhausting fds on the NN. Caching clients per user is probably an
attempt to address that but won't work. All subsequent file streams are
relying on the original token. If that token is cancelled, all sessions will
die even though they had their own valid tokens.
Cached clients pose problems for leases. If a stream is orphaned, the NN
should eventually recover the lease but the cached client will keep the lease
alive. So now you must have some additional mechanism for timing out open
streams and closing them.
Lease recovery should always be a premeditated action, not an implicit action.
If a webhdfs client opens a file, and another client forcibly revokes the
lease, the original webhdfs client shouldn't just "steal" it back. If the
intention is to abort the cached dfsclient on another node, the client won't
know the lease is gone until it tries to add or complete a block - but an idle
stream isn't going to do that.
I'm not sure how this can be done cleanly & correctly. Webhdfs has been so
problematic that I'm hesitant for it to become more complex.
> Support hflush/hsync in WebHDFS
> -------------------------------
>
> Key: HDFS-9020
> URL: https://issues.apache.org/jira/browse/HDFS-9020
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
> Reporter: Chris Douglas
> Attachments: HDFS-9020-alt.txt
>
>
> In the current implementation, hflush/hsync have no effect on WebHDFS
> streams, particularly w.r.t. visibility to other clients. This proposes to
> extend the protocol and implementation to enable this functionality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
