[
https://issues.apache.org/jira/browse/HDFS-4750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13642065#comment-13642065
]
Brandon Li commented on HDFS-4750:
----------------------------------
@Brock
{quote}
What happens if the 10 seconds expires and the prerequisite write has not been
received? The biggest issue I had when moving the proxy from basically working
to handling multiple heavy use clients was memory consumption while waiting for
pre-requisite writes. I eventually had to write pending writes to a file. This
is documented in this issue https://github.com/brockn/hdfs-nfs-proxy/issues/7
{quote}
The pending write requests will fail after timeout. Saving pending writes in
files can help in some cases but also introduces some problems. First, it
doesn't eliminate the problem. The prerequisite write may never arrive if 10
seconds is not long enough. Even the prerequisite write arrives finally, the
accumulated writes in the file may have timed out. Secondly, it makes the
server stateful (or have more state information). To support HA later, we have
to move the state information from one NFS gateway to another in order to
recover. If the state recovery takes too long to finish, it can cause the
clients new requests to fail.
More testing and research work is needed here.
{quote}
Interesting, what version of linux have tried? I believe I was using RHEL 5.X.
{quote}
CentOS6.3 and Mac 10.7.5
{quote}
Additionally, is there anything the user can do to force the writes? i.e. If
the user has control over the program, could they do a fsync(fd) to force the
flush?
{quote} fsync could trigger NFS commit, which will sync and persist the data.
{quote} I could imagine many users believing once they have closed a file via
NFS that exact file will be available via one the other APIs. We'll need to
make this limitation blatantly obvious to users otherwise it will likely become
a support headache.{quote}
If the application expects that, after closing file through one NFS gateway,
the new data is immediately available to all other NFS gateways, the
application should do a sync call after close.
This is not a limitation only to this NFS implemtation. POSIX close doesn't
sync data implicitly.
> Support NFSv3 interface to HDFS
> -------------------------------
>
> Key: HDFS-4750
> URL: https://issues.apache.org/jira/browse/HDFS-4750
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 3.0.0
> Reporter: Brandon Li
> Assignee: Brandon Li
> Attachments: HADOOP-NFS-Proposal.pdf
>
>
> Access HDFS is usually done through HDFS Client or webHDFS. Lack of seamless
> integration with client’s file system makes it difficult for users and
> impossible for some applications to access HDFS. NFS interface support is one
> way for HDFS to have such easy integration.
> This JIRA is to track the NFS protocol support for accessing HDFS. With HDFS
> client, webHDFS and the NFS interface, HDFS will be easier to access and be
> able support more applications and use cases.
> We will upload the design document and the initial implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
