[
https://issues.apache.org/jira/browse/HDFS-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17140885#comment-17140885
]
Daniel Howard edited comment on HDFS-13082 at 7/10/20, 7:28 PM:
----------------------------------------------------------------
I am running into this as well on Ubuntu 20.04. I can confirm that setting
*{{nfs.aix.compatibility.mode.enabled}}* to *{{true}}* resolves this problem.
For example:
{{0-15:58 djh@c24-03-06 ~> *ls /hadoop/wxxxs/data/*}}
{{# No files listed}}
{{0-16:01 djh@c24-03-06 ~> *touch /hadoop/wxxxs/data/foo*}}
{{0-16:01 djh@c24-03-06 ~> *ls /hadoop/wxxxs/data/*}}
{{foo packed-hbfs/ raw/ tmp/}}
{{0-16:01 djh@c24-03-06 ~> *rm /hadoop/wxxxs/data/foo*}}
{{0-16:01 djh@c24-03-06 ~> *ls /hadoop/wxxxs/data/*}}
{{packed-hbfs/ raw/ tmp/}}
Writing to this directory forced the NFS server to return the correct directory
contents.
I have a bunch of this in the log:
{{2020-06-19 16:01:35,281 ERROR org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3:
cookieverf mismatch. request cookieverf: 1591897331315 dir cookieverf:
1592428367587}}
{{2020-06-19 16:01:35,287 ERROR
org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: cookieverf mismatch. request
cookieverf: 1591897331315 dir cookieverf: 1592428367587}}
{{2020-06-19 16:01:35,454 ERROR
org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: cookieverf mismatch. request
cookieverf: 1591897331315 dir cookieverf: 1592428367587}}
If AIX compatibility is enabled, the log messages change FROM
{{ERROR org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: cookieverf
mismatch.[...]}}
TO
{{WARN org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: AIX compatibility mode
enabled...}}
Judging by the code in {{RpcProgramNfs3.java}} if the {{cookieverf}} does not
match a directory's {{mtime}}, then normally the Nfs3 server will return an
error to the client. In AIX compatibility mode, the Nfs3 server instead logs a
warning and then constructs the response it would have constructed had there
been no {{cookieverf}} mismatch. What does this all mean? I don't know, but I
am working to see if I can trigger an empty directory situation with the AIX
compat enabled.
was (Author: dannyman):
I am running into this as well on Ubuntu 20.04. I am in the process of testing
the AIX compatibility mode.
For example:
{{0-15:58 djh@c24-03-06 ~> *ls /hadoop/wxxxs/data/*}}
{{# No files listed}}
{{0-16:01 djh@c24-03-06 ~> *touch /hadoop/wxxxs/data/foo*}}
{{0-16:01 djh@c24-03-06 ~> *ls /hadoop/wxxxs/data/*}}
{{foo packed-hbfs/ raw/ tmp/}}
{{0-16:01 djh@c24-03-06 ~> *rm /hadoop/wxxxs/data/foo*}}
{{0-16:01 djh@c24-03-06 ~> *ls /hadoop/wxxxs/data/*}}
{{packed-hbfs/ raw/ tmp/}}
Writing to this directory forced the NFS server to return the correct directory
contents.
I have a bunch of this in the log:
{{2020-06-19 16:01:35,281 ERROR org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3:
cookieverf mismatch. request cookieverf: 1591897331315 dir cookieverf:
1592428367587}}
{{2020-06-19 16:01:35,287 ERROR org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3:
cookieverf mismatch. request cookieverf: 1591897331315 dir cookieverf:
1592428367587}}
{{2020-06-19 16:01:35,454 ERROR org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3:
cookieverf mismatch. request cookieverf: 1591897331315 dir cookieverf:
1592428367587}}
If AIX compatibility is enabled, the log messages change FROM
{{ERROR org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: cookieverf
mismatch.[...]}}
TO
{{WARN org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3: AIX compatibility mode
enabled...}}
Judging by the code in {{RpcProgramNfs3.java}} if the {{cookieverf}} does not
match a directory's {{mtime}}, then normally the Nfs3 server will return an
error to the client. In AIX compatibility mode, the Nfs3 server instead logs a
warning and then constructs the response it would have constructed had there
been no {{cookieverf}} mismatch. What does this all mean? I don't know, but I
am working to see if I can trigger an empty directory situation with the AIX
compat enabled.
> cookieverf mismatch error over NFS gateway on Linux
> ---------------------------------------------------
>
> Key: HDFS-13082
> URL: https://issues.apache.org/jira/browse/HDFS-13082
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Affects Versions: 2.7.3
> Reporter: Dan Moraru
> Priority: Minor
>
> Running 'ls' on some directories over an HDFS-NFS gateway sometimes fails to
> list the contents of those directories. Running 'ls' on those same
> directories mounted via FUSE works. The NFS gateway logs errors like the
> following:
> 2018-01-29 11:53:01,130 ERROR org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3:
> cookieverf mismatch. request cookieverf: 1513390944415 dir cookieverf:
> 1516920857335
> Reviewing
> hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java
> suggested that these errors can be avoided by setting
> nfs.aix.compatibility.mode.enabled=true, and that is indeed the case. The
> documentation lists https://issues.apache.org/jira/browse/HDFS-6549 as a
> known issue, but also goes on to say that "regular, non-AIX clients should
> NOT enable AIX compatibility mode. The work-arounds implemented by AIX
> compatibility mode effectively disable safeguards to ensure that listing of
> directory contents via NFS returns consistent results, and that all data sent
> to the NFS server can be assured to have been committed." Server and client
> is this case are one and the same, running Scientific Linux 7.4.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
