Hello, Strahil, it looks to me like gluster issue #876 is not the same problem, but I confess that I don't understand all of that issue.
Stefan, thanks very much for the suggestion. Using nfsvers=4.1 does solve the problem on an NFS client running CentOS 7, while nfsversion=4.0 has the problem, so that's good. Using nfsvers=4.1 doesn't solve the problem on a client using CentOS 6 though. I have verified that version 4.1 is reported by "mount": gfs1:/mnt/glusterfs on /var/lib/gfs type nfs (rw,noatime,hard,bg,intr,nfsvers=4,minorversion=1,addr=1.2.3.4,clientaddr=5.6.7.8) So it's strange that NFS version 4.1 doesn't solve it on the older CentOS release. Unfortunately our customer does have a few of these old boxes still around. Thanks again. On Fri, 13 Aug 2021 at 02:33, Stefan Solbrig <[email protected]> wrote: > A while ago, I had a similar problem with an nfs re-export of GlusterFS. > In my case, enforcing NFS version 4.2 solved the issuse: > > mount options: vers=4,rw,hard,nosuid,minorversion=2 > > best wishes, > Stefan > > -- > Dr. Stefan Solbrig > Universität Regensburg, Fakultät für Physik, > 93040 Regensburg, Germany > Tel +49-941-943-2097 > > > Am 12.08.2021 um 07:40 schrieb Strahil Nikolov <[email protected]>: > > > > The FUSE client mount options "do not ring any bells" to me - they look > fine. > > > > For the ACL stuff, I couldn't find the commit too.Maybe it's for a newer > version than yours. > > > > You can check if gluster issue #876 matches your case. > > > > Which version of gluster are you using ? > > > > Best Regards, > > Strahil Nikolov > > > > > > On Thu, Aug 12, 2021 at 6:34, David Cunningham > > <[email protected]> wrote: > > I've noticed something interesting: when the "cp" over NFS is done the > first time it gives the "cp: closing" permission denied error as described > before and the destination file is created with zero size. > > > > If we run the same "cp" over NFS a second time the files are written > correctly, with the full file data, and there's no error. So it appears > that the problem does not occur if the destination file exists. > > > > Does that give anyone a clue as to what's happening? Thanks. > > > > > > On Thu, 12 Aug 2021 at 13:50, David Cunningham < > [email protected]> wrote: > > Hi, > > > > Gilberto, sorry I didn't realise those were server options. After adding > them to the /etc/exports entry and restarting nfsd the same problem still > occurs. > > > > Strahil, I ran "gluster volume set help | egrep -i 'acl|xlator'" on the > GlusterFS node/FUSE client and got the following results. Is one of them > the option you mention? > > Description: inode-read fops happen only on one of the bricks in > replicate. Afr will prefer the one specified using this option if it is not > stale. Option value must be one of the xlator names of the children. Ex: > <volname>-client-0 till <volname>-client-<number-of-bricks - 1> > > Description: Cache samba metadata (user.DOSATTRIB, security.NTACL xattr) > > Description: maximum size of cache consumed by readdir-ahead xlator. > This value is global and total memory consumption by readdir-ahead is > capped by this value, irrespective of the number/size of directories cached > > > > Could the issue be anything to do with the FUSE client mount options? > The output of "mount | grep glusterfs" gives: > > gfs1:/gvol0 on /mnt/glusterfs type fuse.glusterfs > (rw,noatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > > > > Thanks again. > > > > > > On Thu, 12 Aug 2021 at 01:25, Gilberto Ferreira < > [email protected]> wrote: > > Those options you need put it in the NFS server options, generally in > /etc/exports > > --- > > Gilberto Nunes Ferreira > > (47) 99676-7530 - Whatsapp / Telegram > > > > > > > > > > > > Em ter., 10 de ago. de 2021 às 18:24, David Cunningham < > [email protected]> escreveu: > > Hi Strahil and Gilberto, > > > > Thanks very much for your replies. SELinux is disabled on the NFS server > (and the client too), and both have the same UID and GID for the user who > owns the files. > > > > On the NFS mount we had options "rw,noatime,hard,bg,intr,vers=4". I > added "async" which did not solve the problem, and the NFS client mount > gave an error when trying to use "no_root_squash" or "no_subtree_check". > Gilberto, is there a specific reason why you suggested those options? > > > > Thanks again. > > > > > > On Wed, 11 Aug 2021 at 03:55, Gilberto Ferreira < > [email protected]> wrote: > > HOw about the NFS options? > > (rw,async,no_root_squash,no_subtree_check) > > --- > > Gilberto Nunes Ferreira > > (47) 99676-7530 - Whatsapp / Telegram > > > > > > > > > > > > Em ter., 10 de ago. de 2021 às 12:46, Strahil Nikolov < > [email protected]> escreveu: > > Hey David, > > > > can you give the volume info ? > > > > Also, I assume SELINUX is in permissive/disabled state. > > > > What about the uod of the user on the nfs client and the nfs server ? Is > it the same ? > > > > Best Regards, > > Strahil Nikolov > > > > On Tue, Aug 10, 2021 at 5:52, David Cunningham > > <[email protected]> wrote: > > Hello, > > > > We have a GlusterFS node which also uses the FUSE client to mount the > filesystem. The same GlusterFS node server also runs an NFS server which > exports the FUSE client mount, and another machine NFS mounts it. > > > > When the NFS client writes data to the mounted filesystem we are seeing > "Permission denied" errors like this: > > > > cp: closing > `/var/lib/gfs/company/david/1075/Copyrec/1628448189883606-203-17184805327-out-08-08-21-14~43~10-203.mp3': > Permission denied > > > > The file mentioned in the error is actually created on the GlusterFS > filesystem, but has zero size, so the problem is not a normal Linux > filesystem permission one. > > > > In the brick log nodirectwritedata-gluster-gvol0.log on the GlusterFS > node we see an error as follows. Would anyone have a suggestion on what the > problem might be? Thank you in advance! > > > > [2021-08-10 02:30:20.359159] I [MSGID: 139001] > [posix-acl.c:262:posix_acl_log_permit_denied] 0-gvol0-access-control: > client: > CTX_ID:8f69363a-f0f4-44e1-84e9-69dfa77a8164-GRAPH_ID:0-PID:2657-HOST:gfs1.company.com-PC_NAME:gvol0-client-0-RECON_NO:-0, > gfid: f70b1cd6-745a-4ea6-b0a5-1fcfef960f15, > req(uid:106,gid:111,perm:2,ngrps:0), > ctx(uid:106,gid:111,in-groups:1,perm:000,updated-fop:SETATTR, acl:-) > [Permission denied] > > [2021-08-10 02:30:20.359187] E [MSGID: 115070] > [server-rpc-fops_v2.c:1502:server4_open_cbk] 0-gvol0-server: 5554927: OPEN > /company/david/1075/Copyrec/1628448189883606-203-17184805327-out-08-08-21-14~43~10-203.mp3 > (f70b1cd6-745a-4ea6-b0a5-1fcfef960f15), client: > CTX_ID:8f69363a-f0f4-44e1-84e9-69dfa77a8164-GRAPH_ID:0-PID:2657-HOST:gfs1.company.com-PC_NAME:gvol0-client-0-RECON_NO:-0, > error-xlator: gvol0-access-control [Permission denied] > > > > -- > > David Cunningham, Voisonics Limited > > http://voisonics.com/ > > USA: +1 213 221 1092 > > New Zealand: +64 (0)28 2558 3782 > > ________ > > > > > > > > Community Meeting Calendar: > > > > Schedule - > > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > > Bridge: https://meet.google.com/cpu-eiue-hvk > > Gluster-users mailing list > > [email protected] > > https://lists.gluster.org/mailman/listinfo/gluster-users > > ________ > > > > > > > > Community Meeting Calendar: > > > > Schedule - > > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > > Bridge: https://meet.google.com/cpu-eiue-hvk > > Gluster-users mailing list > > [email protected] > > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > > > -- > > David Cunningham, Voisonics Limited > > http://voisonics.com/ > > USA: +1 213 221 1092 > > New Zealand: +64 (0)28 2558 3782 > > > > > > -- > > David Cunningham, Voisonics Limited > > http://voisonics.com/ > > USA: +1 213 221 1092 > > New Zealand: +64 (0)28 2558 3782 > > > > > > -- > > David Cunningham, Voisonics Limited > > http://voisonics.com/ > > USA: +1 213 221 1092 > > New Zealand: +64 (0)28 2558 3782 > > ________ > > > > > > > > Community Meeting Calendar: > > > > Schedule - > > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > > Bridge: https://meet.google.com/cpu-eiue-hvk > > Gluster-users mailing list > > [email protected] > > https://lists.gluster.org/mailman/listinfo/gluster-users > > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://meet.google.com/cpu-eiue-hvk > Gluster-users mailing list > [email protected] > https://lists.gluster.org/mailman/listinfo/gluster-users > -- David Cunningham, Voisonics Limited http://voisonics.com/ USA: +1 213 221 1092 New Zealand: +64 (0)28 2558 3782
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list [email protected] https://lists.gluster.org/mailman/listinfo/gluster-users
