Re: [Gluster-users] I/O error repaired only by owner or root access
-- Dan Bretherton ESSC Computer System Manager Department of Meteorology Harry Pitt Building, 3 Earley Gate University of Reading Reading, RG6 7BE (or RG6 6AL for postal service deliveries) UK Tel. +44 118 378 5205, Fax: +44 118 378 6413 -- ## Please sponsor me to run in VSO's 30km Race to the Eye ## ##http://www.justgiving.com/DanBretherton ## On 02/25/2013 04:44 PM, Dan Bretherton wrote: On 02/25/2013 03:49 PM, Shawn Nock wrote: Dan Bretherton d.a.brether...@reading.ac.uk writes: Hello Rajesh- Here are the permissions. The path in question is a directory. [sms05dab@jupiter ~]$ ls -ld /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN/Ctl drwxr-xr-x 60 vq901510 nemo 110592 Feb 23 04:37 /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN/Ctl [sms05dab@jupiter ~]$ ls -ld /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN lrwxrwxrwx 1 gcs nemo 49 Feb 1 2012 /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN - /data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN [sms05dab@jupiter ~]$ ls -ld /data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN drwxr-xr-x 27 gcs nemo 99210 Feb 23 03:14 /data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN As you can see the parent directory in this case was a symlink but that's not significant. I ran the ls -l commands using my account - sms05dab, but the problem was originally reported by user vq901510. until I did ls -l as root neither of us could access the directory, because the parent directory was owned by user gcs. Usually the problem is related to ownership of the file or directory itself. This is the first time I have seen the I/O error caused by parent directory permissions. This problem seems to have started following an add-brick operation a few weeks ago, after which I started gluster volume rebalance VOLNAME fix-layout (which is still running). It occurred to me that the problem could be related to link files, many of which need to be rewritten following add-brick operations. This could explain why the ownership of the parent directory is significant, because users sms05dab and vq901510 don't have permission to write in the parent directory owned by user gcs. Normally this wouldn't be a problem because only read access to other users' data is required, but it appears as though read access was being denied because the new link file couldn't be written by unprivileged users. Is this a plausible explanation of the I/O error do you think? This sounds like my recent bug: https://bugzilla.redhat.com/show_bug.cgi?id=913699 In the bug, I said that writing on the fuse mount of one of the brick servers fixed the problem but those were the only hosts I was attempting access as root. Thanks Shawn. To confirm that we are seeing the same bug I will try accessing affected files from a FUSE mount on a server the next time it happens. -Dan. I have updated the bug report with another recent example. In this most recent case, attempting to open a file for reading as an unprivileged user resulted in the error Invalid argument, although ls -l worked without error. This happened on a compute server and via a GlusterFS client mount point on a GlusterFS storage server. Changing the ownership of the parent directory to give the unprivileged user write access to the directory (making it writeable by group) allowed the user to open the file for reading. -Dan. ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] I/O error repaired only by owner or root access
Hello Rajesh- Here are the permissions. The path in question is a directory. [sms05dab@jupiter ~]$ ls -ld /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN/Ctl drwxr-xr-x 60 vq901510 nemo 110592 Feb 23 04:37 /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN/Ctl [sms05dab@jupiter ~]$ ls -ld /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN lrwxrwxrwx 1 gcs nemo 49 Feb 1 2012 /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN - /data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN [sms05dab@jupiter ~]$ ls -ld /data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN drwxr-xr-x 27 gcs nemo 99210 Feb 23 03:14 /data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN As you can see the parent directory in this case was a symlink but that's not significant. I ran the ls -l commands using my account - sms05dab, but the problem was originally reported by user vq901510. until I did ls -l as root neither of us could access the directory, because the parent directory was owned by user gcs. Usually the problem is related to ownership of the file or directory itself. This is the first time I have seen the I/O error caused by parent directory permissions. This problem seems to have started following an add-brick operation a few weeks ago, after which I started gluster volume rebalance VOLNAME fix-layout (which is still running). It occurred to me that the problem could be related to link files, many of which need to be rewritten following add-brick operations. This could explain why the ownership of the parent directory is significant, because users sms05dab and vq901510 don't have permission to write in the parent directory owned by user gcs. Normally this wouldn't be a problem because only read access to other users' data is required, but it appears as though read access was being denied because the new link file couldn't be written by unprivileged users. Is this a plausible explanation of the I/O error do you think? -Dan. -- Dan Bretherton ESSC Computer System Manager Department of Meteorology Harry Pitt Building, 3 Earley Gate University of Reading Reading, RG6 7BE (or RG6 6AL for postal service deliveries) UK Tel. +44 118 378 5205, Fax: +44 118 378 6413 -- ## Please sponsor me to run in VSO's 30km Race to the Eye ## ##http://www.justgiving.com/DanBretherton ## On 02/22/2013 11:56 AM, Rajesh Amaravathi wrote: Hi Dan, Could you please provide the following info (1) the exact permissions of the file you are accessing and its parent directory, (2) the user from which 'ls -l' is issued, and (3) the owner of the file, and the parent directory. You could open a bug for it if it is seen several times. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: Dan Bretherton d.a.brether...@reading.ac.uk To: gluster-users gluster-users@gluster.org Sent: Thursday, February 21, 2013 8:16:40 PM Subject: [Gluster-users] I/O error repaired only by owner or root access Dear All- Several users are having a lot of trouble reading files belonging to other users. Here is an example. [sms05dab@jupiter ~]$ ls -l /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN/Ctl ls: /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN/Ctl: Input/output error The corresponding nfs.log messages are shown below. [2013-02-21 12:11:39.204659] W [nfs3.c:727:nfs3svc_getattr_stat_cbk] 0-nfs: fe2ba5b8: /gorgon/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN = -1 (Invalid argument) [2013-02-21 12:11:39.204778] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: fe2ba5b8, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument) [2013-02-21 12:11:39.215345] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-nemo2-dht: deleting stale linkfile /gorgon/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN on nemo2-replicate-0 [2013-02-21 12:11:39.225674] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-nemo2-client-1: remote operation failed: Permission denied [2013-02-21 12:11:39.225786] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-nemo2-client-0: remote operation failed: Permission denied [2013-02-21 12:11:39.681029] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-nemo2-client-18: remote operation failed: Permission denied. Path: /gorgon/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN (1662aa0a-d43b-4c2e-9be9-407eb7a89e85) [2013-02-21 12:11:39.681400] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-nemo2-client-19: remote operation failed: Permission denied. Path: /gorgon/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN (1662aa0a-d43b-4c2e-9be9-407eb7a89e85) [2013-02-21 12:11:39.682268] W [nfs3.c:1627:nfs3svc_readlink_cbk] 0-nfs: 2ca5b8: /gorgon/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN = -1 (Invalid argument) [2013-02-21 12:11:39.682338] W [nfs3-helpers.c:3403:nfs3_log_readlink_res] 0-nfs-nfsv3: XID: 2ca5b8, READLINK: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument), target: (null) I managed to access the same directory as the owner (or any user with write access including root) without any trouble, and after that access from my normal user account was fine
Re: [Gluster-users] I/O error repaired only by owner or root access
Dan Bretherton d.a.brether...@reading.ac.uk writes: Hello Rajesh- Here are the permissions. The path in question is a directory. [sms05dab@jupiter ~]$ ls -ld /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN/Ctl drwxr-xr-x 60 vq901510 nemo 110592 Feb 23 04:37 /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN/Ctl [sms05dab@jupiter ~]$ ls -ld /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN lrwxrwxrwx 1 gcs nemo 49 Feb 1 2012 /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN - /data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN [sms05dab@jupiter ~]$ ls -ld /data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN drwxr-xr-x 27 gcs nemo 99210 Feb 23 03:14 /data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN As you can see the parent directory in this case was a symlink but that's not significant. I ran the ls -l commands using my account - sms05dab, but the problem was originally reported by user vq901510. until I did ls -l as root neither of us could access the directory, because the parent directory was owned by user gcs. Usually the problem is related to ownership of the file or directory itself. This is the first time I have seen the I/O error caused by parent directory permissions. This problem seems to have started following an add-brick operation a few weeks ago, after which I started gluster volume rebalance VOLNAME fix-layout (which is still running). It occurred to me that the problem could be related to link files, many of which need to be rewritten following add-brick operations. This could explain why the ownership of the parent directory is significant, because users sms05dab and vq901510 don't have permission to write in the parent directory owned by user gcs. Normally this wouldn't be a problem because only read access to other users' data is required, but it appears as though read access was being denied because the new link file couldn't be written by unprivileged users. Is this a plausible explanation of the I/O error do you think? This sounds like my recent bug: https://bugzilla.redhat.com/show_bug.cgi?id=913699 In the bug, I said that writing on the fuse mount of one of the brick servers fixed the problem but those were the only hosts I was attempting access as root. -- Shawn Nock (OpenPGP: 0x65118FA5) pgpdFuvpbtdLl.pgp Description: PGP signature ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] I/O error repaired only by owner or root access
On 02/25/2013 03:49 PM, Shawn Nock wrote: Dan Bretherton d.a.brether...@reading.ac.uk writes: Hello Rajesh- Here are the permissions. The path in question is a directory. [sms05dab@jupiter ~]$ ls -ld /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN/Ctl drwxr-xr-x 60 vq901510 nemo 110592 Feb 23 04:37 /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN/Ctl [sms05dab@jupiter ~]$ ls -ld /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN lrwxrwxrwx 1 gcs nemo 49 Feb 1 2012 /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN - /data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN [sms05dab@jupiter ~]$ ls -ld /data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN drwxr-xr-x 27 gcs nemo 99210 Feb 23 03:14 /data/pegasus/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN As you can see the parent directory in this case was a symlink but that's not significant. I ran the ls -l commands using my account - sms05dab, but the problem was originally reported by user vq901510. until I did ls -l as root neither of us could access the directory, because the parent directory was owned by user gcs. Usually the problem is related to ownership of the file or directory itself. This is the first time I have seen the I/O error caused by parent directory permissions. This problem seems to have started following an add-brick operation a few weeks ago, after which I started gluster volume rebalance VOLNAME fix-layout (which is still running). It occurred to me that the problem could be related to link files, many of which need to be rewritten following add-brick operations. This could explain why the ownership of the parent directory is significant, because users sms05dab and vq901510 don't have permission to write in the parent directory owned by user gcs. Normally this wouldn't be a problem because only read access to other users' data is required, but it appears as though read access was being denied because the new link file couldn't be written by unprivileged users. Is this a plausible explanation of the I/O error do you think? This sounds like my recent bug: https://bugzilla.redhat.com/show_bug.cgi?id=913699 In the bug, I said that writing on the fuse mount of one of the brick servers fixed the problem but those were the only hosts I was attempting access as root. Thanks Shawn. To confirm that we are seeing the same bug I will try accessing affected files from a FUSE mount on a server the next time it happens. -Dan. -- Dan Bretherton ESSC Computer System Manager Department of Meteorology Harry Pitt Building, 3 Earley Gate University of Reading Reading, RG6 7BE (or RG6 6AL for postal service deliveries) UK Tel. +44 118 378 5205, Fax: +44 118 378 6413 -- ## Please sponsor me to run in VSO's 30km Race to the Eye ## ##http://www.justgiving.com/DanBretherton ## ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] I/O error repaired only by owner or root access
Hi Dan, Could you please provide the following info (1) the exact permissions of the file you are accessing and its parent directory, (2) the user from which 'ls -l' is issued, and (3) the owner of the file, and the parent directory. You could open a bug for it if it is seen several times. Regards, Rajesh Amaravathi, Software Engineer, GlusterFS RedHat Inc. - Original Message - From: Dan Bretherton d.a.brether...@reading.ac.uk To: gluster-users gluster-users@gluster.org Sent: Thursday, February 21, 2013 8:16:40 PM Subject: [Gluster-users] I/O error repaired only by owner or root access Dear All- Several users are having a lot of trouble reading files belonging to other users. Here is an example. [sms05dab@jupiter ~]$ ls -l /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN/Ctl ls: /users/gcs/WORK/ORCA1/ORCA1-R07-MEAN/Ctl: Input/output error The corresponding nfs.log messages are shown below. [2013-02-21 12:11:39.204659] W [nfs3.c:727:nfs3svc_getattr_stat_cbk] 0-nfs: fe2ba5b8: /gorgon/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN = -1 (Invalid argument) [2013-02-21 12:11:39.204778] W [nfs3-helpers.c:3389:nfs3_log_common_res] 0-nfs-nfsv3: XID: fe2ba5b8, GETATTR: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument) [2013-02-21 12:11:39.215345] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-nemo2-dht: deleting stale linkfile /gorgon/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN on nemo2-replicate-0 [2013-02-21 12:11:39.225674] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-nemo2-client-1: remote operation failed: Permission denied [2013-02-21 12:11:39.225786] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-nemo2-client-0: remote operation failed: Permission denied [2013-02-21 12:11:39.681029] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-nemo2-client-18: remote operation failed: Permission denied. Path: /gorgon/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN (1662aa0a-d43b-4c2e-9be9-407eb7a89e85) [2013-02-21 12:11:39.681400] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-nemo2-client-19: remote operation failed: Permission denied. Path: /gorgon/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN (1662aa0a-d43b-4c2e-9be9-407eb7a89e85) [2013-02-21 12:11:39.682268] W [nfs3.c:1627:nfs3svc_readlink_cbk] 0-nfs: 2ca5b8: /gorgon/users/gcs/WORK/ORCA1/ORCA1-R07-MEAN = -1 (Invalid argument) [2013-02-21 12:11:39.682338] W [nfs3-helpers.c:3403:nfs3_log_readlink_res] 0-nfs-nfsv3: XID: 2ca5b8, READLINK: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument), target: (null) I managed to access the same directory as the owner (or any user with write access including root) without any trouble, and after that access from my normal user account was fine as well. The permissions on the directory allowed read access by everyone, but the Permission denied messages in nfs.log indicate that some sort of operation is not being allowed when the directory is accessed by other users. I have seen this happen with files and directories, and with the GlusterFS native client and NFS. I presume this is a bug; I would be grateful if someone could confirm this. I would file a bug report, but the trouble is that I don't know how to reproduce the problem that causes the I/O error in the first place. It only happens with some files and directories, not all. Would a bug report without any way to reproduce the error be any use, and can anyone suggest a way to dig deeper (eg looking at xattrs) next time I come across an example? -Dan. -- Dan Bretherton ESSC Computer System Manager Department of Meteorology Harry Pitt Building, 3 Earley Gate University of Reading Reading, RG6 7BE (or RG6 6AL for postal service deliveries) UK Tel. +44 118 378 5205, Fax: +44 118 378 6413 -- ## Please sponsor me to run in VSO's 30km Race to the Eye ## ##http://www.justgiving.com/DanBretherton ## ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://supercolony.gluster.org/mailman/listinfo/gluster-users
