Hi Anuradha, I am doing the same which is mentioned in the link you shared and It has been resolved the issue.
But my question is if it is the split-brain scenario then why the command "gluster volume heal info split-brain" not showing these files in the output even not the parent directory is present in split-brain. Please find the requested logs On Wed, Mar 16, 2016 at 4:20 PM, Anuradha Talur <[email protected]> wrote: > Hi Abhishek, > > The files that are reporting i/o error have gfid-mismatch. This situation > is called directory or entry split-brain. You can find steps to resolve > this kind of split brain here : > https://gluster.readthedocs.org/en/latest/Troubleshooting/split-brain/ . > > Ideally, the parent directories of these files have to be listed in heal > info split-brain output. Can you please get extended attributes of parent > directories of the files that show i/o error (Same getfattr command that > you previously used.) ? > > ----- Original Message ----- > > From: "ABHISHEK PALIWAL" <[email protected]> > > To: "Anuradha Talur" <[email protected]> > > Cc: [email protected], [email protected] > > Sent: Thursday, March 10, 2016 11:22:35 AM > > Subject: Re: [Gluster-users] gluster volume heal info split brain > command not showing files in split-brain > > > > Hi Anuradha, > > > > Please find the glusterfs and glusterd logs directory as an attachment. > > > > Regards, > > Abhishek > > > > > > > > On Wed, Mar 9, 2016 at 5:54 PM, ABHISHEK PALIWAL < > [email protected]> > > wrote: > > > > > Hi Anuradha, > > > > > > Sorry for late reply. > > > > > > Please find the requested logs below: > > > > > > Remote: 10.32.0.48 > > > Local : 10.32.1.144 > > > > > > Local: > > > #gluster volume heal c_glusterfs info split-brain > > > Brick 10.32.1.144:/opt/lvmdir/c2/brick > > > Number of entries in split-brain: 0 > > > > > > Brick 10.32.0.48:/opt/lvmdir/c2/brick > > > Number of entries in split-brain: 0 > > > > > > Remote: > > > #gluster volume heal c_glusterfs info split-brain > > > Brick 10.32.1.144:/opt/lvmdir/c2/brick > > > Number of entries in split-brain: 0 > > > > > > Brick 10.32.0.48:/opt/lvmdir/c2/brick > > > Number of entries in split-brain: 0 > > > > > > auto-sync.sh. > > > Here you can see that i/o error is detected. Below is the required meta > > > data from both the bricks. > > > > > > 1) > > > stat: cannot stat '/mnt/c//public_html/cello/ior_files/nameroot.ior': > > > Input/output error > > > Remote: > > > > > > getfattr -d -m . -e hex > > > opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior > > > # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior > > > trusted.afr.dirty=0x000000000000000000000000 > > > trusted.bit-rot.version=0x000000000000000256ded2f6000ad80f > > > trusted.gfid=0x771221a7bb3c4f1aade40ce9e38a95ee > > > > > > Local: > > > > > > getfattr -d -m . -e hex > > > opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior > > > # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior > > > trusted.bit-rot.version=0x000000000000000256ded38f000e3a51 > > > trusted.gfid=0x8ea33f46703c4e2d95c09153c1b858fd > > > > > > > > > 2) > > > stat: cannot stat '/mnt/c//security/corbasecurity': Input/output error > > > Remote: > > > > > > getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity > > > # file: opt/lvmdir/c2/brick/security/corbasecurity > > > trusted.afr.dirty=0x000000000000000000000000 > > > trusted.bit-rot.version=0x000000000000000256ded2f6000ad80f > > > trusted.gfid=0xd298b7a0c8834f3e99abb39741363013 > > > > > > Local: > > > > > > getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity > > > # file: opt/lvmdir/c2/brick/security/corbasecurity > > > trusted.bit-rot.version=0x000000000000000256ded38f000e3a51 > > > trusted.gfid=0x890df0f706184b52803fac3242a2f15b > > > > > > I observed that getfattr command output doesn't show all the fields all > > > the times. > > > > > > Here you can check that gluster split-brain command hasn't reported any > > > split-brains but resulted in IO errors when accessed few files. > > > > > > Could you please tell me if "split-brain" command doesn't reported any > > > entry as output, then is there any way through which we can find out > that > > > the files are in split-brain if we are getting the IO error on those > file. > > > > > > > > > Regards, > > > Abhishek > > > > > > > > > > > > On Thu, Mar 3, 2016 at 5:32 PM, Anuradha Talur <[email protected]> > wrote: > > > > > >> > > >> > > >> ----- Original Message ----- > > >> > From: "ABHISHEK PALIWAL" <[email protected]> > > >> > To: [email protected], [email protected] > > >> > Sent: Thursday, March 3, 2016 12:10:42 PM > > >> > Subject: [Gluster-users] gluster volume heal info split brain > command > > >> not showing files in split-brain > > >> > > > >> > > > >> > Hello, > > >> > > > >> > In gluster, we use the command "gluster volume heal c_glusterfs info > > >> > split-brain" to find the files that are in split-brain scenario. > > >> > We run heal script (developed by Windriver prime team) on the files > > >> reported > > >> > by above command to resolve split-brain issue. > > >> > > > >> > But we observed that the above command is not showing all files that > > >> are in > > >> > split-brain, > > >> > even though split brain scenario actually exists on the node. > > >> > > > >> > Now a days this issue is seen more often and IO errors are reported > when > > >> > tried to access these files under split-brain. > > >> > > > >> > Can you please check why this gluster command is not showing files > under > > >> > split-brain? > > >> > We can provide you required logs and support to resolve this issue. > > >> Hi, > > >> > > >> Could you paste the output of getfattr -m. -de hex > > >> <path-to-files-in-split-brain> from all the bricks that the files lie > in? > > >> > > >> > > > >> > Please reply on this because I am not getting any reply from the > > >> community. > > >> > > > >> > -- > > >> > > > >> > Regards > > >> > Abhishek Paliwal > > >> > > > >> > _______________________________________________ > > >> > Gluster-users mailing list > > >> > [email protected] > > >> > http://www.gluster.org/mailman/listinfo/gluster-users > > >> > > >> -- > > >> Thanks, > > >> Anuradha. > > >> > > > > > > > -- > Thanks, > Anuradha. > -- Regards Abhishek Paliwal
1.File Name : nameroot.ior stat: cannot stat '/mnt/c//public_html/cello/ior_files/nameroot.ior': Input/output error Remote File: getfattr -d -m . -e hex opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior trusted.afr.dirty=0x000000000000000000000000 trusted.bit-rot.version=0x000000000000000256e812da0007bf13 trusted.gfid=0xc18f775d94de42879235d1331d85c860 Remote Dir: getfattr -d -m . -e hex opt/lvmdir/c2/brick/public_html/cello/ior_files # file: opt/lvmdir/c2/brick/public_html/cello/ior_files trusted.afr.c_glusterfs-client-1=0x000000000000000000000000 trusted.afr.c_glusterfs-client-207=0x000000000000000000000002 trusted.afr.c_glusterfs-client-209=0x000000000000000000000000 trusted.afr.c_glusterfs-client-215=0x000000000000000000000000 trusted.afr.c_glusterfs-client-39=0x000000000000000000000000 trusted.afr.c_glusterfs-client-47=0x000000000000000000000000 trusted.afr.c_glusterfs-client-49=0x000000000000000000000002 trusted.afr.c_glusterfs-client-51=0x000000000000000000000000 trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0xd9cd3be03fa44d1e8a8da8523535ef0a trusted.glusterfs.dht=0x000000010000000000000000ffffffff stat: cannot stat '/mnt/c//public_html/cello/ior_files/nameroot.ior': Input/output error Local File: getfattr -d -m . -e hex opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior trusted.bit-rot.version=0x000000000000000256e813c50008b4e2 trusted.gfid=0x32145e0378864767989335f37c108409 Local Dir: getfattr -d -m . -e hex opt/lvmdir/c2/brick/public_html/cello/ior_files # file: opt/lvmdir/c2/brick/public_html/cello/ior_files trusted.afr.c_glusterfs-client-112=0x000000000000000000000000 trusted.afr.c_glusterfs-client-116=0x000000000000000000000000 trusted.afr.c_glusterfs-client-128=0x000000000000000000000000 trusted.afr.c_glusterfs-client-130=0x000000000000000000000000 trusted.afr.c_glusterfs-client-150=0x000000000000000000000000 trusted.afr.c_glusterfs-client-164=0x000000000000000000000000 trusted.afr.c_glusterfs-client-166=0x000000000000000000000000 trusted.afr.c_glusterfs-client-194=0x000000000000000000000000 trusted.afr.c_glusterfs-client-196=0x000000000000000000000000 trusted.afr.c_glusterfs-client-200=0x000000000000000000000000 trusted.afr.c_glusterfs-client-224=0x000000000000000000000000 trusted.afr.c_glusterfs-client-26=0x000000000000000000000000 trusted.afr.c_glusterfs-client-36=0x000000000000000000000000 trusted.afr.c_glusterfs-client-38=0x000000000000000000000000 trusted.afr.c_glusterfs-client-40=0x000000000000000000000000 trusted.afr.c_glusterfs-client-50=0x000000000000000000000000 trusted.afr.c_glusterfs-client-54=0x000000000000000000000000 trusted.afr.c_glusterfs-client-58=0x000000000000000000000002 trusted.afr.c_glusterfs-client-64=0x000000000000000000000000 trusted.afr.c_glusterfs-client-66=0x000000000000000000000000 trusted.afr.c_glusterfs-client-70=0x000000000000000000000000 trusted.afr.c_glusterfs-client-76=0x000000000000000000000000 trusted.afr.c_glusterfs-client-84=0x000000000000000000000000 trusted.afr.c_glusterfs-client-90=0x000000000000000000000000 trusted.afr.c_glusterfs-client-98=0x000000000000000000000000 trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0xd9cd3be03fa44d1e8a8da8523535ef0a trusted.glusterfs.dht=0x000000010000000000000000ffffffff ********************************************************************* 2.File Name : corbasecurity stat: cannot stat '/mnt/c//security/corbasecurity': Input/output error Remote File: getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity # file: opt/lvmdir/c2/brick/security/corbasecurity trusted.afr.dirty=0x000000000000000000000000 trusted.bit-rot.version=0x000000000000000256e812da0007bf13 trusted.gfid=0xe477a482ed354795a83b00281d85ef69 Remote Dir: getfattr -d -m . -e hex opt/lvmdir/c2/brick/security # file: opt/lvmdir/c2/brick/security trusted.afr.c_glusterfs-client-1=0x000000000000000000000000 trusted.afr.c_glusterfs-client-207=0x000000000000000000000002 trusted.afr.c_glusterfs-client-209=0x000000000000000000000000 trusted.afr.c_glusterfs-client-215=0x000000000000000000000000 trusted.afr.c_glusterfs-client-39=0x000000000000000000000000 trusted.afr.c_glusterfs-client-47=0x000000000000000000000000 trusted.afr.c_glusterfs-client-49=0x000000000000000000000000 trusted.afr.c_glusterfs-client-51=0x000000000000000000000000 trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0x69eee88cddb64c27b99048123191b112 trusted.glusterfs.dht=0x000000010000000000000000ffffffff stat: cannot stat '/mnt/c//security/corbasecurity': Input/output error Local File: getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity # file: opt/lvmdir/c2/brick/security/corbasecurity trusted.bit-rot.version=0x000000000000000256e813c50008b4e2 trusted.gfid=0x23400a9240ce422a99e24a09d7da8f6c Local Dir: getfattr -d -m . -e hex opt/lvmdir/c2/brick/security # file: opt/lvmdir/c2/brick/security trusted.afr.c_glusterfs-client-112=0x000000000000000000000000 trusted.afr.c_glusterfs-client-116=0x000000000000000000000000 trusted.afr.c_glusterfs-client-128=0x000000000000000000000000 trusted.afr.c_glusterfs-client-130=0x000000000000000000000000 trusted.afr.c_glusterfs-client-150=0x000000000000000000000000 trusted.afr.c_glusterfs-client-164=0x000000000000000000000000 trusted.afr.c_glusterfs-client-166=0x000000000000000000000000 trusted.afr.c_glusterfs-client-194=0x000000000000000000000000 trusted.afr.c_glusterfs-client-196=0x000000000000000000000000 trusted.afr.c_glusterfs-client-200=0x000000000000000000000000 trusted.afr.c_glusterfs-client-224=0x000000000000000000000000 trusted.afr.c_glusterfs-client-26=0x000000000000000000000000 trusted.afr.c_glusterfs-client-36=0x000000000000000000000000 trusted.afr.c_glusterfs-client-38=0x000000000000000000000000 trusted.afr.c_glusterfs-client-40=0x000000000000000000000000 trusted.afr.c_glusterfs-client-50=0x000000000000000000000000 trusted.afr.c_glusterfs-client-54=0x000000000000000000000000 trusted.afr.c_glusterfs-client-58=0x000000000000000000000002 trusted.afr.c_glusterfs-client-64=0x000000000000000000000000 trusted.afr.c_glusterfs-client-66=0x000000000000000000000000 trusted.afr.c_glusterfs-client-70=0x000000000000000000000000 trusted.afr.c_glusterfs-client-76=0x000000000000000000000000 trusted.afr.c_glusterfs-client-84=0x000000000000000000000000 trusted.afr.c_glusterfs-client-90=0x000000000000000000000000 trusted.afr.c_glusterfs-client-98=0x000000000000000000000000 trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0x69eee88cddb64c27b99048123191b112 trusted.glusterfs.dht=0x000000010000000000000000ffffffff
_______________________________________________ Gluster-devel mailing list [email protected] http://www.gluster.org/mailman/listinfo/gluster-devel
