Thanks for detailed description. Do you have a plans add resolution GFID split-brain by 'gluster volume heal VOLNAME split-brain ...' ? What the main different between GFID split-brain and data split brain? On nodes this file absolutely different by data content and size or it isn't 'data' in glusterfs meaning?
Best regards, Igor 2015-10-14 20:13 GMT+03:00 Ravishankar N <[email protected]>: > > > On 10/14/2015 10:05 PM, Игорь Бирюлин wrote: > > Thanks for your replay. > > If I do listing in mount point (/repo): > # ls /repo/xxx/keyrings/debian-keyring.gpg > ls: cannot access /repo/xxx/keyrings/debian-keyring.gpg: Input/output error > # > In log /var/log/glusterfs/repo.log I see: > [2015-10-14 16:27:36.006815] W [MSGID: 108008] > [afr-self-heal-name.c:359:afr_selfheal_name_gfid_mismatch_check] > 0-repofiles-replicate-0: GFID mismatch for > <gfid:4a99bf9d-7423-47d9-a09d-fabaa333eccf>/debian-keyring.gpg > 69aaeee6-624b-400a-aa46-b5c6166c014c on repofiles-client-1 and > b95ad06e-786a-44e5-ba71-af661982071f on repofiles-client-0 > > > So the file has ended up in GFID split-brain (The trusted.gfid value is > different in both bricks as seen in your output below.), which cannot be > handled by the split-brain resolution commands. These commands can only > resolve data and metadata split-brain. I'm afraid you'll manually need to > delete one of the file and the .glusterfs hardlink from the brick. Not sure > why the parent-directory was not listed in 'gluster v heal VOLNAME info > split-brain' output. > > [2015-10-14 16:27:36.008996] W [fuse-bridge.c:451:fuse_entry_cbk] > 0-glusterfs-fuse: 65961: LOOKUP() /xxx/keyrings/debian-keyring.gpg => -1 > (Input/output error) > > On first node getfattr return: > # getfattr -d -m . -e hex > /storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg > getfattr: Removing leading '/' from absolute path names > # file: storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg > trusted.afr.dirty=0x000000000000000000000000 > trusted.afr.repofiles-client-1=0x000000020000000100000000 > trusted.bit-rot.version=0x020000000000000055fdf0910003b37b > trusted.gfid=0xb95ad06e786a44e5ba71af661982071f > # ls -l /storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg > -rw-r--r-- 2 root root 3456271 Oct 13 19:00 > /storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg > # > > On second node getfattr return: > # getfattr -d -m . -e hex > /storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg > getfattr: Removing leading '/' from absolute path names > # file: storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg > trusted.afr.dirty=0x000000000000000000000000 > trusted.afr.repofiles-client-0=0x000000000000000000000000 > trusted.bit-rot.version=0x020000000000000055f97b57000dc3c6 > trusted.gfid=0x69aaeee6624b400aaa46b5c6166c014c > # ls -l /storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg > -rw-r--r-- 2 root root 3450346 Oct 9 16:22 > /storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg > # > > Best regards, > Igor > > > > > > > 2015-10-14 19:14 GMT+03:00 Ravishankar N <[email protected]>: > >> >> >> On 10/14/2015 07:02 PM, Игорь Бирюлин wrote: >> >> Hello, >> today in my 2 nodes replica set I've found split-brain. Command 'ls' >> start told 'Input/output error'. >> >> >> What does the mount log (/var/log/glusterfs/<path-to-mount>.log) say when >> you get this error? >> >> Can you run getfattr as root for the file from *both* bricks and share >> the result? >> `getfattr -d -m . -e hex >> /storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg` >> >> Thanks. >> Ravi >> >> >> But command 'gluster v heal VOLNAME info split-brain' does not show >> problem files: >> # gluster v heal repofiles info split-brain >> Brick dist-int-master03.xxx:/storage/gluster_brick_repofiles >> Number of entries in split-brain: 0 >> >> Brick dist-int-master04.xxx:/storage/gluster_brick_repofiles >> Number of entries in split-brain: 0 >> # >> In output of 'gluster v heal VOLNAME info' I see problem files >> (/xxx/keyrings/debian-keyring.gpg, /repos.json), but without split-brain >> markers: >> # gluster v heal repofiles info >> Brick dist-int-master03.xxx:/storage/gluster_brick_repofiles >> /xxx/keyrings/debian-keyring.gpg >> <gfid:09ec49c9-911a-4b83-abe8-080fe79e7c69> >> <gfid:35c51b11-a7fb-496d-9e88-6d5a54fda7da> >> /repos.json >> <gfid:4f5cb2b5-30e2-43b0-a935-cfc42af883bf> >> <gfid:9d2fc354-37c0-47a7-b9f3-379504cba797> >> <gfid:cd86a246-9fc4-47d2-bb4d-67566677f77a> >> <gfid:b932eed0-07e9-45c5-943e-7478e9f654b4> >> <gfid:28bf2ffe-948c-4c7d-bce6-966242338581> >> <gfid:ee5659ae-1335-42c5-a852-790387b4213b> >> <gfid:fdfb6b8c-3c04-435a-b8d3-8d8341b66409> >> Number of entries: 11 >> >> Brick dist-int-master04.xxx:/storage/gluster_brick_repofiles >> Number of entries: 0 >> # >> >> I couldn't solve split-brain by new standard command: >> # gluster v heal repofiles split-brain bigger-file /repos.json >> Lookup failed on /repos.json:Input/output error >> Volume heal failed. >> # >> >> Additional info: >> # gluster v info >> Volume Name: repofiles >> Type: Replicate >> Volume ID: 4b0e2a74-f1ca-4fe7-8518-23919e1b5fa0 >> Status: Started >> Number of Bricks: 1 x 2 = 2 >> Transport-type: tcp >> Bricks: >> Brick1: dist-int-master03.xxx:/storage/gluster_brick_repofiles >> Brick2: dist-int-master04.xxx:/storage/gluster_brick_repofiles >> Options Reconfigured: >> performance.readdir-ahead: on >> client.event-threads: 4 >> server.event-threads: 4 >> cluster.lookup-optimize: on >> # cat /etc/issue >> Ubuntu 14.04.3 LTS \n \l >> # dpkg -l | grep glusterfs >> ii glusterfs-client >> 3.7.5-ubuntu1~trusty1 amd64 clustered file-system >> (client package) >> ii glusterfs-common >> 3.7.5-ubuntu1~trusty1 amd64 GlusterFS common >> libraries and translator modules >> ii glusterfs-server >> 3.7.5-ubuntu1~trusty1 amd64 clustered file-system >> (server package) >> # >> >> I have 2 questions: >> 1. Why 'gluster v heal VOLNAME info split-brain' doesn't show actual >> split-brain? Why in 'gluster v heal VOLNAME info' I doesn't see markers >> like 'possible in split-brain'? >> How I can monitor my gluster installation if these commands doesn't show >> problems? >> 2. Why 'gluster volume heal VOLNAME split-brain bigger-file FILE' doesn't >> solve split-brain? I understand that I can solve split-brain remove files >> from brick but I thought to use this killer feature. >> >> Best regards, >> Igor >> >> >> _______________________________________________ >> Gluster-users mailing >> [email protected]http://www.gluster.org/mailman/listinfo/gluster-users >> >> >> > >
_______________________________________________ Gluster-users mailing list [email protected] http://www.gluster.org/mailman/listinfo/gluster-users
