Hi Ravi,
Did you get a chance to have a look at the log files I have attached in my last
mail?
Best,
Mabi
> -------- Original Message --------
> Subject: Re: [Gluster-users] self-heal not working
> Local Time: August 24, 2017 12:08 PM
> UTC Time: August 24, 2017 10:08 AM
> From: [email protected]
> To: Ravishankar N <[email protected]>
> Ben Turner <[email protected]>, Gluster Users <[email protected]>
>
> Thanks for confirming the command. I have now enabled DEBUG client-log-level,
> run a heal and then attached the glustershd log files of all 3 nodes in this
> mail.
>
> The volume concerned is called myvol-pro, the other 3 volumes have no problem
> so far.
>
> Also note that in the mean time it looks like the file has been deleted by
> the user and as such the heal info command does not show the file name
> anymore but just is GFID which is:
>
> gfid:1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea
>
> Hope that helps for debugging this issue.
>
>> -------- Original Message --------
>> Subject: Re: [Gluster-users] self-heal not working
>> Local Time: August 24, 2017 5:58 AM
>> UTC Time: August 24, 2017 3:58 AM
>> From: [email protected]
>> To: mabi <[email protected]>
>> Ben Turner <[email protected]>, Gluster Users <[email protected]>
>>
>> Unlikely. In your case only the afr.dirty is set, not the
>> afr.volname-client-xx xattr.
>>
>> `gluster volume set myvolume diagnostics.client-log-level DEBUG` is right.
>>
>> On 08/23/2017 10:31 PM, mabi wrote:
>>
>>> I just saw the following bug which was fixed in 3.8.15:
>>>
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1471613
>>>
>>> Is it possible that the problem I described in this post is related to that
>>> bug?
>>>
>>>> -------- Original Message --------
>>>> Subject: Re: [Gluster-users] self-heal not working
>>>> Local Time: August 22, 2017 11:51 AM
>>>> UTC Time: August 22, 2017 9:51 AM
>>>> From: [email protected]
>>>> To: mabi [<[email protected]>](mailto:[email protected])
>>>> Ben Turner [<[email protected]>](mailto:[email protected]), Gluster
>>>> Users [<[email protected]>](mailto:[email protected])
>>>>
>>>> On 08/22/2017 02:30 PM, mabi wrote:
>>>>
>>>>> Thanks for the additional hints, I have the following 2 questions first:
>>>>>
>>>>> - In order to launch the index heal is the following command correct:
>>>>> gluster volume heal myvolume
>>>>
>>>> Yes
>>>>
>>>>> - If I run a "volume start force" will it have any short disruptions on
>>>>> my clients which mount the volume through FUSE? If yes, how long? This is
>>>>> a production system that's why I am asking.
>>>>
>>>> No. You can actually create a test volume on your personal linux box to
>>>> try these kinds of things without needing multiple machines. This is how
>>>> we develop and test our patches :)
>>>> 'gluster volume create testvol replica 3 /home/mabi/bricks/brick{1..3}
>>>> force` and so on.
>>>>
>>>> HTH,
>>>> Ravi
>>>>
>>>>>> -------- Original Message --------
>>>>>> Subject: Re: [Gluster-users] self-heal not working
>>>>>> Local Time: August 22, 2017 6:26 AM
>>>>>> UTC Time: August 22, 2017 4:26 AM
>>>>>> From: [email protected]
>>>>>> To: mabi [<[email protected]>](mailto:[email protected]), Ben Turner
>>>>>> [<[email protected]>](mailto:[email protected])
>>>>>> Gluster Users
>>>>>> [<[email protected]>](mailto:[email protected])
>>>>>>
>>>>>> Explore the following:
>>>>>>
>>>>>> - Launch index heal and look at the glustershd logs of all bricks for
>>>>>> possible errors
>>>>>>
>>>>>> - See if the glustershd in each node is connected to all bricks.
>>>>>>
>>>>>> - If not try to restart shd by `volume start force`
>>>>>>
>>>>>> - Launch index heal again and try.
>>>>>>
>>>>>> - Try debugging the shd log by setting client-log-level to DEBUG
>>>>>> temporarily.
>>>>>>
>>>>>> On 08/22/2017 03:19 AM, mabi wrote:
>>>>>>
>>>>>>> Sure, it doesn't look like a split brain based on the output:
>>>>>>>
>>>>>>> Brick node1.domain.tld:/data/myvolume/brick
>>>>>>> Status: Connected
>>>>>>> Number of entries in split-brain: 0
>>>>>>>
>>>>>>> Brick node2.domain.tld:/data/myvolume/brick
>>>>>>> Status: Connected
>>>>>>> Number of entries in split-brain: 0
>>>>>>>
>>>>>>> Brick node3.domain.tld:/srv/glusterfs/myvolume/brick
>>>>>>> Status: Connected
>>>>>>> Number of entries in split-brain: 0
>>>>>>>
>>>>>>>> -------- Original Message --------
>>>>>>>> Subject: Re: [Gluster-users] self-heal not working
>>>>>>>> Local Time: August 21, 2017 11:35 PM
>>>>>>>> UTC Time: August 21, 2017 9:35 PM
>>>>>>>> From: [email protected]
>>>>>>>> To: mabi [<[email protected]>](mailto:[email protected])
>>>>>>>> Gluster Users
>>>>>>>> [<[email protected]>](mailto:[email protected])
>>>>>>>>
>>>>>>>> Can you also provide:
>>>>>>>>
>>>>>>>> gluster v heal <my vol> info split-brain
>>>>>>>>
>>>>>>>> If it is split brain just delete the incorrect file from the brick and
>>>>>>>> run heal again. I haven"t tried this with arbiter but I assume the
>>>>>>>> process is the same.
>>>>>>>>
>>>>>>>> -b
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "mabi" [<[email protected]>](mailto:[email protected])
>>>>>>>>> To: "Ben Turner" [<[email protected]>](mailto:[email protected])
>>>>>>>>> Cc: "Gluster Users"
>>>>>>>>> [<[email protected]>](mailto:[email protected])
>>>>>>>>> Sent: Monday, August 21, 2017 4:55:59 PM
>>>>>>>>> Subject: Re: [Gluster-users] self-heal not working
>>>>>>>>>
>>>>>>>>> Hi Ben,
>>>>>>>>>
>>>>>>>>> So it is really a 0 kBytes file everywhere (all nodes including the
>>>>>>>>> arbiter
>>>>>>>>> and from the client).
>>>>>>>>> Here below you will find the output you requested. Hopefully that
>>>>>>>>> will help
>>>>>>>>> to find out why this specific file is not healing... Let me know if
>>>>>>>>> you need
>>>>>>>>> any more information. Btw node3 is my arbiter node.
>>>>>>>>>
>>>>>>>>> NODE1:
>>>>>>>>>
>>>>>>>>> STAT:
>>>>>>>>> File:
>>>>>>>>> ‘/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png’
>>>>>>>>> Size: 0 Blocks: 38 IO Block: 131072 regular empty file
>>>>>>>>> Device: 24h/36d Inode: 10033884 Links: 2
>>>>>>>>> Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
>>>>>>>>> Access: 2017-08-14 17:04:55.530681000 +0200
>>>>>>>>> Modify: 2017-08-14 17:11:46.407404779 +0200
>>>>>>>>> Change: 2017-08-14 17:11:46.407404779 +0200
>>>>>>>>> Birth: -
>>>>>>>>>
>>>>>>>>> GETFATTR:
>>>>>>>>> trusted.afr.dirty=0sAAAAAQAAAAAAAAAA
>>>>>>>>> trusted.bit-rot.version=0sAgAAAAAAAABZhuknAAlJAg==
>>>>>>>>> trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==
>>>>>>>>> trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOyo=
>>>>>>>>>
>>>>>>>>> NODE2:
>>>>>>>>>
>>>>>>>>> STAT:
>>>>>>>>> File:
>>>>>>>>> ‘/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png’
>>>>>>>>> Size: 0 Blocks: 38 IO Block: 131072 regular empty file
>>>>>>>>> Device: 26h/38d Inode: 10031330 Links: 2
>>>>>>>>> Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
>>>>>>>>> Access: 2017-08-14 17:04:55.530681000 +0200
>>>>>>>>> Modify: 2017-08-14 17:11:46.403704181 +0200
>>>>>>>>> Change: 2017-08-14 17:11:46.403704181 +0200
>>>>>>>>> Birth: -
>>>>>>>>>
>>>>>>>>> GETFATTR:
>>>>>>>>> trusted.afr.dirty=0sAAAAAQAAAAAAAAAA
>>>>>>>>> trusted.bit-rot.version=0sAgAAAAAAAABZhu6wAA8Hpw==
>>>>>>>>> trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==
>>>>>>>>> trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOVE=
>>>>>>>>>
>>>>>>>>> NODE3:
>>>>>>>>> STAT:
>>>>>>>>> File:
>>>>>>>>> /srv/glusterfs/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
>>>>>>>>> Size: 0 Blocks: 0 IO Block: 4096 regular empty file
>>>>>>>>> Device: ca11h/51729d Inode: 405208959 Links: 2
>>>>>>>>> Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
>>>>>>>>> Access: 2017-08-14 17:04:55.530681000 +0200
>>>>>>>>> Modify: 2017-08-14 17:04:55.530681000 +0200
>>>>>>>>> Change: 2017-08-14 17:11:46.604380051 +0200
>>>>>>>>> Birth: -
>>>>>>>>>
>>>>>>>>> GETFATTR:
>>>>>>>>> trusted.afr.dirty=0sAAAAAQAAAAAAAAAA
>>>>>>>>> trusted.bit-rot.version=0sAgAAAAAAAABZe6ejAAKPAg==
>>>>>>>>> trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==
>>>>>>>>> trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOc4=
>>>>>>>>>
>>>>>>>>> CLIENT GLUSTER MOUNT:
>>>>>>>>> STAT:
>>>>>>>>> File:
>>>>>>>>> "/mnt/myvolume/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png"
>>>>>>>>> Size: 0 Blocks: 0 IO Block: 131072 regular empty file
>>>>>>>>> Device: 1eh/30d Inode: 11897049013408443114 Links: 1
>>>>>>>>> Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
>>>>>>>>> Access: 2017-08-14 17:04:55.530681000 +0200
>>>>>>>>> Modify: 2017-08-14 17:11:46.407404779 +0200
>>>>>>>>> Change: 2017-08-14 17:11:46.407404779 +0200
>>>>>>>>> Birth: -
>>>>>>>>>
>>>>>>>>> > -------- Original Message --------
>>>>>>>>> > Subject: Re: [Gluster-users] self-heal not working
>>>>>>>>> > Local Time: August 21, 2017 9:34 PM
>>>>>>>>> > UTC Time: August 21, 2017 7:34 PM
>>>>>>>>> > From: [email protected]
>>>>>>>>> > To: mabi [<[email protected]>](mailto:[email protected])
>>>>>>>>> > Gluster Users
>>>>>>>>> > [<[email protected]>](mailto:[email protected])
>>>>>>>>> >
>>>>>>>>> > ----- Original Message -----
>>>>>>>>> >> From: "mabi" [<[email protected]>](mailto:[email protected])
>>>>>>>>> >> To: "Gluster Users"
>>>>>>>>> >> [<[email protected]>](mailto:[email protected])
>>>>>>>>> >> Sent: Monday, August 21, 2017 9:28:24 AM
>>>>>>>>> >> Subject: [Gluster-users] self-heal not working
>>>>>>>>> >>
>>>>>>>>> >> Hi,
>>>>>>>>> >>
>>>>>>>>> >> I have a replicat 2 with arbiter GlusterFS 3.8.11 cluster and
>>>>>>>>> >> there is
>>>>>>>>> >> currently one file listed to be healed as you can see below but
>>>>>>>>> >> never gets
>>>>>>>>> >> healed by the self-heal daemon:
>>>>>>>>> >>
>>>>>>>>> >> Brick node1.domain.tld:/data/myvolume/brick
>>>>>>>>> >> /data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
>>>>>>>>> >> Status: Connected
>>>>>>>>> >> Number of entries: 1
>>>>>>>>> >>
>>>>>>>>> >> Brick node2.domain.tld:/data/myvolume/brick
>>>>>>>>> >> /data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
>>>>>>>>> >> Status: Connected
>>>>>>>>> >> Number of entries: 1
>>>>>>>>> >>
>>>>>>>>> >> Brick node3.domain.tld:/srv/glusterfs/myvolume/brick
>>>>>>>>> >> /data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
>>>>>>>>> >> Status: Connected
>>>>>>>>> >> Number of entries: 1
>>>>>>>>> >>
>>>>>>>>> >> As once recommended on this mailing list I have mounted that
>>>>>>>>> >> glusterfs
>>>>>>>>> >> volume
>>>>>>>>> >> temporarily through fuse/glusterfs and ran a "stat" on that file
>>>>>>>>> >> which is
>>>>>>>>> >> listed above but nothing happened.
>>>>>>>>> >>
>>>>>>>>> >> The file itself is available on all 3 nodes/bricks but on the last
>>>>>>>>> >> node it
>>>>>>>>> >> has a different date. By the way this file is 0 kBytes big. Is
>>>>>>>>> >> that maybe
>>>>>>>>> >> the reason why the self-heal does not work?
>>>>>>>>> >
>>>>>>>>> > Is the file actually 0 bytes or is it just 0 bytes on the arbiter(0
>>>>>>>>> > bytes
>>>>>>>>> > are expected on the arbiter, it just stores metadata)? Can you send
>>>>>>>>> > us the
>>>>>>>>> > output from stat on all 3 nodes:
>>>>>>>>> >
>>>>>>>>> > $ stat <file on back end brick>
>>>>>>>>> > $ getfattr -d -m - <file on back end brick>
>>>>>>>>> > $ stat <file from gluster mount>
>>>>>>>>> >
>>>>>>>>> > Lets see what things look like on the back end, it should tell us
>>>>>>>>> > why
>>>>>>>>> > healing is failing.
>>>>>>>>> >
>>>>>>>>> > -b
>>>>>>>>> >
>>>>>>>>> >>
>>>>>>>>> >> And how can I now make this file to heal?
>>>>>>>>> >>
>>>>>>>>> >> Thanks,
>>>>>>>>> >> Mabi
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> _______________________________________________
>>>>>>>>> >> Gluster-users mailing list
>>>>>>>>> >> [email protected]
>>>>>>>>> >> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> [email protected]
>>>>>>>
>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users_______________________________________________
Gluster-users mailing list
[email protected]
http://lists.gluster.org/mailman/listinfo/gluster-users
