Re: [Gluster-users] Glusterd proccess hangs on reboot

[Serkan Çoban]

Hi Gaurav,
Any progress about the problem?

On Thursday, August 24, 2017, Serkan Çoban  wrote:

> Thank you Gaurav,
> Here is more findings:
> Problem does not happen using only 20 servers each has 68 bricks.
> (peer probe only 20 servers)
> If we use 40 servers with single volume, glusterd cpu %100 state
> continues for 5 minutes and it goes to normal state.
> with 80 servers we have no working state yet...
>
> On Thu, Aug 24, 2017 at 1:33 PM, Gaurav Yadav  > wrote:
> >
> > I am working on it and will share my findings as soon as possible.
> >
> >
> > Thanks
> > Gaurav
> >
> > On Thu, Aug 24, 2017 at 3:58 PM, Serkan Çoban  > wrote:
> >>
> >> Restarting glusterd causes the same thing. I tried with 3.12.rc0,
> >> 3.10.5. 3.8.15, 3.7.20 all same behavior.
> >> My OS is centos 6.9, I tried with centos 6.8 problem remains...
> >> Only way to a healthy state is destroy gluster config/rpms, reinstall
> >> and recreate volumes.
> >>
> >> On Thu, Aug 24, 2017 at 8:49 AM, Serkan Çoban  >
> >> wrote:
> >> > Here you can find 10 stack trace samples from glusterd. I wait 10
> >> > seconds between each trace.
> >> > https://www.dropbox.com/s/9f36goq5xn3p1yt/glusterd_pstack.zip?dl=0
> >> >
> >> > Content of the first stack trace is here:
> >> >
> >> > Thread 8 (Thread 0x7f7a8cd4e700 (LWP 43069)):
> >> > #0  0x003aa5c0f00d in nanosleep () from /lib64/libpthread.so.0
> >> > #1  0x00303f837d57 in ?? () from /usr/lib64/libglusterfs.so.0
> >> > #2  0x003aa5c07aa1 in start_thread () from /lib64/libpthread.so.0
> >> > #3  0x003aa58e8bbd in clone () from /lib64/libc.so.6
> >> > Thread 7 (Thread 0x7f7a8c34d700 (LWP 43070)):
> >> > #0  0x003aa5c0f585 in sigwait () from /lib64/libpthread.so.0
> >> > #1  0x0040643b in glusterfs_sigwaiter ()
> >> > #2  0x003aa5c07aa1 in start_thread () from /lib64/libpthread.so.0
> >> > #3  0x003aa58e8bbd in clone () from /lib64/libc.so.6
> >> > Thread 6 (Thread 0x7f7a8b94c700 (LWP 43071)):
> >> > #0  0x003aa58acc4d in nanosleep () from /lib64/libc.so.6
> >> > #1  0x003aa58acac0 in sleep () from /lib64/libc.so.6
> >> > #2  0x00303f8528fb in pool_sweeper () from
> >> > /usr/lib64/libglusterfs.so.0
> >> > #3  0x003aa5c07aa1 in start_thread () from /lib64/libpthread.so.0
> >> > #4  0x003aa58e8bbd in clone () from /lib64/libc.so.6
> >> > Thread 5 (Thread 0x7f7a8af4b700 (LWP 43072)):
> >> > #0  0x003aa5c0ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> >> > /lib64/libpthread.so.0
> >> > #1  0x00303f864afc in syncenv_task () from
> >> > /usr/lib64/libglusterfs.so.0
> >> > #2  0x00303f8729f0 in syncenv_processor () from
> >> > /usr/lib64/libglusterfs.so.0
> >> > #3  0x003aa5c07aa1 in start_thread () from /lib64/libpthread.so.0
> >> > #4  0x003aa58e8bbd in clone () from /lib64/libc.so.6
> >> > Thread 4 (Thread 0x7f7a8a54a700 (LWP 43073)):
> >> > #0  0x003aa5c0ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> >> > /lib64/libpthread.so.0
> >> > #1  0x00303f864afc in syncenv_task () from
> >> > /usr/lib64/libglusterfs.so.0
> >> > #2  0x00303f8729f0 in syncenv_processor () from
> >> > /usr/lib64/libglusterfs.so.0
> >> > #3  0x003aa5c07aa1 in start_thread () from /lib64/libpthread.so.0
> >> > #4  0x003aa58e8bbd in clone () from /lib64/libc.so.6
> >> > Thread 3 (Thread 0x7f7a886ac700 (LWP 43075)):
> >> > #0  0x003aa5c0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from
> >> > /lib64/libpthread.so.0
> >> > #1  0x7f7a898a099b in ?? () from
> >> > /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
> >> > #2  0x003aa5c07aa1 in start_thread () from /lib64/libpthread.so.0
> >> > #3  0x003aa58e8bbd in clone () from /lib64/libc.so.6
> >> > Thread 2 (Thread 0x7f7a87cab700 (LWP 43076)):
> >> > #0  0x003aa5928692 in __strcmp_sse42 () from /lib64/libc.so.6
> >> > #1  0x00303f82244a in ?? () from /usr/lib64/libglusterfs.so.0
> >> > #2  0x00303f82433d in ?? () from /usr/lib64/libglusterfs.so.0
> >> > #3  0x00303f8245f5 in dict_set () from
> /usr/lib64/libglusterfs.so.0
> >> > #4  0x00303f82524c in dict_set_str () from
> >> > /usr/lib64/libglusterfs.so.0
> >> > #5  0x7f7a898da7fd in ?? () from
> >> > /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
> >> > #6  0x7f7a8981b0df in ?? () from
> >> > /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
> >> > #7  0x7f7a8981b47c in ?? () from
> >> > /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
> >> > #8  0x7f7a89831edf in ?? () from
> >> > /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
> >> > #9  0x7f7a897f28f7 in ?? () from
> >> > /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
> >> > #10 0x7f7a897f0bb9 in ?? () from
> >> > /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so
> >> > #11 0x7f7a8984c89a in ?? () from
> >> > /usr/lib64/glusterfs/3.10.5/xlator/mgmt/glusterd.so

Re: [Gluster-users] self-heal not working

[Ravishankar N]

On 08/28/2017 01:57 AM, Ben Turner wrote:

- Original Message -

From: "mabi" 
To: "Ravishankar N" 
Cc: "Ben Turner" , "Gluster Users" 

Sent: Sunday, August 27, 2017 3:15:33 PM
Subject: Re: [Gluster-users] self-heal not working

Thanks Ravi for your analysis. So as far as I understand nothing to worry
about but my question now would be: how do I get rid of this file from the
heal info?

Correct me if I am wrong but clearing this is just a matter of resetting the 
afr.dirty xattr?  @Ravi - Is this correct?


Yes resetting the xattr and launching index heal or running heal-info 
command should serve as a workaround.

-Ravi



-b


 Original Message 
Subject: Re: [Gluster-users] self-heal not working
Local Time: August 27, 2017 3:45 PM
UTC Time: August 27, 2017 1:45 PM
From: ravishan...@redhat.com
To: mabi 
Ben Turner , Gluster Users 

Yes, the shds did pick up the file for healing (I saw messages like " got
entry: 1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea") but no error afterwards.

Anyway I reproduced it by manually setting the afr.dirty bit for a zero
byte file on all 3 bricks. Since there are no afr pending xattrs
indicating good/bad copies and all files are zero bytes, the data
self-heal algorithm just picks the file with the latest ctime as source.
In your case that was the arbiter brick. In the code, there is a check to
prevent data heals if arbiter is the source. So heal was not happening and
the entries were not removed from heal-info output.

Perhaps we should add a check in the code to just remove the entries from
heal-info if size is zero bytes in all bricks.

-Ravi

On 08/25/2017 06:33 PM, mabi wrote:


Hi Ravi,

Did you get a chance to have a look at the log files I have attached in my
last mail?

Best,
Mabi


 Original Message 
Subject: Re: [Gluster-users] self-heal not working
Local Time: August 24, 2017 12:08 PM
UTC Time: August 24, 2017 10:08 AM
From: m...@protonmail.ch
To: Ravishankar N
[](mailto:ravishan...@redhat.com)
Ben Turner [](mailto:btur...@redhat.com), Gluster
Users [](mailto:gluster-users@gluster.org)

Thanks for confirming the command. I have now enabled DEBUG
client-log-level, run a heal and then attached the glustershd log files
of all 3 nodes in this mail.

The volume concerned is called myvol-pro, the other 3 volumes have no
problem so far.

Also note that in the mean time it looks like the file has been deleted
by the user and as such the heal info command does not show the file
name anymore but just is GFID which is:

gfid:1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea

Hope that helps for debugging this issue.


 Original Message 
Subject: Re: [Gluster-users] self-heal not working
Local Time: August 24, 2017 5:58 AM
UTC Time: August 24, 2017 3:58 AM
From: ravishan...@redhat.com
To: mabi [](mailto:m...@protonmail.ch)
Ben Turner [](mailto:btur...@redhat.com), Gluster
Users [](mailto:gluster-users@gluster.org)

Unlikely. In your case only the afr.dirty is set, not the
afr.volname-client-xx xattr.

`gluster volume set myvolume diagnostics.client-log-level DEBUG` is
right.

On 08/23/2017 10:31 PM, mabi wrote:


I just saw the following bug which was fixed in 3.8.15:

https://bugzilla.redhat.com/show_bug.cgi?id=1471613

Is it possible that the problem I described in this post is related to
that bug?


 Original Message 
Subject: Re: [Gluster-users] self-heal not working
Local Time: August 22, 2017 11:51 AM
UTC Time: August 22, 2017 9:51 AM
From: ravishan...@redhat.com
To: mabi [](mailto:m...@protonmail.ch)
Ben Turner [](mailto:btur...@redhat.com), Gluster
Users [](mailto:gluster-users@gluster.org)

On 08/22/2017 02:30 PM, mabi wrote:


Thanks for the additional hints, I have the following 2 questions
first:

- In order to launch the index heal is the following command correct:
gluster volume heal myvolume

Yes


- If I run a "volume start force" will it have any short disruptions
on my clients which mount the volume through FUSE? If yes, how long?
This is a production system that's why I am asking.

No. You can actually create a test volume on  your personal linux box
to try these kinds of things without needing multiple machines. This
is how we develop and test our patches :)
'gluster volume create testvol replica 3 /home/mabi/bricks/brick{1..3}
force` and so on.

HTH,
Ravi


 Original Message 
Subject: Re: [Gluster-users] self-heal not working
Local Time: August 22, 2017 6:26 AM
UTC Time: August 22, 2017 4:26 AM
From: ravishan...@redhat.com
To: mabi [](mailto:m...@protonmail.ch), Ben
Turner 

Re: [Gluster-users] self-heal not working

[Ben Turner]

- Original Message -
> From: "mabi" 
> To: "Ravishankar N" 
> Cc: "Ben Turner" , "Gluster Users" 
> 
> Sent: Sunday, August 27, 2017 3:15:33 PM
> Subject: Re: [Gluster-users] self-heal not working
> 
> Thanks Ravi for your analysis. So as far as I understand nothing to worry
> about but my question now would be: how do I get rid of this file from the
> heal info?

Correct me if I am wrong but clearing this is just a matter of resetting the 
afr.dirty xattr?  @Ravi - Is this correct?

-b

> 
> >  Original Message 
> > Subject: Re: [Gluster-users] self-heal not working
> > Local Time: August 27, 2017 3:45 PM
> > UTC Time: August 27, 2017 1:45 PM
> > From: ravishan...@redhat.com
> > To: mabi 
> > Ben Turner , Gluster Users 
> >
> > Yes, the shds did pick up the file for healing (I saw messages like " got
> > entry: 1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea") but no error afterwards.
> >
> > Anyway I reproduced it by manually setting the afr.dirty bit for a zero
> > byte file on all 3 bricks. Since there are no afr pending xattrs
> > indicating good/bad copies and all files are zero bytes, the data
> > self-heal algorithm just picks the file with the latest ctime as source.
> > In your case that was the arbiter brick. In the code, there is a check to
> > prevent data heals if arbiter is the source. So heal was not happening and
> > the entries were not removed from heal-info output.
> >
> > Perhaps we should add a check in the code to just remove the entries from
> > heal-info if size is zero bytes in all bricks.
> >
> > -Ravi
> >
> > On 08/25/2017 06:33 PM, mabi wrote:
> >
> >> Hi Ravi,
> >>
> >> Did you get a chance to have a look at the log files I have attached in my
> >> last mail?
> >>
> >> Best,
> >> Mabi
> >>
> >>>  Original Message 
> >>> Subject: Re: [Gluster-users] self-heal not working
> >>> Local Time: August 24, 2017 12:08 PM
> >>> UTC Time: August 24, 2017 10:08 AM
> >>> From: m...@protonmail.ch
> >>> To: Ravishankar N
> >>> [](mailto:ravishan...@redhat.com)
> >>> Ben Turner [](mailto:btur...@redhat.com), Gluster
> >>> Users [](mailto:gluster-users@gluster.org)
> >>>
> >>> Thanks for confirming the command. I have now enabled DEBUG
> >>> client-log-level, run a heal and then attached the glustershd log files
> >>> of all 3 nodes in this mail.
> >>>
> >>> The volume concerned is called myvol-pro, the other 3 volumes have no
> >>> problem so far.
> >>>
> >>> Also note that in the mean time it looks like the file has been deleted
> >>> by the user and as such the heal info command does not show the file
> >>> name anymore but just is GFID which is:
> >>>
> >>> gfid:1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea
> >>>
> >>> Hope that helps for debugging this issue.
> >>>
>   Original Message 
>  Subject: Re: [Gluster-users] self-heal not working
>  Local Time: August 24, 2017 5:58 AM
>  UTC Time: August 24, 2017 3:58 AM
>  From: ravishan...@redhat.com
>  To: mabi [](mailto:m...@protonmail.ch)
>  Ben Turner [](mailto:btur...@redhat.com), Gluster
>  Users [](mailto:gluster-users@gluster.org)
> 
>  Unlikely. In your case only the afr.dirty is set, not the
>  afr.volname-client-xx xattr.
> 
>  `gluster volume set myvolume diagnostics.client-log-level DEBUG` is
>  right.
> 
>  On 08/23/2017 10:31 PM, mabi wrote:
> 
> > I just saw the following bug which was fixed in 3.8.15:
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=1471613
> >
> > Is it possible that the problem I described in this post is related to
> > that bug?
> >
> >>  Original Message 
> >> Subject: Re: [Gluster-users] self-heal not working
> >> Local Time: August 22, 2017 11:51 AM
> >> UTC Time: August 22, 2017 9:51 AM
> >> From: ravishan...@redhat.com
> >> To: mabi [](mailto:m...@protonmail.ch)
> >> Ben Turner [](mailto:btur...@redhat.com), Gluster
> >> Users [](mailto:gluster-users@gluster.org)
> >>
> >> On 08/22/2017 02:30 PM, mabi wrote:
> >>
> >>> Thanks for the additional hints, I have the following 2 questions
> >>> first:
> >>>
> >>> - In order to launch the index heal is the following command correct:
> >>> gluster volume heal myvolume
> >>
> >> Yes
> >>
> >>> - If I run a "volume start force" will it have any short disruptions
> >>> on my clients which mount the volume through FUSE? If yes, how long?
> >>> This is a production system that's why I am asking.
> >>
> >> No. You can actually create a test volume on  your personal 

Re: [Gluster-users] self-heal not working

[mabi]

Thanks Ravi for your analysis. So as far as I understand nothing to worry about 
but my question now would be: how do I get rid of this file from the heal info?

>  Original Message 
> Subject: Re: [Gluster-users] self-heal not working
> Local Time: August 27, 2017 3:45 PM
> UTC Time: August 27, 2017 1:45 PM
> From: ravishan...@redhat.com
> To: mabi 
> Ben Turner , Gluster Users 
>
> Yes, the shds did pick up the file for healing (I saw messages like " got 
> entry: 1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea") but no error afterwards.
>
> Anyway I reproduced it by manually setting the afr.dirty bit for a zero byte 
> file on all 3 bricks. Since there are no afr pending xattrs indicating 
> good/bad copies and all files are zero bytes, the data self-heal algorithm 
> just picks the file with the latest ctime as source. In your case that was 
> the arbiter brick. In the code, there is a check to prevent data heals if 
> arbiter is the source. So heal was not happening and the entries were not 
> removed from heal-info output.
>
> Perhaps we should add a check in the code to just remove the entries from 
> heal-info if size is zero bytes in all bricks.
>
> -Ravi
>
> On 08/25/2017 06:33 PM, mabi wrote:
>
>> Hi Ravi,
>>
>> Did you get a chance to have a look at the log files I have attached in my 
>> last mail?
>>
>> Best,
>> Mabi
>>
>>>  Original Message 
>>> Subject: Re: [Gluster-users] self-heal not working
>>> Local Time: August 24, 2017 12:08 PM
>>> UTC Time: August 24, 2017 10:08 AM
>>> From: m...@protonmail.ch
>>> To: Ravishankar N [](mailto:ravishan...@redhat.com)
>>> Ben Turner [](mailto:btur...@redhat.com), Gluster Users 
>>> [](mailto:gluster-users@gluster.org)
>>>
>>> Thanks for confirming the command. I have now enabled DEBUG 
>>> client-log-level, run a heal and then attached the glustershd log files of 
>>> all 3 nodes in this mail.
>>>
>>> The volume concerned is called myvol-pro, the other 3 volumes have no 
>>> problem so far.
>>>
>>> Also note that in the mean time it looks like the file has been deleted by 
>>> the user and as such the heal info command does not show the file name 
>>> anymore but just is GFID which is:
>>>
>>> gfid:1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea
>>>
>>> Hope that helps for debugging this issue.
>>>
  Original Message 
 Subject: Re: [Gluster-users] self-heal not working
 Local Time: August 24, 2017 5:58 AM
 UTC Time: August 24, 2017 3:58 AM
 From: ravishan...@redhat.com
 To: mabi [](mailto:m...@protonmail.ch)
 Ben Turner [](mailto:btur...@redhat.com), Gluster 
 Users [](mailto:gluster-users@gluster.org)

 Unlikely. In your case only the afr.dirty is set, not the 
 afr.volname-client-xx xattr.

 `gluster volume set myvolume diagnostics.client-log-level DEBUG` is right.

 On 08/23/2017 10:31 PM, mabi wrote:

> I just saw the following bug which was fixed in 3.8.15:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1471613
>
> Is it possible that the problem I described in this post is related to 
> that bug?
>
>>  Original Message 
>> Subject: Re: [Gluster-users] self-heal not working
>> Local Time: August 22, 2017 11:51 AM
>> UTC Time: August 22, 2017 9:51 AM
>> From: ravishan...@redhat.com
>> To: mabi [](mailto:m...@protonmail.ch)
>> Ben Turner [](mailto:btur...@redhat.com), Gluster 
>> Users [](mailto:gluster-users@gluster.org)
>>
>> On 08/22/2017 02:30 PM, mabi wrote:
>>
>>> Thanks for the additional hints, I have the following 2 questions first:
>>>
>>> - In order to launch the index heal is the following command correct:
>>> gluster volume heal myvolume
>>
>> Yes
>>
>>> - If I run a "volume start force" will it have any short disruptions on 
>>> my clients which mount the volume through FUSE? If yes, how long? This 
>>> is a production system that's why I am asking.
>>
>> No. You can actually create a test volume on  your personal linux box to 
>> try these kinds of things without needing multiple machines. This is how 
>> we develop and test our patches :)
>> 'gluster volume create testvol replica 3 /home/mabi/bricks/brick{1..3} 
>> force` and so on.
>>
>> HTH,
>> Ravi
>>
  Original Message 
 Subject: Re: [Gluster-users] self-heal not working
 Local Time: August 22, 2017 6:26 AM
 UTC Time: August 22, 2017 4:26 AM
 From: ravishan...@redhat.com
 To: mabi [](mailto:m...@protonmail.ch), Ben Turner 
 

Re: [Gluster-users] self-heal not working

[Ravishankar N]

Yes, the shds did pick up the file for healing (I saw messages like " 
got entry: 1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea") but no error afterwards.


Anyway I reproduced it by manually setting the afr.dirty bit for a zero 
byte file on all 3 bricks. Since there are no afr pending xattrs 
indicating good/bad copies and all files are zero bytes, the data 
self-heal algorithm just picks the file with the latest ctime as source. 
In your case that was the arbiter brick. In the code, there is a check 
to prevent data heals if arbiter is the source. So heal was not 
happening and the entries were not removed from heal-info output.


Perhaps we should add a check in the code to just remove the entries 
from heal-info if size is zero bytes in all bricks.


-Ravi


On 08/25/2017 06:33 PM, mabi wrote:

Hi Ravi,

Did you get a chance to have a look at the log files I have attached 
in my last mail?


Best,
Mabi




 Original Message 
Subject: Re: [Gluster-users] self-heal not working
Local Time: August 24, 2017 12:08 PM
UTC Time: August 24, 2017 10:08 AM
From: m...@protonmail.ch
To: Ravishankar N 
Ben Turner , Gluster Users 



Thanks for confirming the command. I have now enabled DEBUG 
client-log-level, run a heal and then attached the glustershd log 
files of all 3 nodes in this mail.


The volume concerned is called myvol-pro, the other 3 volumes have no 
problem so far.


Also note that in the mean time it looks like the file has been 
deleted by the user and as such the heal info command does not show 
the file name anymore but just is GFID which is:


gfid:1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea



Hope that helps for debugging this issue.


 Original Message 
Subject: Re: [Gluster-users] self-heal not working
Local Time: August 24, 2017 5:58 AM
UTC Time: August 24, 2017 3:58 AM
From: ravishan...@redhat.com
To: mabi 
Ben Turner , Gluster Users 




Unlikely. In your case only the afr.dirty is set, not the 
afr.volname-client-xx xattr.


`gluster volume set myvolume diagnostics.client-log-level DEBUG` is 
right.



On 08/23/2017 10:31 PM, mabi wrote:

I just saw the following bug which was fixed in 3.8.15:

https://bugzilla.redhat.com/show_bug.cgi?id=1471613

Is it possible that the problem I described in this post is related 
to that bug?





 Original Message 
Subject: Re: [Gluster-users] self-heal not working
Local Time: August 22, 2017 11:51 AM
UTC Time: August 22, 2017 9:51 AM
From: ravishan...@redhat.com
To: mabi 
Ben Turner , Gluster Users 






On 08/22/2017 02:30 PM, mabi wrote:
Thanks for the additional hints, I have the following 2 questions 
first:


- In order to launch the index heal is the following command correct:
gluster volume heal myvolume


Yes

- If I run a "volume start force" will it have any short 
disruptions on my clients which mount the volume through FUSE? If 
yes, how long? This is a production system that's why I am asking.



No. You can actually create a test volume on  your personal linux 
box to try these kinds of things without needing multiple 
machines. This is how we develop and test our patches :)
'gluster volume create testvol replica 3 
/home/mabi/bricks/brick{1..3} force` and so on.


HTH,
Ravi





 Original Message 
Subject: Re: [Gluster-users] self-heal not working
Local Time: August 22, 2017 6:26 AM
UTC Time: August 22, 2017 4:26 AM
From: ravishan...@redhat.com
To: mabi , Ben Turner 
Gluster Users 


Explore the following:

- Launch index heal and look at the glustershd logs of all 
bricks for possible errors


- See if the glustershd in each node is connected to all bricks.

- If not try to restart shd by `volume start force`

- Launch index heal again and try.

- Try debugging the shd log by setting client-log-level to DEBUG 
temporarily.



On 08/22/2017 03:19 AM, mabi wrote:

Sure, it doesn't look like a split brain based on the output:

Brick node1.domain.tld:/data/myvolume/brick
Status: Connected
Number of entries in split-brain: 0

Brick node2.domain.tld:/data/myvolume/brick
Status: Connected
Number of entries in split-brain: 0

Brick node3.domain.tld:/srv/glusterfs/myvolume/brick
Status: Connected
Number of entries in split-brain: 0





 Original Message 
Subject: Re: [Gluster-users] self-heal not working
Local Time: August 21, 2017 11:35 PM
UTC Time: August 21, 2017 9:35 PM
From: btur...@redhat.com
To: mabi 
Gluster Users 

Can you also provide:

gluster v heal  info split-brain

If it is split brain just delete the incorrect file from the 
brick and run heal again. I haven"t tried this with arbiter 
but I assume the process is the same.


-b

- Original Message 

[Gluster-users] Using glusterfind to backup to rsync.net?

[Niels de Vos]

Hi Milind,

I am thinking of doing backups from my Gluster volume at home. Would
glusterfind make that straight forward to do? My plan would be to have a
cron-job that runs every night and copies documents and pictures to
rsync.net. For me it is not a lot of data, and I think a plain recursive
scanning rsync might work sufficiently, but hooking it up with
glusterfind would be more optimized.

Can you point me to the documentation or an example that show which
documents have been modified and which pictures have been added? Basing
it on filenames would be ideal of course, it would be annoying to have
to resolve GFIDs.

Thanks,
Niels
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users