Re: [Gluster-users] split-brain errors under heavy load when one brick down

[Erik Jacobson]

Thank you for replying!

> Okay so 0-cm_shared-replicate-1 means these 3 bricks:
> 
> Brick4: 172.23.0.6:/data/brick_cm_shared
> Brick5: 172.23.0.7:/data/brick_cm_shared
> Brick6: 172.23.0.8:/data/brick_cm_shared

The above is correct.


> Were there any pending self-heals for this volume? Is it possible that the
> server (one of Brick 4, 5 or 6 ) that is down had the only good copy and the
> other 2 online bricks had a bad copy (needing heal)? Clients can get EIO in
> that case.

So I did check for heals and saw nothing. The storage at this time was in a
read-only use case. What I mean by that is the NFS clients mount it read only
and there were no write activities going to shared storage anyway at that
time.  So it was not surprising that no heals were listed.

I did inspect both remaining bricks for several of the example problem files
and found them with matching md5sums.

The strange thing, as I mentioned, is it only happened under the job
launch workload. The nfs boot workload, which is also very stressful,
ran clean with one brick down.

> When you say accessing the file from the compute nodes afterwards works
> fine, it is still with that one server (brick) down?

I can no longer check this system personally but as I recall when we
fixed the ethernet problem, all seemed well. I don't have a better
answer for this one than that. I am starting a document of things to try
when we have a large system in the factory to run on. I'll put this in
there.

> 
> There was a case of AFR reporting spurious split-brain errors but that was
> fixed long back (http://review.gluster.org/16362
> ) and seems to be present in glusterf-4.1.6.


So I brought this up. In my case, we know the files on the NFS client
side really were missing because we saw errors on the clients. That is
to say, the above bug seems to mean that split-brain was reported in
error with no other impacts. However, in my case, the error resulted in
actual problems accessing the files on the NFS clients.

> Side note: Why are you using replica 9 for the ctdb volume? All
> development/tests are usually done on (distributed) replica 3 setup.

I am happy to change this. Whatever guide I used to set this up
suggested replica 9. I don't even know which resource was incorrect as
it was so long ago. I have no other reason.

I'm filing an incident now to change our setup tools to use replica-3 for
CTDB for new setups.

Again, I appreciate that you followed up with me. Thank you,

Erik


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain errors under heavy load when one brick down

[Ravishankar N]

On 16/09/19 7:34 pm, Erik Jacobson wrote:

Example errors:

ex1

[2019-09-06 18:26:42.665050] E [MSGID: 108008]
[afr-read-txn.c:123:afr_read_txn_refresh_done] 0-cm_shared-replicate-1: Failing
ACCESS on gfid ee3f5646-9368-4151-92a3-5b8e7db1fbf9: split-brain observed.
[Input/output error]

Okay so 0-cm_shared-replicate-1 means these 3 bricks:

Brick4: 172.23.0.6:/data/brick_cm_shared
Brick5: 172.23.0.7:/data/brick_cm_shared
Brick6: 172.23.0.8:/data/brick_cm_shared



ex2

[2019-09-06 18:26:55.359272] E [MSGID: 108008]
[afr-read-txn.c:123:afr_read_txn_refresh_done] 0-cm_shared-replicate-1: Failing
READLINK on gfid f2be38c2-1cd1-486b-acad-17f2321a18b3: split-brain observed.
[Input/output error]
[2019-09-06 18:26:55.359367] W [MSGID: 112199]
[nfs3-helpers.c:3435:nfs3_log_readlink_res] 0-nfs-nfsv3:
/image/images_ro_nfs/toss-20190730/usr/lib64/libslurm.so.32 => (XID: 88651c80,
READLINK: NFS: 5(I/O error), POSIX: 5(Input/output error)) target: (null)



The errors seem to happen only on the 'replicate' volume where one
server is down in the subvolume (of course, any NFS server will
trigger that when it accesses the files on the degraded volume).


Were there any pending self-heals for this volume? Is it possible that 
the server (one of Brick 4, 5 or 6 ) that is down had the only good copy 
and the other 2 online bricks had a bad copy (needing heal)? Clients can 
get EIO in that case.


When you say accessing the file from the compute nodes afterwards works 
fine, it is still with that one server (brick) down?


There was a case of AFR reporting spurious split-brain errors but that 
was fixed long back (http://review.gluster.org/16362) and seems to be 
present in glusterf-4.1.6.


Side note: Why are you using replica 9 for the ctdb volume? All 
development/tests are usually done on (distributed) replica 3 setup.


Thanks,

Ravi



Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split brain? but where?

[Thing]

Next I ran a test and your find worksI am wondering if I can simply
delete this GFID?

8><-
[root@glusterp2 fb]# find /bricks/brick1/gv0/ -samefile
/bricks/brick1/gv0/.glusterfs/74/75/7475bd15-05a6-45c2-b395-bc9fd3d1763f
/bricks/brick1/gv0/.glusterfs/74/75/7475bd15-05a6-45c2-b395-bc9fd3d1763f
/bricks/brick1/gv0/dell6430-001/virtual-machines-backups/21-3-2018/images/debian9-002.qcow2
[root@glusterp2 fb]#
8><-

On another node also no return,

8><
[root@glusterp1 ~]# find /bricks/brick1/gv0 -samefile
/bricks/brick1/gv0/.glusterfs/ea/fb/eafb8799-4e7a-4264-9213-26997c5a4693
/bricks/brick1/gv0/.glusterfs/ea/fb/eafb8799-4e7a-4264-9213-26997c5a4693
[root@glusterp1 ~]#
8><

and on the final node,
8><---
[root@glusterp3 ~]# find /bricks/brick1/gv0 -samefile
/bricks/brick1/gv0/.glusterfs/ea/fb/eafb8799-4e7a-4264-9213-26997c5a4693
/bricks/brick1/gv0/.glusterfs/ea/fb/eafb8799-4e7a-4264-9213-26997c5a4693
[root@glusterp3 ~]#
8><---

So none of the 3 nodes has an actual file?



On 23 May 2018 at 08:35, Thing  wrote:

> I tried looking for a file of the same size and the gfid doesnt show up,
>
> 8><---
> [root@glusterp2 fb]# pwd
> /bricks/brick1/gv0/.glusterfs/ea/fb
> [root@glusterp2 fb]# ls -al
> total 3130892
> drwx--. 2 root root 64 May 22 13:01 .
> drwx--. 4 root root 24 May  8 14:27 ..
> -rw---. 1 root root 3294887936 May  4 11:07 eafb8799-4e7a-4264-9213-
> 26997c5a4693
> -rw-r--r--. 1 root root   1396 May 22 13:01 gfid.run
>
> so the gfid seems largebut du cant see it...
>
> [root@glusterp2 fb]# du -a /bricks/brick1/gv0 | sort -n -r | head -n 10
> 275411712 /bricks/brick1/gv0
> 275411696 /bricks/brick1/gv0/.glusterfs
> 22484988 /bricks/brick1/gv0/.glusterfs/57
> 20974980 /bricks/brick1/gv0/.glusterfs/a5
> 20974976 /bricks/brick1/gv0/.glusterfs/d5/99/d5991797-848d-4d60-b9dc-
> d31174f63f72
> 20974976 /bricks/brick1/gv0/.glusterfs/d5/99
> 20974976 /bricks/brick1/gv0/.glusterfs/d5
> 20974976 /bricks/brick1/gv0/.glusterfs/a5/27/a5279083-4d24-4dc8-be2d-
> 4f507c5372cf
> 20974976 /bricks/brick1/gv0/.glusterfs/a5/27
> 20974976 /bricks/brick1/gv0/.glusterfs/74/75/7475bd15-05a6-45c2-b395-
> bc9fd3d1763f
> [root@glusterp2 fb]#
>
> 8><---
>
>
>
> On 23 May 2018 at 08:29, Thing  wrote:
>
>> I tried this already.
>>
>> 8><---
>> [root@glusterp2 fb]# find /bricks/brick1/gv0 -samefile
>> /bricks/brick1/gv0/.glusterfs/ea/fb/eafb8799-4e7a-4264-9213-26997c5a4693
>> /bricks/brick1/gv0/.glusterfs/ea/fb/eafb8799-4e7a-4264-9213-26997c5a4693
>> [root@glusterp2 fb]#
>> 8><---
>>
>> gluster 4
>> Centos 7.4
>>
>> 8><---
>> df -h
>> [root@glusterp2 fb]# df -h
>> Filesystem   Size  Used Avail
>> Use% Mounted on
>> /dev/mapper/centos-root   19G  3.4G
>>  16G  19% /
>> devtmpfs 3.8G 0
>> 3.8G   0% /dev
>> tmpfs3.8G   12K
>> 3.8G   1% /dev/shm
>> tmpfs3.8G  9.0M
>> 3.8G   1% /run
>> tmpfs3.8G 0
>> 3.8G   0% /sys/fs/cgroup
>> /dev/mapper/centos-data1 112G   33M
>> 112G   1% /data1
>> /dev/mapper/centos-var19G  219M
>>  19G   2% /var
>> /dev/mapper/centos-home   47G   38M
>>  47G   1% /home
>> /dev/mapper/centos-var_lib   9.4G  178M
>> 9.2G   2% /var/lib
>> /dev/mapper/vg--gluster--prod--1--2-gluster--prod--1--2  932G  263G
>> 669G  29% /bricks/brick1
>> /dev/sda1950M  235M
>> 715M  25% /boot
>> 8><---
>>
>>
>>
>> So the output isnt helping..
>>
>>
>>
>>
>>
>>
>>
>> On 23 May 2018 at 00:29, Karthik Subrahmanya  wrote:
>>
>>> Hi,
>>>
>>> Which version of gluster you are using?
>>>
>>> You can find which file is that using the following command
>>> find  -samefile >> gfid>//
>>>
>>> Please provide the getfatr output of the file which is in split brain.
>>> The steps to recover from split-brain can be found here,
>>> http://gluster.readthedocs.io/en/latest/Troubleshooting/reso
>>> lving-splitbrain/
>>>
>>> HTH,
>>> Karthik
>>>
>>> On Tue, May 22, 2018 at 4:03 AM, Joe Julian 
>>> wrote:
>>>
 How do I find what  "eafb8799-4e7a-4264-9213-26997c5a4693"  is?

 https://docs.gluster.org/en/v3/Troubleshooting/gfid-to-path/


 On May 21, 2018 3:22:01 PM PDT, Thing  wrote:
 >Hi,
 >
 >I seem to have a split brain issue, but I cannot figure out where this
 >is
 >and what it is, can someone help me pls,  I cant find what to fix here.
 >
 >==
 >root@salt-001:~# salt gluster* cmd.run 'df -h'
 >glusterp2.graywitch.co.nz:
 >Filesystem  

Re: [Gluster-users] split brain? but where?

[Thing]

I tried looking for a file of the same size and the gfid doesnt show up,

8><---
[root@glusterp2 fb]# pwd
/bricks/brick1/gv0/.glusterfs/ea/fb
[root@glusterp2 fb]# ls -al
total 3130892
drwx--. 2 root root 64 May 22 13:01 .
drwx--. 4 root root 24 May  8 14:27 ..
-rw---. 1 root root 3294887936 May  4 11:07
eafb8799-4e7a-4264-9213-26997c5a4693
-rw-r--r--. 1 root root   1396 May 22 13:01 gfid.run

so the gfid seems largebut du cant see it...

[root@glusterp2 fb]# du -a /bricks/brick1/gv0 | sort -n -r | head -n 10
275411712 /bricks/brick1/gv0
275411696 /bricks/brick1/gv0/.glusterfs
22484988 /bricks/brick1/gv0/.glusterfs/57
20974980 /bricks/brick1/gv0/.glusterfs/a5
20974976
/bricks/brick1/gv0/.glusterfs/d5/99/d5991797-848d-4d60-b9dc-d31174f63f72
20974976 /bricks/brick1/gv0/.glusterfs/d5/99
20974976 /bricks/brick1/gv0/.glusterfs/d5
20974976
/bricks/brick1/gv0/.glusterfs/a5/27/a5279083-4d24-4dc8-be2d-4f507c5372cf
20974976 /bricks/brick1/gv0/.glusterfs/a5/27
20974976
/bricks/brick1/gv0/.glusterfs/74/75/7475bd15-05a6-45c2-b395-bc9fd3d1763f
[root@glusterp2 fb]#

8><---



On 23 May 2018 at 08:29, Thing  wrote:

> I tried this already.
>
> 8><---
> [root@glusterp2 fb]# find /bricks/brick1/gv0 -samefile
> /bricks/brick1/gv0/.glusterfs/ea/fb/eafb8799-4e7a-4264-9213-26997c5a4693
> /bricks/brick1/gv0/.glusterfs/ea/fb/eafb8799-4e7a-4264-9213-26997c5a4693
> [root@glusterp2 fb]#
> 8><---
>
> gluster 4
> Centos 7.4
>
> 8><---
> df -h
> [root@glusterp2 fb]# df -h
> Filesystem   Size  Used Avail
> Use% Mounted on
> /dev/mapper/centos-root   19G  3.4G   16G
> 19% /
> devtmpfs 3.8G 0  3.8G
>  0% /dev
> tmpfs3.8G   12K  3.8G
>  1% /dev/shm
> tmpfs3.8G  9.0M  3.8G
>  1% /run
> tmpfs3.8G 0  3.8G
>  0% /sys/fs/cgroup
> /dev/mapper/centos-data1 112G   33M  112G
>  1% /data1
> /dev/mapper/centos-var19G  219M   19G
>  2% /var
> /dev/mapper/centos-home   47G   38M   47G
>  1% /home
> /dev/mapper/centos-var_lib   9.4G  178M  9.2G
>  2% /var/lib
> /dev/mapper/vg--gluster--prod--1--2-gluster--prod--1--2  932G  263G
> 669G  29% /bricks/brick1
> /dev/sda1950M  235M  715M
> 25% /boot
> 8><---
>
>
>
> So the output isnt helping..
>
>
>
>
>
>
>
> On 23 May 2018 at 00:29, Karthik Subrahmanya  wrote:
>
>> Hi,
>>
>> Which version of gluster you are using?
>>
>> You can find which file is that using the following command
>> find  -samefile > gfid>//
>>
>> Please provide the getfatr output of the file which is in split brain.
>> The steps to recover from split-brain can be found here,
>> http://gluster.readthedocs.io/en/latest/Troubleshooting/reso
>> lving-splitbrain/
>>
>> HTH,
>> Karthik
>>
>> On Tue, May 22, 2018 at 4:03 AM, Joe Julian  wrote:
>>
>>> How do I find what  "eafb8799-4e7a-4264-9213-26997c5a4693"  is?
>>>
>>> https://docs.gluster.org/en/v3/Troubleshooting/gfid-to-path/
>>>
>>>
>>> On May 21, 2018 3:22:01 PM PDT, Thing  wrote:
>>> >Hi,
>>> >
>>> >I seem to have a split brain issue, but I cannot figure out where this
>>> >is
>>> >and what it is, can someone help me pls,  I cant find what to fix here.
>>> >
>>> >==
>>> >root@salt-001:~# salt gluster* cmd.run 'df -h'
>>> >glusterp2.graywitch.co.nz:
>>> >Filesystem   Size  Used
>>> >Avail Use% Mounted on
>>> >/dev/mapper/centos-root   19G  3.4G
>>> > 16G  19% /
>>> >devtmpfs 3.8G 0
>>> >3.8G   0% /dev
>>> >tmpfs3.8G   12K
>>> >3.8G   1% /dev/shm
>>> >tmpfs3.8G  9.1M
>>> >3.8G   1% /run
>>> >tmpfs3.8G 0
>>> >3.8G   0% /sys/fs/cgroup
>>> >/dev/mapper/centos-tmp   3.8G   33M
>>> >3.7G   1% /tmp
>>> >/dev/mapper/centos-var19G  213M
>>> > 19G   2% /var
>>> >/dev/mapper/centos-home   47G   38M
>>> > 47G   1% /home
>>> >/dev/mapper/centos-data1 112G   33M
>>> >112G   1% /data1
>>> >/dev/mapper/centos-var_lib   9.4G  178M
>>> >9.2G   2% /var/lib
>>> >/dev/mapper/vg--gluster--prod--1--2-gluster--prod--1--2  932G  264G
>>> >668G  29% /bricks/brick1
>>> >/dev/sda1950M  

Re: [Gluster-users] split brain? but where?

[Thing]

I tried this already.

8><---
[root@glusterp2 fb]# find /bricks/brick1/gv0 -samefile
/bricks/brick1/gv0/.glusterfs/ea/fb/eafb8799-4e7a-4264-9213-26997c5a4693
/bricks/brick1/gv0/.glusterfs/ea/fb/eafb8799-4e7a-4264-9213-26997c5a4693
[root@glusterp2 fb]#
8><---

gluster 4
Centos 7.4

8><---
df -h
[root@glusterp2 fb]# df -h
Filesystem   Size  Used Avail
Use% Mounted on
/dev/mapper/centos-root   19G  3.4G   16G
19% /
devtmpfs 3.8G 0  3.8G
 0% /dev
tmpfs3.8G   12K  3.8G
 1% /dev/shm
tmpfs3.8G  9.0M  3.8G
 1% /run
tmpfs3.8G 0  3.8G
 0% /sys/fs/cgroup
/dev/mapper/centos-data1 112G   33M  112G
 1% /data1
/dev/mapper/centos-var19G  219M   19G
 2% /var
/dev/mapper/centos-home   47G   38M   47G
 1% /home
/dev/mapper/centos-var_lib   9.4G  178M  9.2G
 2% /var/lib
/dev/mapper/vg--gluster--prod--1--2-gluster--prod--1--2  932G  263G  669G
29% /bricks/brick1
/dev/sda1950M  235M  715M
25% /boot
8><---



So the output isnt helping..







On 23 May 2018 at 00:29, Karthik Subrahmanya  wrote:

> Hi,
>
> Which version of gluster you are using?
>
> You can find which file is that using the following command
> find  -samefile  gfid>//
>
> Please provide the getfatr output of the file which is in split brain.
> The steps to recover from split-brain can be found here,
> http://gluster.readthedocs.io/en/latest/Troubleshooting/
> resolving-splitbrain/
>
> HTH,
> Karthik
>
> On Tue, May 22, 2018 at 4:03 AM, Joe Julian  wrote:
>
>> How do I find what  "eafb8799-4e7a-4264-9213-26997c5a4693"  is?
>>
>> https://docs.gluster.org/en/v3/Troubleshooting/gfid-to-path/
>>
>>
>> On May 21, 2018 3:22:01 PM PDT, Thing  wrote:
>> >Hi,
>> >
>> >I seem to have a split brain issue, but I cannot figure out where this
>> >is
>> >and what it is, can someone help me pls,  I cant find what to fix here.
>> >
>> >==
>> >root@salt-001:~# salt gluster* cmd.run 'df -h'
>> >glusterp2.graywitch.co.nz:
>> >Filesystem   Size  Used
>> >Avail Use% Mounted on
>> >/dev/mapper/centos-root   19G  3.4G
>> > 16G  19% /
>> >devtmpfs 3.8G 0
>> >3.8G   0% /dev
>> >tmpfs3.8G   12K
>> >3.8G   1% /dev/shm
>> >tmpfs3.8G  9.1M
>> >3.8G   1% /run
>> >tmpfs3.8G 0
>> >3.8G   0% /sys/fs/cgroup
>> >/dev/mapper/centos-tmp   3.8G   33M
>> >3.7G   1% /tmp
>> >/dev/mapper/centos-var19G  213M
>> > 19G   2% /var
>> >/dev/mapper/centos-home   47G   38M
>> > 47G   1% /home
>> >/dev/mapper/centos-data1 112G   33M
>> >112G   1% /data1
>> >/dev/mapper/centos-var_lib   9.4G  178M
>> >9.2G   2% /var/lib
>> >/dev/mapper/vg--gluster--prod--1--2-gluster--prod--1--2  932G  264G
>> >668G  29% /bricks/brick1
>> >/dev/sda1950M  235M
>> >715M  25% /boot
>> >tmpfs771M   12K
>> >771M   1% /run/user/42
>> >glusterp2:gv0/glusterp2/images   932G  273G
>> >659G  30% /var/lib/libvirt/images
>> >glusterp2:gv0932G  273G
>> >659G  30% /isos
>> >tmpfs771M   48K
>> >771M   1% /run/user/1000
>> >tmpfs771M 0
>> >771M   0% /run/user/0
>> >glusterp1.graywitch.co.nz:
>> >   Filesystem Size  Used Avail Use%
>> >Mounted on
>> > /dev/mapper/centos-root 20G  3.5G   17G  18% /
>> >   devtmpfs   3.8G 0  3.8G   0%
>> >/dev
>> >   tmpfs  3.8G   12K  3.8G   1%
>> >/dev/shm
>> >   tmpfs  3.8G  9.0M  3.8G   1%
>> >/run
>> >   tmpfs  3.8G 0  3.8G   0%
>> >/sys/fs/cgroup
>> >   /dev/sda1  969M  206M  713M  23%
>> >/boot
>> >   /dev/mapper/centos-home 50G  4.3G   46G   9%
>> >/home
>> >   /dev/mapper/centos-tmp 

Re: [Gluster-users] split brain? but where?

[Karthik Subrahmanya]

Hi,

Which version of gluster you are using?

You can find which file is that using the following command
find  -samefile //

Please provide the getfatr output of the file which is in split brain.
The steps to recover from split-brain can be found here,
http://gluster.readthedocs.io/en/latest/Troubleshooting/resolving-splitbrain/

HTH,
Karthik

On Tue, May 22, 2018 at 4:03 AM, Joe Julian  wrote:

> How do I find what  "eafb8799-4e7a-4264-9213-26997c5a4693"  is?
>
> https://docs.gluster.org/en/v3/Troubleshooting/gfid-to-path/
>
> On May 21, 2018 3:22:01 PM PDT, Thing  wrote:
> >Hi,
> >
> >I seem to have a split brain issue, but I cannot figure out where this
> >is
> >and what it is, can someone help me pls,  I cant find what to fix here.
> >
> >==
> >root@salt-001:~# salt gluster* cmd.run 'df -h'
> >glusterp2.graywitch.co.nz:
> >Filesystem   Size  Used
> >Avail Use% Mounted on
> >/dev/mapper/centos-root   19G  3.4G
> > 16G  19% /
> >devtmpfs 3.8G 0
> >3.8G   0% /dev
> >tmpfs3.8G   12K
> >3.8G   1% /dev/shm
> >tmpfs3.8G  9.1M
> >3.8G   1% /run
> >tmpfs3.8G 0
> >3.8G   0% /sys/fs/cgroup
> >/dev/mapper/centos-tmp   3.8G   33M
> >3.7G   1% /tmp
> >/dev/mapper/centos-var19G  213M
> > 19G   2% /var
> >/dev/mapper/centos-home   47G   38M
> > 47G   1% /home
> >/dev/mapper/centos-data1 112G   33M
> >112G   1% /data1
> >/dev/mapper/centos-var_lib   9.4G  178M
> >9.2G   2% /var/lib
> >/dev/mapper/vg--gluster--prod--1--2-gluster--prod--1--2  932G  264G
> >668G  29% /bricks/brick1
> >/dev/sda1950M  235M
> >715M  25% /boot
> >tmpfs771M   12K
> >771M   1% /run/user/42
> >glusterp2:gv0/glusterp2/images   932G  273G
> >659G  30% /var/lib/libvirt/images
> >glusterp2:gv0932G  273G
> >659G  30% /isos
> >tmpfs771M   48K
> >771M   1% /run/user/1000
> >tmpfs771M 0
> >771M   0% /run/user/0
> >glusterp1.graywitch.co.nz:
> >   Filesystem Size  Used Avail Use%
> >Mounted on
> > /dev/mapper/centos-root 20G  3.5G   17G  18% /
> >   devtmpfs   3.8G 0  3.8G   0%
> >/dev
> >   tmpfs  3.8G   12K  3.8G   1%
> >/dev/shm
> >   tmpfs  3.8G  9.0M  3.8G   1%
> >/run
> >   tmpfs  3.8G 0  3.8G   0%
> >/sys/fs/cgroup
> >   /dev/sda1  969M  206M  713M  23%
> >/boot
> >   /dev/mapper/centos-home 50G  4.3G   46G   9%
> >/home
> >   /dev/mapper/centos-tmp 3.9G   33M  3.9G   1%
> >/tmp
> >   /dev/mapper/centos-data1   120G   36M  120G   1%
> >/data1
> >   /dev/mapper/vg--gluster--prod1-gluster--prod1  932G  260G  673G  28%
> >/bricks/brick1
> >   /dev/mapper/centos-var  20G  413M   20G   3%
> >/var
> >   /dev/mapper/centos00-var_lib   9.4G  179M  9.2G   2%
> >/var/lib
> >   tmpfs  771M  8.0K  771M   1%
> >/run/user/42
> >   glusterp1:gv0  932G  273G  659G  30%
> >/isos
> >   glusterp1:gv0/glusterp1/images 932G  273G  659G  30%
> >/var/lib/libvirt/images
> >glusterp3.graywitch.co.nz:
> >Filesystem   Size  Used
> >Avail Use% Mounted on
> >/dev/mapper/centos-root   20G  3.5G
> > 17G  18% /
> >devtmpfs 3.8G 0
> >3.8G   0% /dev
> >tmpfs3.8G   12K
> >3.8G   1% /dev/shm
> >tmpfs3.8G  9.0M
> >3.8G   1% /run
> >tmpfs3.8G 0
> >3.8G   0% /sys/fs/cgroup
> >/dev/sda1969M  206M
> >713M  23% /boot
> >/dev/mapper/centos-var20G  206M
> > 20G   2% /var
> >/dev/mapper/centos-tmp   3.9G   33M
> >3.9G   1% /tmp
> >/dev/mapper/centos-home 

Re: [Gluster-users] split brain? but where?

[Joe Julian]

How do I find what  "eafb8799-4e7a-4264-9213-26997c5a4693"  is?

https://docs.gluster.org/en/v3/Troubleshooting/gfid-to-path/

On May 21, 2018 3:22:01 PM PDT, Thing  wrote:
>Hi,
>
>I seem to have a split brain issue, but I cannot figure out where this
>is
>and what it is, can someone help me pls,  I cant find what to fix here.
>
>==
>root@salt-001:~# salt gluster* cmd.run 'df -h'
>glusterp2.graywitch.co.nz:
>Filesystem   Size  Used
>Avail Use% Mounted on
>/dev/mapper/centos-root   19G  3.4G
> 16G  19% /
>devtmpfs 3.8G 0
>3.8G   0% /dev
>tmpfs3.8G   12K
>3.8G   1% /dev/shm
>tmpfs3.8G  9.1M
>3.8G   1% /run
>tmpfs3.8G 0
>3.8G   0% /sys/fs/cgroup
>/dev/mapper/centos-tmp   3.8G   33M
>3.7G   1% /tmp
>/dev/mapper/centos-var19G  213M
> 19G   2% /var
>/dev/mapper/centos-home   47G   38M
> 47G   1% /home
>/dev/mapper/centos-data1 112G   33M
>112G   1% /data1
>/dev/mapper/centos-var_lib   9.4G  178M
>9.2G   2% /var/lib
>/dev/mapper/vg--gluster--prod--1--2-gluster--prod--1--2  932G  264G
>668G  29% /bricks/brick1
>/dev/sda1950M  235M
>715M  25% /boot
>tmpfs771M   12K
>771M   1% /run/user/42
>glusterp2:gv0/glusterp2/images   932G  273G
>659G  30% /var/lib/libvirt/images
>glusterp2:gv0932G  273G
>659G  30% /isos
>tmpfs771M   48K
>771M   1% /run/user/1000
>tmpfs771M 0
>771M   0% /run/user/0
>glusterp1.graywitch.co.nz:
>   Filesystem Size  Used Avail Use%
>Mounted on
> /dev/mapper/centos-root 20G  3.5G   17G  18% /
>   devtmpfs   3.8G 0  3.8G   0%
>/dev
>   tmpfs  3.8G   12K  3.8G   1%
>/dev/shm
>   tmpfs  3.8G  9.0M  3.8G   1%
>/run
>   tmpfs  3.8G 0  3.8G   0%
>/sys/fs/cgroup
>   /dev/sda1  969M  206M  713M  23%
>/boot
>   /dev/mapper/centos-home 50G  4.3G   46G   9%
>/home
>   /dev/mapper/centos-tmp 3.9G   33M  3.9G   1%
>/tmp
>   /dev/mapper/centos-data1   120G   36M  120G   1%
>/data1
>   /dev/mapper/vg--gluster--prod1-gluster--prod1  932G  260G  673G  28%
>/bricks/brick1
>   /dev/mapper/centos-var  20G  413M   20G   3%
>/var
>   /dev/mapper/centos00-var_lib   9.4G  179M  9.2G   2%
>/var/lib
>   tmpfs  771M  8.0K  771M   1%
>/run/user/42
>   glusterp1:gv0  932G  273G  659G  30%
>/isos
>   glusterp1:gv0/glusterp1/images 932G  273G  659G  30%
>/var/lib/libvirt/images
>glusterp3.graywitch.co.nz:
>Filesystem   Size  Used
>Avail Use% Mounted on
>/dev/mapper/centos-root   20G  3.5G
> 17G  18% /
>devtmpfs 3.8G 0
>3.8G   0% /dev
>tmpfs3.8G   12K
>3.8G   1% /dev/shm
>tmpfs3.8G  9.0M
>3.8G   1% /run
>tmpfs3.8G 0
>3.8G   0% /sys/fs/cgroup
>/dev/sda1969M  206M
>713M  23% /boot
>/dev/mapper/centos-var20G  206M
> 20G   2% /var
>/dev/mapper/centos-tmp   3.9G   33M
>3.9G   1% /tmp
>/dev/mapper/centos-home   50G   37M
> 50G   1% /home
>/dev/mapper/centos-data01120G   33M
>120G   1% /data1
>/dev/mapper/centos00-var_lib 9.4G  180M
>9.2G   2% /var/lib
>/dev/mapper/vg--gluster--prod--1--3-gluster--prod--1--3  932G  262G
>670G  29% /bricks/brick1
>glusterp3:gv0932G  273G
>659G  30% /isos
>glusterp3:gv0/glusterp3/images   932G  273G
>659G  30% /var/lib/libvirt/images
>tmpfs771M   12K
>771M   1% /run/user/42

Re: [Gluster-users] Split brain directory

[Karthik Subrahmanya]

Hey,

>From the getfattr output you have provided, the directory is clearly not in
split brain.
If all the bricks are being blamed by others then it is called split brain.
In your case only client-13 that is Brick-14 in the volume info output had
a pending entry heal on the directory.
That is the last replica subvol which consists of the bricks

Brick13: glusterserver03.mydomain.local:/bricks/video/brick3/safe
Brick14: glusterserver04.mydomain.local:/bricks/video/brick3/safe
Brick15: glusterserver05.mydomain.local:/bricks/video/brick3/safe (arbiter)

Which got healed as part of the heal you ran, or part of the self heal
crawl and pending xattrs got reset to all zeros.
Which file are you not able to access? Can you give the getfattr output of
that file and give the shd log
and the mount log where you were not able to access the file.

Regards,
Karthik

On Wed, Jan 24, 2018 at 2:00 PM, Luca Gervasi 
wrote:

> Hello,
> I'm trying to fix an issue with a Directory Split on a gluster 3.10.3. The
> effect consist of a specific file in this splitted directory to randomly be
> unavailable on some clients.
> I have gathered all the informations on this gist: https://gist.
> githubusercontent.com/lucagervasi/534e0024d349933eef44615fa8a5c374/raw/
> 52ff8dd6a9cc8ba09b7f258aa85743d2854f9acc/splitinfo.txt
>
> I discovered the splitted directory by the extended attributes (lines
> 172,173, 291,292,
> trusted.afr.dirty=0x
> trusted.afr.vol-video-client-13=0x
> Seen on the bricks
> * /bricks/video/brick3/safe/video.mysite.it/htdocs/ su glusterserver05
> (lines 278 ro 294)
> * /bricks/video/brick3/safe/video.mysite.it/htdocs/ su glusterserver03
> (lines 159 to 175)
>
> Reading the documentation about afr extended attributes, this situation
> seems unclear (Docs from [1] and [2])
> as own changelog is 0, same as client-13 (glusterserver02.mydomain.
> local:/bricks/video/brick3/safe)
> as my understanding, such "dirty" attributes seems to indicate no split at
> all (feel free to correct me).
>
> Some days ago, I issued a "gluster volume heal vol-video full", which
> endend (probably) that day, leaving no info on /var/log/gluster/glustershd.log
> nor fixing this split.
> I tried to trigger a self heal using "stat" and "ls -l" over the splitted
> directory from a glusterfs mounted client directory, without having the bit
> set cleared.
> The volume heal info split-brain itself shows zero items to be healed
> (lines 388 to 446).
>
> All the clients mount this volume using glusterfs-fuse.
>
> I don't know what to do, please help.
>
> Thanks.
>
> Luca Gervasi
>
> References:
> [1] https://access.redhat.com/documentation/en-US/Red_Hat_
> Storage/2.1/html/Administration_Guide/Recovering_from_File_Split-
> brain.html
> [2] https://access.redhat.com/documentation/en-us/red_hat_
> gluster_storage/3.3/html/administration_guide/sect-managing_split-brain
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain

[Ravishankar N]

On 03/20/2017 06:31 PM, Bernhard Dübi wrote:

Hi Ravi,

thank you very much for looking into this
The gluster volumes are used by CommVault Simpana to store backup 
data. Nothing/Nobody should access the underlying infrastructure.


while looking at the xattrs of the files, I noticed that the only 
difference was the bit-rot.version. So, I assume that something in the 
synchronization of the bit-rot data went wrong and having different 
bit-rot.versions is considered like a split-brain situation and access 
is denied because there is no guarantee of correctness. this is just a 
wild guess.

Hi Bernhard,

bit-rot version can be different between bricks of the replica when I/O 
is successful only on one brick of the replica when the other brick was 
down. (though AFR self-heal will later heal the contents, but not modify 
bitrot xattrs). So that is not a problem.




over the weekend I identified hundreds of files with input/output 
errors. I compared the sha256sum of both bricks, they were always the 
same. I then deleted the affected files from gluster and recreated 
them. this should have fixed the issue. Verification is still running.


if you're interested in the root cause, I can send you more log files 
and the xattrs of some files


If you did not access the underlying bricks directly like you said then 
it could possibly be a bitrot bug. If you don't mind please raise a BZ  
under the bitrot component and the appropriate gluster version with all 
client and brick logs attached.

Also if you do have some kind of reproducer, that would help a lot.
-Ravi




Best Regards
Bernhard


2017-03-20 12:57 GMT+01:00 Ravishankar N >:


SFILE_CONTAINER_080 is the one which seems to be in split-brain.
SFILE_CONTAINER_046, for which you have provided the getfattr
output, hard links etc doesn't seem to be in split-brain.  We do
see that the fops on SFILE_CONTAINER_046 are failing on the client
translator itself due to EIO:

[2017-03-17 19:49:56.088867] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk]
0-Server_Legal_01-client-0: remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
(bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]

[2017-03-17 19:49:56.089012] E [MSGID: 114031]
[client-rpc-fops.c:444:client3_3_open_cbk]
0-Server_Legal_01-client-1: remote operation failed. Path:
/Server_Legal/CV_MAGNETIC/V_944453/CHUNK_9291168/SFILE_CONTAINER_046
(bfdfe21a-1af3-474b-a6a4-bc0e17edb529) [Input/output error]

which is  why the sha256sum on the mount gave EIO.  And that is
because the file seems to be corrupt on both bricks because the
'trusted.bit-rot.bad-file' xattr is set.

Did you write to the files directly on the backend? What is
interesting is that the sha256sum is same on both the bricks
despite being both marked as bad by bitrot.

-Ravi


On 03/18/2017 03:20 AM, Bernhard Dübi wrote:

Hi,

I have a situation

the volume logfile reports a possible split-brain but when I try
to heal it fails because the file is not in split-brain. Any ideas?




Regards

Bernhard



___
Gluster-users mailing list
Gluster-users@gluster.org 
http://lists.gluster.org/mailman/listinfo/gluster-users



___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain issue with non-existent file

[Ravishankar N]

On 07/15/2016 08:23 PM, Rob Janney wrote:

Currently we are on 3.6:  glusterfs 3.6.9 built on Mar  2 2016 18:21:17


That version does not have the real-time gfapi based version of the 
`info split-brain` command and it just prints prior event history from 
the selfheal daemon's memory. You can restart selfheal daemon to get rid 
of it.  Or you could ignore the message. Probably a good time to upgrade 
to 3.7 or 3.8.

-Ravi



Yes, the file gfid file still was present at 
/.glusterfs/ab/da/abdab36b-b90b-4201-98fe-7a36059da81d



On Thu, Jul 14, 2016 at 7:51 PM, Ravishankar N > wrote:




On 07/15/2016 02:24 AM, Rob Janney wrote:

I have a 2 node cluster that reports a single file as split
brained .. but the file itself doesn't exist on either node.

What is the version of gluster you are running?


I did not find this scenario in the split brain docs, but based
on other scenarios I deleted the gfid file

You mean even though the file did not exist, the
.glusterfs/ still was present?
-Ravi

and then ran a full heal, however; it still complains about split
brain.   I am considering just restoring the bricks from
snapshot, but in the future that may not be ideal.   Is there
another way to resolve this?


split brain output:

Gathering list of split brain entries on volume storage has been
successful

Brick

storage-9-ceed295d-f8ee-4fa4-ae88-f483abff54ef.wpestorage.net:/gbrick/ceed295d-f8ee-4fa4-ae88-f483abff54ef
Number of entries: 0

Brick

storage-9-a98da283-c44b-4450-9d7e-093337256461.wpestorage.net:/gbrick/a98da283-c44b-4450-9d7e-093337256461
Number of entries: 1
atpath on brick
---
2016-06-23 18:12:44

/17973/Manifest.php115__tIRAjD/meta

Thanks,
Rob


___
Gluster-users mailing list
Gluster-users@gluster.org 
http://www.gluster.org/mailman/listinfo/gluster-users






___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain issue with non-existent file

[Rob Janney]

Currently we are on 3.6:  glusterfs 3.6.9 built on Mar  2 2016 18:21:17

Yes, the file gfid file still was present at
/.glusterfs/ab/da/abdab36b-b90b-4201-98fe-7a36059da81d


On Thu, Jul 14, 2016 at 7:51 PM, Ravishankar N 
wrote:

>
>
> On 07/15/2016 02:24 AM, Rob Janney wrote:
>
> I have a 2 node cluster that reports a single file as split brained .. but
> the file itself doesn't exist on either node.
>
> What is the version of gluster you are running?
>
> I did not find this scenario in the split brain docs, but based on other
> scenarios I deleted the gfid file
>
> You mean even though the file did not exist, the
> .glusterfs/ still was present?
> -Ravi
>
> and then ran a full heal, however; it still complains about split brain.
> I am considering just restoring the bricks from snapshot, but in the future
> that may not be ideal.   Is there another way to resolve this?
>
>
> split brain output:
>
> Gathering list of split brain entries on volume storage has been
> successful
>
> Brick storage-9-ceed295d-f8ee-4fa4-ae88-f483abff54ef.wpestorage.net:
> /gbrick/ceed295d-f8ee-4fa4-ae88-f483abff54ef
> Number of entries: 0
>
> Brick storage-9-a98da283-c44b-4450-9d7e-093337256461.wpestorage.net:
> /gbrick/a98da283-c44b-4450-9d7e-093337256461
> Number of entries: 1
> atpath on brick
> ---
> 2016-06-23 18:12:44
> /17973/Manifest.php115__tIRAjD/meta
>
> Thanks,
> Rob
>
>
> ___
> Gluster-users mailing 
> listGluster-users@gluster.orghttp://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain issue with non-existent file

[Ravishankar N]

On 07/15/2016 02:24 AM, Rob Janney wrote:
I have a 2 node cluster that reports a single file as split brained .. 
but the file itself doesn't exist on either node.

What is the version of gluster you are running?

I did not find this scenario in the split brain docs, but based on 
other scenarios I deleted the gfid file
You mean even though the file did not exist, the 
.glusterfs/ still was present?

-Ravi
and then ran a full heal, however; it still complains about split 
brain.   I am considering just restoring the bricks from snapshot, but 
in the future that may not be ideal.   Is there another way to resolve 
this?



split brain output:

Gathering list of split brain entries on volume storage has been 
successful


Brick 
storage-9-ceed295d-f8ee-4fa4-ae88-f483abff54ef.wpestorage.net:/gbrick/ceed295d-f8ee-4fa4-ae88-f483abff54ef

Number of entries: 0

Brick 
storage-9-a98da283-c44b-4450-9d7e-093337256461.wpestorage.net:/gbrick/a98da283-c44b-4450-9d7e-093337256461

Number of entries: 1
atpath on brick
---
2016-06-23 18:12:44 
/17973/Manifest.php115__tIRAjD/meta


Thanks,
Rob


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users



___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain recovery automation, any plans?

[Dmitry Melekhov]

13.07.2016 07:44, Pranith Kumar Karampuri пишет:



On Tue, Jul 12, 2016 at 9:27 PM, Dmitry Melekhov > wrote:




12.07.2016 17:38, Pranith Kumar Karampuri пишет:

Did you wait for heals to complete before upgrading second node?


no...


So basically if you have operations in progress on the mount, you 
should wait for heals to complete before you upgrade second node. If 
you have all the operations on all the mounts stopped or you unmounted 
all the mounts for the volume, then you can upgrade all the servers 
one by one then clients. Otherwise it will lead to problems. That said 
in 3 way replica it shouldn't cause split-brains. So I would like to 
know exact steps that lead to this problem.


Thank you, this is all I can remember :-(

We know of one issue which leads to split-brains in case of VM 
workloads where we take down bricks in cyclic manner without waiting 
for heals to complete. I wonder if the steps that lead to split-brain 
on your setup are similar. We are targetting this for future releases...

I guess we hit this...






On Tue, Jul 12, 2016 at 3:08 PM, Dmitry Melekhov > wrote:

12.07.2016 13:31, Pranith Kumar Karampuri пишет:



On Mon, Jul 11, 2016 at 2:26 PM, Dmitry Melekhov
> wrote:

11.07.2016 12:47, Gandalf Corvotempesta пишет:

2016-07-11 9:54 GMT+02:00 Dmitry Melekhov
>:

We just got split-brain during update to 3.7.13 ;-)

This is an interesting point.
Could you please tell me which replica count did you
set ?


3


With replica "3" split brain should not occurs, right ?


I guess we did something wrong :-)


Or there is a bug we never found? Could you please share
details about what you did?


upgraded to 3.7.13 from 3.7.11 using yum, while at least one
VM is running :-)
on all 3 servers, one by one:

yum upgrade
systemctl stop glusterd
than killed glusterfsd processes using kill
and systemctl start glusterd

then next server

after this we tried to restart VM, but it failed, because we
forget to restart libvirtd, and it used old libraries,
I guess this is point where we got this problem.



I'm planning a new cluster and I would like to be
protected against
split brains.


___
Gluster-users mailing list
Gluster-users@gluster.org 
http://www.gluster.org/mailman/listinfo/gluster-users




-- 
Pranith





-- 
Pranith





--
Pranith


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain recovery automation, any plans?

[Dmitry Melekhov]

13.07.2016 00:20, Darrell Budic пишет:
FYI, it’s my experience that “yum upgrade” will stop the running 
glistered (and possibly the running glusterfsds)


This is main point- looks like glusterfsds are not stopped by upgrade, 
not sure, though...


Thank you!

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain recovery automation, any plans?

[Pranith Kumar Karampuri]

On Tue, Jul 12, 2016 at 9:27 PM, Dmitry Melekhov  wrote:

>
>
> 12.07.2016 17:38, Pranith Kumar Karampuri пишет:
>
> Did you wait for heals to complete  before upgrading second node?
>
>
> no...
>

So basically if you have operations in progress on the mount, you should
wait for heals to complete before you upgrade second node. If you have all
the operations on all the mounts stopped or you unmounted all the mounts
for the volume, then you can upgrade all the servers one by one then
clients. Otherwise it will lead to problems. That said in 3 way replica it
shouldn't cause split-brains. So I would like to know exact steps that lead
to this problem. We know of one issue which leads to split-brains in case
of VM workloads where we take down bricks in cyclic manner without waiting
for heals to complete. I wonder if the steps that lead to split-brain on
your setup are similar. We are targetting this for future releases...


>
>
>
> On Tue, Jul 12, 2016 at 3:08 PM, Dmitry Melekhov  wrote:
>
>> 12.07.2016 13:31, Pranith Kumar Karampuri пишет:
>>
>>
>>
>> On Mon, Jul 11, 2016 at 2:26 PM, Dmitry Melekhov < 
>> d...@belkam.com> wrote:
>>
>>> 11.07.2016 12:47, Gandalf Corvotempesta пишет:
>>>
 2016-07-11 9:54 GMT+02:00 Dmitry Melekhov < 
 d...@belkam.com>:

> We just got split-brain during update to 3.7.13 ;-)
>
 This is an interesting point.
 Could you please tell me which replica count did you set ?

>>>
>>> 3
>>>

 With replica "3" split brain should not occurs, right ?

>>>
>>> I guess we did something wrong :-)
>>
>>
>> Or there is a bug we never found? Could you please share details about
>> what you did?
>>
>>
>> upgraded to 3.7.13 from 3.7.11 using yum, while at least one VM is
>> running :-)
>> on all 3 servers, one by one:
>>
>> yum upgrade
>> systemctl stop glusterd
>> than killed glusterfsd processes using kill
>> and systemctl start glusterd
>>
>> then next server
>>
>> after this we tried to restart VM, but it failed, because we forget to
>> restart libvirtd, and it used old libraries,
>> I guess this is point where we got this problem.
>>
>>
>>
>>>
>>> I'm planning a new cluster and I would like to be protected against
 split brains.

>>>
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
>>
>> --
>> Pranith
>>
>>
>>
>
>
> --
> Pranith
>
>
>


-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain recovery automation, any plans?

[Darrell Budic]

FYI, it’s my experience that “yum upgrade” will stop the running glistered (and 
possibly the running glusterfsds) during it’s installation of new gluster 
components. I’ve also noticed it starts them back up again during the process.

Ie, yesterday I upgraded a system to 3.7.13:

systemctl stop glusterd

yum upgrade

and discovered that glusterd was running again, and had started during the yum 
upgrade processing. All it’s glusterfsds had also started. Somewhat annoying, 
actually, because I had been planning to reboot the server to switch to the 
latest kernel as part of the process, but really didn’t feel like interrupting 
the heals at that time.

This probably didn’t have much impact on you, but it would have restarted any 
healing that was a result of the upgrade downtime twice. You may have caused 
yourself some extra wait for the first round of healing to conclude if stuff 
was using those volumes at the time. If you didn’t wait for those to conclude 
before starting your next upgrade, you could have caused a split brain on 
affected active files.

  -Darrell

> On Jul 12, 2016, at 10:57 AM, Dmitry Melekhov  wrote:
> 
> 
> 
> 12.07.2016 17:38, Pranith Kumar Karampuri пишет:
>> Did you wait for heals to complete  before upgrading second node?
> 
> no...
> 
>> 
>> On Tue, Jul 12, 2016 at 3:08 PM, Dmitry Melekhov > > wrote:
>> 12.07.2016 13:31, Pranith Kumar Karampuri пишет:
>>> 
>>> 
>>> On Mon, Jul 11, 2016 at 2:26 PM, Dmitry Melekhov < 
>>> d...@belkam.com > wrote:
>>> 11.07.2016 12:47, Gandalf Corvotempesta пишет:
>>> 2016-07-11 9:54 GMT+02:00 Dmitry Melekhov < 
>>> d...@belkam.com >:
>>> We just got split-brain during update to 3.7.13 ;-)
>>> This is an interesting point.
>>> Could you please tell me which replica count did you set ?
>>> 
>>> 3
>>> 
>>> With replica "3" split brain should not occurs, right ?
>>> 
>>> I guess we did something wrong :-)
>>> 
>>> Or there is a bug we never found? Could you please share details about what 
>>> you did?
>> 
>> upgraded to 3.7.13 from 3.7.11 using yum, while at least one VM is running 
>> :-)
>> on all 3 servers, one by one:
>> 
>> yum upgrade
>> systemctl stop glusterd 
>> than killed glusterfsd processes using kill 
>> and systemctl start glusterd
>> 
>> then next server
>> 
>> after this we tried to restart VM, but it failed, because we forget to 
>> restart libvirtd, and it used old libraries,
>> I guess this is point where we got this problem.
>> 
>>>  
>>> 
>>> I'm planning a new cluster and I would like to be protected against
>>> split brains.
>>> 
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org 
>>> http://www.gluster.org/mailman/listinfo/gluster-users 
>>> 
>>> 
>>> 
>>> -- 
>>> Pranith
>> 
>> 
>> 
>> 
>> -- 
>> Pranith
> 
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain recovery automation, any plans?

[Dmitry Melekhov]

12.07.2016 17:38, Pranith Kumar Karampuri пишет:

Did you wait for heals to complete  before upgrading second node?


no...



On Tue, Jul 12, 2016 at 3:08 PM, Dmitry Melekhov > wrote:


12.07.2016 13:31, Pranith Kumar Karampuri пишет:



On Mon, Jul 11, 2016 at 2:26 PM, Dmitry Melekhov > wrote:

11.07.2016 12:47, Gandalf Corvotempesta пишет:

2016-07-11 9:54 GMT+02:00 Dmitry Melekhov >:

We just got split-brain during update to 3.7.13 ;-)

This is an interesting point.
Could you please tell me which replica count did you set ?


3


With replica "3" split brain should not occurs, right ?


I guess we did something wrong :-)


Or there is a bug we never found? Could you please share details
about what you did?


upgraded to 3.7.13 from 3.7.11 using yum, while at least one VM is
running :-)
on all 3 servers, one by one:

yum upgrade
systemctl stop glusterd
than killed glusterfsd processes using kill
and systemctl start glusterd

then next server

after this we tried to restart VM, but it failed, because we
forget to restart libvirtd, and it used old libraries,
I guess this is point where we got this problem.



I'm planning a new cluster and I would like to be
protected against
split brains.


___
Gluster-users mailing list
Gluster-users@gluster.org 
http://www.gluster.org/mailman/listinfo/gluster-users




-- 
Pranith





--
Pranith


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain recovery automation, any plans?

[Pranith Kumar Karampuri]

Did you wait for heals to complete  before upgrading second node?

On Tue, Jul 12, 2016 at 3:08 PM, Dmitry Melekhov  wrote:

> 12.07.2016 13:31, Pranith Kumar Karampuri пишет:
>
>
>
> On Mon, Jul 11, 2016 at 2:26 PM, Dmitry Melekhov < 
> d...@belkam.com> wrote:
>
>> 11.07.2016 12:47, Gandalf Corvotempesta пишет:
>>
>>> 2016-07-11 9:54 GMT+02:00 Dmitry Melekhov < d...@belkam.com
>>> >:
>>>
 We just got split-brain during update to 3.7.13 ;-)

>>> This is an interesting point.
>>> Could you please tell me which replica count did you set ?
>>>
>>
>> 3
>>
>>>
>>> With replica "3" split brain should not occurs, right ?
>>>
>>
>> I guess we did something wrong :-)
>
>
> Or there is a bug we never found? Could you please share details about
> what you did?
>
>
> upgraded to 3.7.13 from 3.7.11 using yum, while at least one VM is running
> :-)
> on all 3 servers, one by one:
>
> yum upgrade
> systemctl stop glusterd
> than killed glusterfsd processes using kill
> and systemctl start glusterd
>
> then next server
>
> after this we tried to restart VM, but it failed, because we forget to
> restart libvirtd, and it used old libraries,
> I guess this is point where we got this problem.
>
>
>
>>
>> I'm planning a new cluster and I would like to be protected against
>>> split brains.
>>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
>
>
>
> --
> Pranith
>
>
>


-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain recovery automation, any plans?

[Dmitry Melekhov]

12.07.2016 13:31, Pranith Kumar Karampuri пишет:



On Mon, Jul 11, 2016 at 2:26 PM, Dmitry Melekhov > wrote:


11.07.2016 12:47, Gandalf Corvotempesta пишет:

2016-07-11 9:54 GMT+02:00 Dmitry Melekhov >:

We just got split-brain during update to 3.7.13 ;-)

This is an interesting point.
Could you please tell me which replica count did you set ?


3


With replica "3" split brain should not occurs, right ?


I guess we did something wrong :-)


Or there is a bug we never found? Could you please share details about 
what you did?


upgraded to 3.7.13 from 3.7.11 using yum, while at least one VM is 
running :-)

on all 3 servers, one by one:

yum upgrade
systemctl stop glusterd
than killed glusterfsd processes using kill
and systemctl start glusterd

then next server

after this we tried to restart VM, but it failed, because we forget to 
restart libvirtd, and it used old libraries,

I guess this is point where we got this problem.



I'm planning a new cluster and I would like to be protected
against
split brains.


___
Gluster-users mailing list
Gluster-users@gluster.org 
http://www.gluster.org/mailman/listinfo/gluster-users




--
Pranith


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain recovery automation, any plans?

[Pranith Kumar Karampuri]

On Mon, Jul 11, 2016 at 2:26 PM, Dmitry Melekhov  wrote:

> 11.07.2016 12:47, Gandalf Corvotempesta пишет:
>
>> 2016-07-11 9:54 GMT+02:00 Dmitry Melekhov :
>>
>>> We just got split-brain during update to 3.7.13 ;-)
>>>
>> This is an interesting point.
>> Could you please tell me which replica count did you set ?
>>
>
> 3
>
>>
>> With replica "3" split brain should not occurs, right ?
>>
>
> I guess we did something wrong :-)


Or there is a bug we never found? Could you please share details about what
you did?


>
> I'm planning a new cluster and I would like to be protected against
>> split brains.
>>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>



-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain recovery automation, any plans?

[Dmitry Melekhov]

11.07.2016 12:47, Gandalf Corvotempesta пишет:

2016-07-11 9:54 GMT+02:00 Dmitry Melekhov :

We just got split-brain during update to 3.7.13 ;-)

This is an interesting point.
Could you please tell me which replica count did you set ?


3


With replica "3" split brain should not occurs, right ?


I guess we did something wrong :-)

I'm planning a new cluster and I would like to be protected against
split brains.


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain recovery automation, any plans?

[Gandalf Corvotempesta]

2016-07-11 9:54 GMT+02:00 Dmitry Melekhov :
> We just got split-brain during update to 3.7.13 ;-)

This is an interesting point.
Could you please tell me which replica count did you set ?

With replica "3" split brain should not occurs, right ?
I'm planning a new cluster and I would like to be protected against
split brains.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain recovery automation, any plans?

[Ravishankar N]

On 07/11/2016 01:54 PM, Dmitry Melekhov wrote:

11.07.2016 12:18, Ravishankar N пишет:

On 07/11/2016 01:24 PM, Dmitry Melekhov wrote:

Hello!


We just got split-brain during update to 3.7.13 ;-)

This was our first time we hit this, and we found that it is not 
easy to follow and change xattrs metadata.

So, could you tell me, are there any plans to simplify this?
Let's say, add tool to choose preferred copy (brick) to heal others 
from and, thus, avoid all this direct xattr manipulations?


Commands are already available to resolve split-brains from the 
gluster command line or from the fuse mount too. 
https://github.com/gluster/glusterfs-specs/blob/master/done/Features/heal-info-and-split-brain-resolution.md 
is a bit dated but has some examples.


-Ravi



Thank you!

Unfortunately, first link I found was
https://gluster.readthedocs.io/en/latest/Troubleshooting/split-brain/
and it contains old info... :-(


You were looking at the right place, it is just that readthedocs hasn't 
been updated to show the content that I referred to. I'll send a patch 
to update the doc.

Thanks for reporting.
Ravi





Thank you!

btw, 3.7.13 works ok with qemu gfapi on centos7 :-)

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users





___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users



___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain recovery automation, any plans?

[Dmitry Melekhov]

11.07.2016 12:18, Ravishankar N пишет:

On 07/11/2016 01:24 PM, Dmitry Melekhov wrote:

Hello!


We just got split-brain during update to 3.7.13 ;-)

This was our first time we hit this, and we found that it is not easy 
to follow and change xattrs metadata.

So, could you tell me, are there any plans to simplify this?
Let's say, add tool to choose preferred copy (brick) to heal others 
from and, thus, avoid all this direct xattr manipulations?


Commands are already available to resolve split-brains from the 
gluster command line or from the fuse mount too. 
https://github.com/gluster/glusterfs-specs/blob/master/done/Features/heal-info-and-split-brain-resolution.md 
is a bit dated but has some examples.


-Ravi



Thank you!

Unfortunately, first link I found was
https://gluster.readthedocs.io/en/latest/Troubleshooting/split-brain/
and it contains old info... :-(



Thank you!

btw, 3.7.13 works ok with qemu gfapi on centos7 :-)

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users





___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain recovery automation, any plans?

[Ravishankar N]

On 07/11/2016 01:24 PM, Dmitry Melekhov wrote:

Hello!


We just got split-brain during update to 3.7.13 ;-)

This was our first time we hit this, and we found that it is not easy 
to follow and change xattrs metadata.

So, could you tell me, are there any plans to simplify this?
Let's say, add tool to choose preferred copy (brick) to heal others 
from and, thus, avoid all this direct xattr manipulations?


Commands are already available to resolve split-brains from the gluster 
command line or from the fuse mount too. 
https://github.com/gluster/glusterfs-specs/blob/master/done/Features/heal-info-and-split-brain-resolution.md 
is a bit dated but has some examples.


-Ravi



Thank you!

btw, 3.7.13 works ok with qemu gfapi on centos7 :-)

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users



___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain?

[Pranith Kumar Karampuri]

You need to follow the steps to understand and resolve split-brains:

https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md
https://github.com/gluster/glusterfs-specs/blob/master/done/Features/heal-info-and-split-brain-resolution.md

Feel free to let us know if you have any doubts. If you do not want to 
run into any split-brains please use arbiter based configuration or 3 
way replication


Pranith
On 02/09/2016 10:06 PM, Rodolfo Gonzalez wrote:

Sure:

administrador@data4:~$ sudo gluster volume heal data info
[sudo] password for administrador:
Brick data1:/data/
/path/Report5.csv - Is in split-brain

Number of entries: 1

Brick data2:/data/
/path/Report5.csv - Is in split-brain

Number of entries: 1

Brick data3:/data
Status: El otro extremo de la conexión no está conectado

Brick data4:/data/




















... (tons more)



2016-02-08 23:47 GMT-06:00 Pranith Kumar Karampuri 
>:




On 02/08/2016 09:05 PM, Rodolfo Gonzalez wrote:

Hello, I think that I have a split-brain issue in a
replicated-striped gluster cluster with 4 bricks: brick1+brick2
- brick3+brick4

In both brick3 and brick4 I'm getting these kind of messages:

[2016-02-08 15:01:49.720343] E
[afr-self-heal-entry.c:246:afr_selfheal_detect_gfid_and_type_mismatch]
0-data-replicate-1: Gfid mismatch detected for
<572624f9-6752-44c4-b403-d1775dc7ea0d/Report.csv>,
05dabf44-56aa-471f-a898-02ce225d31b0 on data-client-3 and
1cf52b86-6c69-4483-8f17-686f50b3f316 on data-client-2. Skipping
conservative merge on the file.

gluster volumen status show 4 bricks online and no errors.
gluster peer status shows all 4 bricks connected. bricks 2 and 1
show no errors. Gluster version is 3.6.8 in all 4 servers.

How can this be solved? Any help is appreciated :)


Could you give "gluster volume heal  info" output?

Pranith


Thank you.


___
Gluster-users mailing list
Gluster-users@gluster.org  
http://www.gluster.org/mailman/listinfo/gluster-users





___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain?

[Pranith Kumar Karampuri]

On 02/10/2016 02:03 PM, Pranith Kumar Karampuri wrote:

You need to follow the steps to understand and resolve split-brains:

https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md
https://github.com/gluster/glusterfs-specs/blob/master/done/Features/heal-info-and-split-brain-resolution.md

Feel free to let us know if you have any doubts. If you do not want to 
run into any split-brains please use arbiter based configuration or 3 
way replication


Please use 
https://github.com/gluster/glusterfs-specs/blob/master/done/Features/afr-arbiter-volumes.md 
for arbiter based volumes. It is better to use 3.7.8 for this though.


Pranith


Pranith
On 02/09/2016 10:06 PM, Rodolfo Gonzalez wrote:

Sure:

administrador@data4:~$ sudo gluster volume heal data info
[sudo] password for administrador:
Brick data1:/data/
/path/Report5.csv - Is in split-brain

Number of entries: 1

Brick data2:/data/
/path/Report5.csv - Is in split-brain

Number of entries: 1

Brick data3:/data
Status: El otro extremo de la conexión no está conectado

Brick data4:/data/




















... (tons more)



2016-02-08 23:47 GMT-06:00 Pranith Kumar Karampuri 
>:




On 02/08/2016 09:05 PM, Rodolfo Gonzalez wrote:

Hello, I think that I have a split-brain issue in a
replicated-striped gluster cluster with 4 bricks: brick1+brick2
- brick3+brick4

In both brick3 and brick4 I'm getting these kind of messages:

[2016-02-08 15:01:49.720343] E
[afr-self-heal-entry.c:246:afr_selfheal_detect_gfid_and_type_mismatch]
0-data-replicate-1: Gfid mismatch detected for
<572624f9-6752-44c4-b403-d1775dc7ea0d/Report.csv>,
05dabf44-56aa-471f-a898-02ce225d31b0 on data-client-3 and
1cf52b86-6c69-4483-8f17-686f50b3f316 on data-client-2. Skipping
conservative merge on the file.

gluster volumen status show 4 bricks online and no errors.
gluster peer status shows all 4 bricks connected. bricks 2 and 1
show no errors. Gluster version is 3.6.8 in all 4 servers.

How can this be solved? Any help is appreciated :)


Could you give "gluster volume heal  info" output?

Pranith


Thank you.


___
Gluster-users mailing list
Gluster-users@gluster.org  
http://www.gluster.org/mailman/listinfo/gluster-users







___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain?

[Pranith Kumar Karampuri]

On 02/08/2016 09:05 PM, Rodolfo Gonzalez wrote:
Hello, I think that I have a split-brain issue in a replicated-striped 
gluster cluster with 4 bricks: brick1+brick2 - brick3+brick4


In both brick3 and brick4 I'm getting these kind of messages:

[2016-02-08 15:01:49.720343] E 
[afr-self-heal-entry.c:246:afr_selfheal_detect_gfid_and_type_mismatch] 
0-data-replicate-1: Gfid mismatch detected for 
<572624f9-6752-44c4-b403-d1775dc7ea0d/Report.csv>, 
05dabf44-56aa-471f-a898-02ce225d31b0 on data-client-3 and 
1cf52b86-6c69-4483-8f17-686f50b3f316 on data-client-2. Skipping 
conservative merge on the file.


gluster volumen status show 4 bricks online and no errors. gluster 
peer status shows all 4 bricks connected. bricks 2 and 1 show no 
errors. Gluster version is 3.6.8 in all 4 servers.


How can this be solved? Any help is appreciated :)


Could you give "gluster volume heal  info" output?

Pranith


Thank you.


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split-brain after uploading file

[Miloš Kozák]

any thoughts? 

Miloš

> 30. 11. 2015 v 23:45, Miloš Kozák :
> 
> I am using Gluster for a few years without any significant issue (after I 
> tweaked configuration  for v3.5). My configuration is as follows:
> 
> network.remote-dio: enable
> cluster.eager-lock: enable
> performance.stat-prefetch: off
> performance.io-cache: off
> performance.read-ahead: off
> performance.quick-read: off
> performance.io-thread-count: 6
> network.ping-timeout: 2
> performance.cache-max-file-size: 0
> performance.flush-behind: on
> features.barrier: disable
> snap-max-soft-limit: 7
> auto-delete: on
> 
> I use it for running virtual servers on top such a volume.  Currently I run 
> this version of Gluster:
> 
> glusterfs-cli-3.6.5-1.el6.x86_64
> glusterfs-3.6.5-1.el6.x86_64
> glusterfs-api-3.6.5-1.el6.x86_64
> glusterfs-server-3.6.5-1.el6.x86_64
> glusterfs-libs-3.6.5-1.el6.x86_64
> glusterfs-fuse-3.6.5-1.el6.x86_64
> 
> With recent CentOS 6.
> 
> I have experienced an issue when I move some files from an hdd onto gluster 
> volume such that one node gets overloaded in the middle of file upload. 
> Therefore, I decided to upload it through ssh onto other server than where 
> original images are store. I know that this sounds just weird, but it does 
> not lead to overloading! 
> 
> Along these lines, I decided to upload 10G image onto gluster volume and the 
> upload speed varied, but no overloading at all… Right after upload was done I 
> realized that some virtuals are not running properly. Hence I checked heal 
> status where I discoverd that 4 images are in split-brain state. I had to act 
> quickly, so I resolved the split brain, and let gluster heal. When heal was 
> done everything works… 
> 
> However, I have got a few more VMs to upload, and I am not sure what can 
> happen.. 
> 
> My volume configuration:
> 
> Volume Name: ph-fs-0
> Type: Replicate
> Volume ID: 71ac6456-03e4-4bb3-a624-937f4605b2cb
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: 10.11.100.1:/gfs/s3-sata-10k/fs
> Brick2: 10.11.100.2:/gfs/s3-sata-10k/fs
> Options Reconfigured:
> network.remote-dio: enable
> cluster.eager-lock: enable
> performance.stat-prefetch: off
> performance.io-cache: off
> performance.read-ahead: off
> performance.quick-read: off
> performance.io-thread-count: 6
> network.ping-timeout: 2
> performance.cache-max-file-size: 0
> performance.flush-behind: on
> features.barrier: disable
> snap-max-soft-limit: 7
> auto-delete: on
> 
> 
> and logs are attached.
> 
> Miloš
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain files not shown in gluster volume heal info split-brain command

[Andreas Hollaus]

Hi,

Isn't the trusted.afr.dirty attribute missing from Brick 2? Shouldn't it be 
increased
and decreased, but never removed?
Could that be the reason why GlusterFS is confused?

What could be the reason for gfid mismatches?


Regards
Andreas
 Brick1 getfattr -d -m . -e hex config.ior
 # file: config.ior
 trusted.afr.c_glusterfs-client-5=0x
 trusted.afr.dirty=0x
 trusted.bit-rot.version=0x000255c0c95f0002ebc7
 trusted.gfid=0xa88805cdad454e71b30a97f88fdceed8

 Brick2 cat config.ior
 IOR:002B49444C3A6F6D672E6F72672F436F734E616D696E672F4E616D696E67436F6E746578744578743A312E3100580001020C31302E36372E32392E333600DE0200255374616E64617264496D706C4E616D652F4E616D655365727665722D504F412F5F726F6F740100084A414300
 Brick2 getfattr -d -m . -e hex config.ior
 # file: config.ior
 trusted.afr.c_glusterfs-client-4=0x00040001
 trusted.bit-rot.version=0x000255c0e3cf0006246e
 trusted.gfid=0x0b6d4b8a18c247f8abf59065eb85beb2
 ##


On 08/12/15 14:08, Ravishankar N wrote:


 On 08/11/2015 03:28 PM, Lakshmi Anusha wrote:
 From the extended attributes, an entry split-brain seems to be appeared. 
 Since
 gfid is different for the original file and its replica.
 Can you please let us know why the split-brain files are not shown in 
 gluster
 volume heal vol info split-brain command?

 For gfid mismatches, 'info split-brain` should indicate that the parent 
 directory
 of that file is in split-brain. Strange that it isn't.
 If you still have the setup, can you provide the getfattr output of the 
 parent dir
 on both bricks? Also, does `gluster vol heal vol info ` list the file or the
 parent dir?

 Kindly let us know if any known issue exists.



 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 http://www.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain files not shown in gluster volume heal info split-brain command

[Ravishankar N]

On 08/13/2015 07:42 PM, Andreas Hollaus wrote:


Isn't the trusted.afr.dirty attribute missing from Brick 2? Shouldn't 
it be increased and decreased, but never removed?
If one brick of a replica 2 setup is down, and files are written to, the 
dirty xattr is never set on the brick that is up.  Instead, it marks the 
trusted.afr pending xattr for the down brick directly.

-Ravi
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain files not shown in gluster volume heal info split-brain command

[Ravishankar N]

On 08/13/2015 07:30 PM, Lakshmi Anusha wrote:

Hello,

We managed to collect below command outputs:

Brick1 getfattr -d -m . -e hex 
/opt/lvmdir/c2/brick/logfiles/security/EVENT_LOG.xml

getfattr: Removing leading '/' from absolute path names
# file: opt/lvmdir/c2/brick/logfiles/security/EVENT_LOG.xml
trusted.afr.dirty=0x
trusted.bit-rot.version=0x000255cc791e000491d6
trusted.gfid=0x700021362c414760993055a5b67aa942


Brick2 getfattr -d -m . -e hex 
/opt/lvmdir/c2/brick/logfiles/security/EVENT_LOG.xml

getfattr: Removing leading '/' from absolute path names
# file: opt/lvmdir/c2/brick/logfiles/security/EVENT_LOG.xml
trusted.afr.c_glusterfs-client-11=0x0006
trusted.bit-rot.version=0x000455cc7dfbdc2e
trusted.gfid=0x344107bb14544b0a95a0d0e3af230770



# Brick1 gluster volume heal c_glusterfs info split-brain
Brick oamhost:/opt/lvmdir/c2/brick/
Number of entries in split-brain: 0

Brick oamhost:/opt/lvmdir/c2/brick/
Number of entries in split-brain: 0



Why is the same hostname:brick being listed twice? Could you list the 
output of `gluster volume info c_glusterfs` ?



#
#
# Brick1 gluster volume heal c_glusterfs info
Brick oamhost:/opt/lvmdir/c2/brick/
/logfiles/security/EVENT_LOG.xml
Number of entries: 1

Brick oamhost:/opt/lvmdir/c2/brick/
Number of entries: 0

We couldn't collect getfattr output on parent directory.
We will try to reproduce the problem again and provide you the logs.




Thanks, a reproducer +logs would really be helpful.



Meanwhile, can you please check and let us know
why gluster volume heal c_glusterfs info split-brain is showing 
number of entries in split-brain as zero?


Because the gfid-string of the parent directory seems to be missing from 
the brick/.glusterfs/indices/xattrop of the bricks.



Regards,
Anusha

On Wed, Aug 12, 2015 at 2:08 PM, Ravishankar N ravishan...@redhat.com 
mailto:ravishan...@redhat.com wrote:




On 08/11/2015 03:28 PM, Lakshmi Anusha wrote:

From the extended attributes, an entry split-brain seems to be
appeared. Since gfid is different for the original file and its
replica.
Can you please let us know why the split-brain files are not
shown in gluster volume heal vol info split-brain command?


For gfid mismatches, 'info split-brain` should indicate that the
parent directory of that file is in split-brain. Strange that it
isn't.
If you still have the setup, can you provide the getfattr output
of the parent dir on both bricks? Also, does `gluster vol heal
vol info ` list the file or the parent dir?


Kindly let us know if any known issue exists.





___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain files not shown in gluster volume heal info split-brain command

[Ravishankar N]

On 08/11/2015 03:28 PM, Lakshmi Anusha wrote:
From the extended attributes, an entry split-brain seems to be 
appeared. Since gfid is different for the original file and its replica.
Can you please let us know why the split-brain files are not shown in 
gluster volume heal vol info split-brain command?

**
For gfid mismatches, 'info split-brain` should indicate that the parent 
directory of that file is in split-brain. Strange that it isn't.
If you still have the setup, can you provide the getfattr output of the 
parent dir on both bricks? Also, does `gluster vol heal vol info ` 
list the file or the parent dir?



Kindly let us know if any known issue exists.


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain after rebooting half of a two-node cluster

[Peter Becker]

Hi Ravi,

I updated to gluster 3.6 via PPA and that seems to work well. I did a number of 
alternating reboots with ActiveMQ under load and did not see any problem. From 
my rough measurements it seems we got a bit of a performance improvement as 
well, although I might be making that one up.

Cheers,
   Peter


From: Ravishankar N [mailto:ravishan...@redhat.com]
Sent: Wednesday, 5 August 2015 12:03 PM
To: Peter Becker; Gluster-users@gluster.org
Subject: Re: [Gluster-users] Split brain after rebooting half of a two-node 
cluster


On 08/05/2015 07:01 AM, Peter Becker wrote:
Hi Ravi,

Not easily: 3.2.5 is what comes with Ubuntu 12.04 and based on previous posts 
on this list it seems I can't just go and compile a newer one.
Hi Peter,
Would adding the PPA (https://launchpad.net/~gluster) not work? 3.5 seems to be 
available [1] for Ubuntu precise.

[1] http://www.gluster.org/pipermail/gluster-users/2015-August/023058.html



If our setup should work and an upgrade might fix the issues we see, then we 
could try Ubuntu 14.04 - it's not a supported environment in our organisation 
yet, but we might be able to push that.

Given that it's a bit involved: are there other venues we could try first?

Cheers,
  Peter


From: Ravishankar N [mailto:ravishan...@redhat.com]
Sent: Wednesday, 5 August 2015 11:12 AM
To: Peter Becker; Gluster-users@gluster.orgmailto:Gluster-users@gluster.org
Subject: Re: [Gluster-users] Split brain after rebooting half of a two-node 
cluster


On 08/05/2015 04:49 AM, Peter Becker wrote:
qmaster@srvamqpy01:~$ gluster --version
glusterfs 3.2.5 built on Jan 31 2012 07:39:59
FWIW, this is a rather old release. Can you see if the issue is recurring with 
glusterfs 3.7?

-Ravi

  


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain after rebooting half of a two-node cluster

[Ravishankar N]

On 08/05/2015 01:04 PM, Peter Becker wrote:


Hi Ravi,

I updated to gluster 3.6 via PPA and that seems to work well. I did a 
number of alternating reboots with ActiveMQ under load and did not see 
any problem. From my rough measurements it seems we got a bit of a 
performance improvement as well, although I might be making that one up.




Glad it worked out for you Peter.


Cheers,

Peter

*From:*Ravishankar N [mailto:ravishan...@redhat.com]
*Sent:* Wednesday, 5 August 2015 12:03 PM
*To:* Peter Becker; Gluster-users@gluster.org
*Subject:* Re: [Gluster-users] Split brain after rebooting half of a 
two-node cluster


On 08/05/2015 07:01 AM, Peter Becker wrote:

Hi Ravi,

Not easily: 3.2.5 is what comes with Ubuntu 12.04 and based on
previous posts on this list it seems I can’t just go and compile a
newer one.

Hi Peter,
Would adding the PPA (https://launchpad.net/~gluster 
https://launchpad.net/%7Egluster) not work? 3.5 seems to be 
available [1] for Ubuntu precise.


[1] http://www.gluster.org/pipermail/gluster-users/2015-August/023058.html


If our setup should work and an upgrade might fix the issues we
see, then we could try Ubuntu 14.04 – it’s not a supported
environment in our organisation yet, but we might be able to push
that.

Given that it’s a bit involved: are there other venues we could
try first?

Cheers,

Peter

*From:*Ravishankar N [mailto:ravishan...@redhat.com]
*Sent:* Wednesday, 5 August 2015 11:12 AM
*To:* Peter Becker; Gluster-users@gluster.org
mailto:Gluster-users@gluster.org
*Subject:* Re: [Gluster-users] Split brain after rebooting half of
a two-node cluster

On 08/05/2015 04:49 AM, Peter Becker wrote:

qmaster@srvamqpy01:~$ gluster --version

glusterfs 3.2.5 built on Jan 31 2012 07:39:59

FWIW, this is a rather old release. Can you see if the issue is
recurring with glusterfs 3.7?

-Ravi


  ­­



___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain after rebooting half of a two-node cluster

[Ravishankar N]

On 08/05/2015 04:49 AM, Peter Becker wrote:


qmaster@srvamqpy01:~$ gluster --version

glusterfs 3.2.5 built on Jan 31 2012 07:39:59

FWIW, this is a rather old release. Can you see if the issue is 
recurring with glusterfs 3.7?


-Ravi
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain after rebooting half of a two-node cluster

[Peter Becker]

Hi Ravi,

Not easily: 3.2.5 is what comes with Ubuntu 12.04 and based on previous posts 
on this list it seems I can't just go and compile a newer one.

If our setup should work and an upgrade might fix the issues we see, then we 
could try Ubuntu 14.04 - it's not a supported environment in our organisation 
yet, but we might be able to push that.

Given that it's a bit involved: are there other venues we could try first?

Cheers,
  Peter


From: Ravishankar N [mailto:ravishan...@redhat.com]
Sent: Wednesday, 5 August 2015 11:12 AM
To: Peter Becker; Gluster-users@gluster.org
Subject: Re: [Gluster-users] Split brain after rebooting half of a two-node 
cluster


On 08/05/2015 04:49 AM, Peter Becker wrote:
qmaster@srvamqpy01:~$ gluster --version
glusterfs 3.2.5 built on Jan 31 2012 07:39:59
FWIW, this is a rather old release. Can you see if the issue is recurring with 
glusterfs 3.7?

-Ravi

___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain after rebooting half of a two-node cluster

[Peter Becker]

Thanks, Ravi - I was not aware of that.

I'll update and try again.

   Peter

From: Ravishankar N [mailto:ravishan...@redhat.com]
Sent: Wednesday, 5 August 2015 12:03 PM
To: Peter Becker; Gluster-users@gluster.org
Subject: Re: [Gluster-users] Split brain after rebooting half of a two-node 
cluster


On 08/05/2015 07:01 AM, Peter Becker wrote:
Hi Ravi,

Not easily: 3.2.5 is what comes with Ubuntu 12.04 and based on previous posts 
on this list it seems I can't just go and compile a newer one.
Hi Peter,
Would adding the PPA (https://launchpad.net/~gluster) not work? 3.5 seems to be 
available [1] for Ubuntu precise.

[1] http://www.gluster.org/pipermail/gluster-users/2015-August/023058.html



If our setup should work and an upgrade might fix the issues we see, then we 
could try Ubuntu 14.04 - it's not a supported environment in our organisation 
yet, but we might be able to push that.

Given that it's a bit involved: are there other venues we could try first?

Cheers,
  Peter


From: Ravishankar N [mailto:ravishan...@redhat.com]
Sent: Wednesday, 5 August 2015 11:12 AM
To: Peter Becker; Gluster-users@gluster.orgmailto:Gluster-users@gluster.org
Subject: Re: [Gluster-users] Split brain after rebooting half of a two-node 
cluster


On 08/05/2015 04:49 AM, Peter Becker wrote:
qmaster@srvamqpy01:~$ gluster --version
glusterfs 3.2.5 built on Jan 31 2012 07:39:59
FWIW, this is a rather old release. Can you see if the issue is recurring with 
glusterfs 3.7?

-Ravi

  


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain after rebooting half of a two-node cluster

[Ravishankar N]

On 08/05/2015 07:01 AM, Peter Becker wrote:


Hi Ravi,

Not easily: 3.2.5 is what comes with Ubuntu 12.04 and based on 
previous posts on this list it seems I can’t just go and compile a 
newer one.



Hi Peter,
Would adding the PPA (https://launchpad.net/~gluster) not work? 3.5 
seems to be available [1] for Ubuntu precise.


[1] http://www.gluster.org/pipermail/gluster-users/2015-August/023058.html

If our setup should work and an upgrade might fix the issues we see, 
then we could try Ubuntu 14.04 – it’s not a supported environment in 
our organisation yet, but we might be able to push that.


Given that it’s a bit involved: are there other venues we could try 
first?


Cheers,

Peter

*From:*Ravishankar N [mailto:ravishan...@redhat.com]
*Sent:* Wednesday, 5 August 2015 11:12 AM
*To:* Peter Becker; Gluster-users@gluster.org
*Subject:* Re: [Gluster-users] Split brain after rebooting half of a 
two-node cluster


On 08/05/2015 04:49 AM, Peter Becker wrote:

qmaster@srvamqpy01:~$ gluster --version

glusterfs 3.2.5 built on Jan 31 2012 07:39:59

FWIW, this is a rather old release. Can you see if the issue is 
recurring with glusterfs 3.7?


-Ravi


  ­­ 


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain.. okay, what next?

[Ravishankar N]

On 07/16/2015 01:28 AM, Игорь Бирюлин wrote:

I have studied information on page:
https://github.com/gluster/glusterfs/blob/master/doc/features/heal-info-and-split-brain-resolution.md
and cannot solve split-brain by this instruction.

I have tested it on gluster 3.6 and it doesn't work, only on gluster 3.7.



Right. We need to explicitly mention in the .md that it is supported 
from 3.7 onwards.



I try to use on gluster 3.7.2.
I have a gluster share in replicate mode:
root@dist-gl2:/# gluster volume info

Volume Name: repofiles
Type: Replicate
Volume ID: 1d5d5d7d-39f2-4011-9fc8-d73c29495e7c
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: dist-gl1:/brick1
Brick2: dist-gl2:/brick1
Options Reconfigured:
performance.readdir-ahead: on
server.allow-insecure: on
root@dist-gl2:/#

And I have one file in split-brain (it is file test):
root@dist-gl2:/# gluster volume heal repofiles info
Brick dist-gl1:/brick1/
/test
/ - Is in split-brain

Number of entries: 2

Brick dist-gl2:/brick1/
/ - Is in split-brain

/test
Number of entries: 2

root@dist-gl2:/# gluster volume heal repofiles info split-brain
Brick dist-gl1:/brick1/
/
Number of entries in split-brain: 1

Brick dist-gl2:/brick1/
/
Number of entries in split-brain: 1

root@dist-gl2:/#

I don't know why these commands show only directory (/) in split-brain.


That is because the file is in gfid split-brain. As listed in the .md 
file,  for a gfid split-brain, the parent directory of the file is 
shown to be in split-brain and the file itself is shown to be needing 
heal. You cannot resolve gfid split-brains using the commands. You need 
to resolve them manually. See Fixing Directory entry split-brain in 
https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md




I try solve split-brain by gluster cli commands (on directory from the 
output previous commands and on file), but it could not help:

root@dist-gl2:/# gluster v heal repofiles split-brain bigger-file /
Healing / failed:Operation not permitted.
Volume heal failed.
root@dist-gl2:/# gluster v heal repofiles split-brain bigger-file /test
Lookup failed on /test:Input/output error
Volume heal failed.
root@dist-gl2:/# gluster v heal repofiles split-brain source-brick 
dist-gl1:/brick1 /

Healing / failed:Operation not permitted.
Volume heal failed.
root@dist-gl2:/# gluster v heal repofiles split-brain source-brick 
dist-gl1:/brick1 /test

Lookup failed on /test:Input/output error
Volume heal failed.
root@dist-gl2:/# gluster v heal repofiles split-brain source-brick 
dist-gl2:/brick1 /

Healing / failed:Operation not permitted.
Volume heal failed.
root@dist-gl2:/# gluster v heal repofiles split-brain source-brick 
dist-gl2:/brick1 /test

Lookup failed on /test:Input/output error
Volume heal failed.
root@dist-gl2:/#

Parts of glfsheal-repofiles.log logs.
When try to solve split-brain on dirictory (/):
[2015-07-15 19:45:30.508670] I 
[event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started 
thread with index 1
[2015-07-15 19:45:30.516662] I 
[event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started 
thread with index 2
[2015-07-15 19:45:30.517201] I [MSGID: 104045] 
[glfs-master.c:95:notify] 0-gfapi: New graph 
64697374-2d67-6c32-2d32-303634362d32 (0) coming up
[2015-07-15 19:45:30.517227] I [MSGID: 114020] [client.c:2118:notify] 
0-repofiles-client-0: parent translators are ready, attempting connect 
on transport
[2015-07-15 19:45:30.525457] I [MSGID: 114020] [client.c:2118:notify] 
0-repofiles-client-1: parent translators are ready, attempting connect 
on transport
[2015-07-15 19:45:30.526788] I [rpc-clnt.c:1819:rpc_clnt_reconfig] 
0-repofiles-client-0: changing port to 49152 (from 0)
[2015-07-15 19:45:30.534012] I [rpc-clnt.c:1819:rpc_clnt_reconfig] 
0-repofiles-client-1: changing port to 49152 (from 0)
[2015-07-15 19:45:30.536252] I [MSGID: 114057] 
[client-handshake.c:1438:select_server_supported_programs] 
0-repofiles-client-0: Using Program GlusterFS 3.3, Num (1298437), 
Version (330)
[2015-07-15 19:45:30.536606] I [MSGID: 114046] 
[client-handshake.c:1214:client_setvolume_cbk] 0-repofiles-client-0: 
Connected to repofiles-client-0, attached to remote volume '/brick1'.
[2015-07-15 19:45:30.536621] I [MSGID: 114047] 
[client-handshake.c:1225:client_setvolume_cbk] 0-repofiles-client-0: 
Server and Client lk-version numbers are not same, reopening the fds
[2015-07-15 19:45:30.536679] I [MSGID: 108005] 
[afr-common.c:3883:afr_notify] 0-repofiles-replicate-0: Subvolume 
'repofiles-client-0' came back up; going online.
[2015-07-15 19:45:30.536819] I [MSGID: 114035] 
[client-handshake.c:193:client_set_lk_version_cbk] 
0-repofiles-client-0: Server lk version = 1
[2015-07-15 19:45:30.543712] I [MSGID: 114057] 
[client-handshake.c:1438:select_server_supported_programs] 
0-repofiles-client-1: Using Program GlusterFS 3.3, Num (1298437), 
Version (330)
[2015-07-15 19:45:30.543919] I [MSGID: 114046] 
[client-handshake.c:1214:client_setvolume_cbk] 

Re: [Gluster-users] split-brain.. okay, what next?

[Игорь Бирюлин]

I have studied information on page:
https://github.com/gluster/glusterfs/blob/master/doc/features/heal-info-and-split-brain-resolution.md
and cannot solve split-brain by this instruction.

I have tested it on gluster 3.6 and it doesn't work, only on gluster 3.7.

I try to use on gluster 3.7.2.
I have a gluster share in replicate mode:
root@dist-gl2:/# gluster volume info

Volume Name: repofiles
Type: Replicate
Volume ID: 1d5d5d7d-39f2-4011-9fc8-d73c29495e7c
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: dist-gl1:/brick1
Brick2: dist-gl2:/brick1
Options Reconfigured:
performance.readdir-ahead: on
server.allow-insecure: on
root@dist-gl2:/#

And I have one file in split-brain (it is file test):
root@dist-gl2:/# gluster volume heal repofiles info
Brick dist-gl1:/brick1/
/test
/ - Is in split-brain

Number of entries: 2

Brick dist-gl2:/brick1/
/ - Is in split-brain

/test
Number of entries: 2

root@dist-gl2:/# gluster volume heal repofiles info split-brain
Brick dist-gl1:/brick1/
/
Number of entries in split-brain: 1

Brick dist-gl2:/brick1/
/
Number of entries in split-brain: 1

root@dist-gl2:/#

I don't know why these commands show only directory (/) in split-brain.

I try solve split-brain by gluster cli commands (on directory from the
output previous commands and on file), but it could not help:
root@dist-gl2:/# gluster v heal repofiles split-brain bigger-file /
Healing / failed:Operation not permitted.
Volume heal failed.
root@dist-gl2:/# gluster v heal repofiles split-brain bigger-file /test
Lookup failed on /test:Input/output error
Volume heal failed.
root@dist-gl2:/# gluster v heal repofiles split-brain source-brick
dist-gl1:/brick1 /
Healing / failed:Operation not permitted.
Volume heal failed.
root@dist-gl2:/# gluster v heal repofiles split-brain source-brick
dist-gl1:/brick1 /test
Lookup failed on /test:Input/output error
Volume heal failed.
root@dist-gl2:/# gluster v heal repofiles split-brain source-brick
dist-gl2:/brick1 /
Healing / failed:Operation not permitted.
Volume heal failed.
root@dist-gl2:/# gluster v heal repofiles split-brain source-brick
dist-gl2:/brick1 /test
Lookup failed on /test:Input/output error
Volume heal failed.
root@dist-gl2:/#

Parts of glfsheal-repofiles.log logs.
When try to solve split-brain on dirictory (/):
[2015-07-15 19:45:30.508670] I
[event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 1
[2015-07-15 19:45:30.516662] I
[event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 2
[2015-07-15 19:45:30.517201] I [MSGID: 104045] [glfs-master.c:95:notify]
0-gfapi: New graph 64697374-2d67-6c32-2d32-303634362d32 (0) coming up
[2015-07-15 19:45:30.517227] I [MSGID: 114020] [client.c:2118:notify]
0-repofiles-client-0: parent translators are ready, attempting connect on
transport
[2015-07-15 19:45:30.525457] I [MSGID: 114020] [client.c:2118:notify]
0-repofiles-client-1: parent translators are ready, attempting connect on
transport
[2015-07-15 19:45:30.526788] I [rpc-clnt.c:1819:rpc_clnt_reconfig]
0-repofiles-client-0: changing port to 49152 (from 0)
[2015-07-15 19:45:30.534012] I [rpc-clnt.c:1819:rpc_clnt_reconfig]
0-repofiles-client-1: changing port to 49152 (from 0)
[2015-07-15 19:45:30.536252] I [MSGID: 114057]
[client-handshake.c:1438:select_server_supported_programs]
0-repofiles-client-0: Using Program GlusterFS 3.3, Num (1298437), Version
(330)
[2015-07-15 19:45:30.536606] I [MSGID: 114046]
[client-handshake.c:1214:client_setvolume_cbk] 0-repofiles-client-0:
Connected to repofiles-client-0, attached to remote volume '/brick1'.
[2015-07-15 19:45:30.536621] I [MSGID: 114047]
[client-handshake.c:1225:client_setvolume_cbk] 0-repofiles-client-0: Server
and Client lk-version numbers are not same, reopening the fds
[2015-07-15 19:45:30.536679] I [MSGID: 108005]
[afr-common.c:3883:afr_notify] 0-repofiles-replicate-0: Subvolume
'repofiles-client-0' came back up; going online.
[2015-07-15 19:45:30.536819] I [MSGID: 114035]
[client-handshake.c:193:client_set_lk_version_cbk] 0-repofiles-client-0:
Server lk version = 1
[2015-07-15 19:45:30.543712] I [MSGID: 114057]
[client-handshake.c:1438:select_server_supported_programs]
0-repofiles-client-1: Using Program GlusterFS 3.3, Num (1298437), Version
(330)
[2015-07-15 19:45:30.543919] I [MSGID: 114046]
[client-handshake.c:1214:client_setvolume_cbk] 0-repofiles-client-1:
Connected to repofiles-client-1, attached to remote volume '/brick1'.
[2015-07-15 19:45:30.543933] I [MSGID: 114047]
[client-handshake.c:1225:client_setvolume_cbk] 0-repofiles-client-1: Server
and Client lk-version numbers are not same, reopening the fds
[2015-07-15 19:45:30.554650] I [MSGID: 114035]
[client-handshake.c:193:client_set_lk_version_cbk] 0-repofiles-client-1:
Server lk version = 1
[2015-07-15 19:45:30.557628] I
[afr-self-heal-entry.c:565:afr_selfheal_entry_do] 0-repofiles-replicate-0:
performing entry selfheal on ----0001
[2015-07-15 

Re: [Gluster-users] split-brain.. okay, what next?

[Joe Julian]

On 07/14/2015 11:19 AM, Roman wrote:

Hi,

played with glusterfs tonight and tried to use recommended XFS for 
gluster.. first try was pretty bad and all of my VM-s hanged (XFS 
wants allocsize=64k to create qcow2 files, which i didn't know about 
and tried to create VM on XFS without this config line in fstab, which 
lead to a lot of IO-s and qemu says it got time out while creating the 
file)..


now i've got this:
Brick stor1:/exports/HA-2TB-TT-Proxmox-cluster/2TB/
/images/124/vm-124-disk-1.qcow2 - Is in split-brain

Number of entries: 1

Brick stor2:/exports/HA-2TB-TT-Proxmox-cluster/2TB/
/images/124/vm-124-disk-1.qcow2 - Is in split-brain

ok. what next?
I've deleted one of files, it didn't help. even more, selfheal 
restored the file on node, where i deleted it... and still split-brain.


how to fix?

--
Best regards,
Roman.




https://github.com/gluster/glusterfs/blob/master/doc/features/heal-info-and-split-brain-resolution.md 



or

https://joejulian.name/blog/glusterfs-split-brain-recovery-made-easy/
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain.. okay, what next?

[Roman]

never mind. I do not have enough time to debug why basic commands of
gluster do not work on production server. It was enough of tonight's system
freeze due to not documented XFS settings MUST have to run glusterfs with
XFS. I'll keep to EXT4 better. Anyway XFS for bricks did not solve my
previous problem.

To solve split-brain this time, I've restored VM from backup.

2015-07-14 21:55 GMT+03:00 Roman rome...@gmail.com:

 Thanx for pointing out...
 but doesn't seem to work... or i am too sleepy due to problems with
 glusterfs and debian8 in other topic which i'm fighting for month..

 root@stor1:~# gluster volume heal HA-2TB-TT-Proxmox-cluster split-brain
 source-brick stor1:HA-2TB-TT-Proxmox-cluster/2TB
 /images/124/vm-124-disk-1.qcow2
 Usage: volume heal VOLNAME [{full | statistics {heal-count {replica
 hostname:brickname}} |info {healed | heal-failed | split-brain}}]

 seems like wrong command...

 2015-07-14 21:23 GMT+03:00 Joe Julian j...@julianfamily.org:

 On 07/14/2015 11:19 AM, Roman wrote:

 Hi,

 played with glusterfs tonight and tried to use recommended XFS for
 gluster.. first try was pretty bad and all of my VM-s hanged (XFS wants
 allocsize=64k to create qcow2 files, which i didn't know about and tried to
 create VM on XFS without this config line in fstab, which lead to a lot of
 IO-s and qemu says it got time out while creating the file)..

 now i've got this:
 Brick stor1:/exports/HA-2TB-TT-Proxmox-cluster/2TB/
 /images/124/vm-124-disk-1.qcow2 - Is in split-brain

 Number of entries: 1

 Brick stor2:/exports/HA-2TB-TT-Proxmox-cluster/2TB/
 /images/124/vm-124-disk-1.qcow2 - Is in split-brain

 ok. what next?
 I've deleted one of files, it didn't help. even more, selfheal restored
 the file on node, where i deleted it... and still split-brain.

 how to fix?

 --
 Best regards,
 Roman.




 https://github.com/gluster/glusterfs/blob/master/doc/features/heal-info-and-split-brain-resolution.md

 or

 https://joejulian.name/blog/glusterfs-split-brain-recovery-made-easy/
 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 http://www.gluster.org/mailman/listinfo/gluster-users




 --
 Best regards,
 Roman.




-- 
Best regards,
Roman.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split brain on / just after installation

[Ravishankar N]

On 06/02/2015 09:10 AM, Carl L Hoffman wrote:

Hello - I was wondering if someone could please help me.

I've just setup Gluster 3.6 on two Ubuntu 14.04 hosts.  Gluster is setup to 
replicate two volumes (prod-volume, dev-volume) between the two hosts.  
Replication is working fine.  The glustershd.log shows:


Are you sure you are running gluster 3.6?  The 
'afr_sh_print_split_brain_log' message appears only in gluster 3.5 or lower.





[2015-06-02 03:28:04.495162] E 
[afr-self-heal-common.c:197:afr_sh_print_split_brain_log] 0-prod-volume-replicate-0: 
Unable to self-heal contents of 'gfid:----0001' 
(possible split-brain). Please delete the file from all but the preferred subvolume.- 
Pending matrix:  [ [ 0 2 ] [ 2 0 ] ]

and the prod-volume logs shows:

[2015-06-02 02:54:28.286268] E 
[afr-self-heal-common.c:197:afr_sh_print_split_brain_log] 
0-prod-volume-replicate-0: Unable to self-heal contents of '/' (possible 
split-brain). Please delete the file from all but the preferred subvolume.- 
Pending matrix:  [ [ 0 2 ] [ 2 0 ] ]
[2015-06-02 02:54:28.287476] E 
[afr-self-heal-common.c:2212:afr_self_heal_completion_cbk] 
0-prod-volume-replicate-0: background  meta-data self-heal failed on /

I've checked against 
https://github.com/gluster/glusterfs/blob/6c578c03f0d44913d264494de5df004544c96271/doc/features/heal-info-and-split-brain-resolution.md
 but I can't see any scenario that covers mine.  The output of bluster volume 
heal prod-volume info is:



Is the metadata same on both bricks on the root?  (Compare  `ls -ld 
/export/prodvol/brick`  and `getfattr -d -m . -e hex 
/export/prodvol/brick` on both servers to see if anything is mismatching).

-Ravi



Gathering Heal info on volume prod-volume has been successful

Brick server1:/export/prodvol/brick
Number of entries: 1
/

Brick server2
Number of entries: 1
/


and doesn't show anything in split-brain.

But the output of gluster volume heal prod-volume info split brain shows:

Gathering Heal info on volume prod-volume has been successful

Brick server1:/export/prodvol/brick
Number of entries: 6
atpath on brick
---
2015-06-02 03:28:04 /
2015-06-02 03:18:04 /
2015-06-02 03:08:04 /
2015-06-02 02:58:04 /
2015-06-02 02:48:04 /
2015-06-02 02:48:04 /

Brick server2:/export/prodvol/brick
Number of entries: 5
atpath on brick
---
2015-06-02 03:28:00 /
2015-06-02 03:18:00 /
2015-06-02 03:08:00 /
2015-06-02 02:58:00 /
2015-06-02 02:48:04 /


And the number continues to grow.  The count on server2 is always one behind 
server1.

Could someone please help?

Cheers,


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split-brain info gluster 3.6

[Ravishankar N]

This is fixed in http://review.gluster.org/9459 and should be available 
in 3.7.
As a workaround, you can restart the selfheal daemon process (gluster v 
start volname force). This should clear its history.


Thanks,
Ravi

On 02/20/2015 01:43 PM, Félix de Lelelis wrote:

Hi,

I generated a split-brain condition, and I solved it but with gluster 
volume heal vol_name info split-brain  yet I can see past entries solved:


Number of entries: 3
atpath on brick
---
2015-02-19 17:13:08 /split
2015-02-19 17:14:09 /split
2015-02-19 17:15:10 /split

Brick srv-vln-des2-priv1:/gfs-to-snap/prueba/brick1/brick
Number of entries: 4
atpath on brick
---
2015-02-19 17:09:32 /split
2015-02-19 17:13:08 /split
2015-02-19 17:14:09 /split
2015-02-19 17:15:10 /split

How can I reset that entries?

Thanks


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain with quorum enabled

[Alexey]

Hi,

Sorry for the delay, it took a long time to reproduce. But currently we
have the same issue again. This was happened afrer reseting all nodes.
Quorum is enabled. Logs and details below.

gluster volume heal vm_storage_volume info split-brain
Gathering list of split brain entries on volume vm_storage_volume has been
successful

Brick svm1:/srv/vol
Number of entries: 0

Brick svm2:/srv/vol
Number of entries: 0

Brick svm3:/srv/vol
Number of entries: 0

Brick svm4:/srv/vol
Number of entries: 0

Brick svm5:/srv/vol
Number of entries: 0

Brick svm6:/srv/vol
Number of entries: 2
atpath on brick
---
2015-01-22 09:08:47 /vm_images_and_config/vm9.img
2015-01-22 09:11:52 /vm_images_and_config/vm20.img

Brick svm7:/srv/vol
Number of entries: 0


gluster volume heal vm_storage_volume statistics
Gathering crawl statistics on volume vm_storage_volume has been successful


Crawl statistics for brick no 0
Hostname of brick svm1

Starting time of crawl: Thu Jan 22 12:53:28 2015

Crawl is in progress
Type of crawl: INDEX
No. of entries healed: 1
No. of entries in split-brain: 0
No. of heal failed entries: 2


Crawl statistics for brick no 1
Hostname of brick svm2

Starting time of crawl: Thu Jan 22 13:02:44 2015

Crawl is in progress
Type of crawl: INDEX
No. of entries healed: 1
No. of entries in split-brain: 0
No. of heal failed entries: 3


Crawl statistics for brick no 2
Hostname of brick svm3

Starting time of crawl: Thu Jan 22 13:11:17 2015

Crawl is in progress
Type of crawl: INDEX
No. of entries healed: 0
No. of entries in split-brain: 0
No. of heal failed entries: 0


Crawl statistics for brick no 3
Hostname of brick svm4

Starting time of crawl: Thu Jan 22 13:11:48 2015

Crawl is in progress
Type of crawl: INDEX
No. of entries healed: 0
No. of entries in split-brain: 0
No. of heal failed entries: 1


Crawl statistics for brick no 4
Hostname of brick svm5

Starting time of crawl: Thu Jan 22 12:55:52 2015

Crawl is in progress
Type of crawl: INDEX
No. of entries healed: 0
No. of entries in split-brain: 0
No. of heal failed entries: 3


Crawl statistics for brick no 5
Hostname of brick svm6

Starting time of crawl: Thu Jan 22 12:53:23 2015

Crawl is in progress
Type of crawl: INDEX
No. of entries healed: 0
No. of entries in split-brain: 2
No. of heal failed entries: 2


Crawl statistics for brick no 6
Hostname of brick svm7

Starting time of crawl: Thu Jan 22 13:24:08 2015

Crawl is in progress
Type of crawl: INDEX
No. of entries healed: 0
No. of entries in split-brain: 0
No. of heal failed entries: 1


[2015-01-22 09:11:51.316542] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
0-vm_storage_volume-client-3: changing port to 49216 (from 0)
[2015-01-22 09:11:51.317179] I
[client-handshake.c:1677:select_server_supported_programs]
0-vm_storage_volume-client-3: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2015-01-22 09:11:51.317459] I
[client-handshake.c:1462:client_setvolume_cbk]
0-vm_storage_volume-client-3: Connected to 11.2.1.204:49216, attached to
remote volume '/srv/vol'.
[2015-01-22 09:11:51.317493] I
[client-handshake.c:1474:client_setvolume_cbk]
0-vm_storage_volume-client-3: Server and Client lk-version numbers are not
same, reopening the fds
[2015-01-22 09:11:51.317528] I
[client-handshake.c:1314:client_post_handshake]
0-vm_storage_volume-client-3: 1 fds open - Delaying child_up until they are
re-opened
[2015-01-22 09:11:51.352698] I
[client-handshake.c:936:client_child_up_reopen_done]
0-vm_storage_volume-client-3: last fd open'd/lock-self-heal'd - notifying
CHILD-UP
[2015-01-22 09:11:51.355238] I
[client-handshake.c:450:client_set_lk_version_cbk]
0-vm_storage_volume-client-3: Server lk version = 1
[2015-01-22 09:11:51.357918] I
[afr-self-heald.c:1690:afr_dir_exclusive_crawl]
0-vm_storage_volume-replicate-0: Another crawl is in progress for
vm_storage_volume-client-5
[2015-01-22 09:11:52.299413] E
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
0-vm_storage_volume-replicate-0: Unable to self-heal contents of
'gfid:f7b77d22-9606-4141-943c-b738aa2a21fc' (possible split-brain).
Please delete the file from all but the preferred subvolume.- Pending
matrix:  [ [ 0 2451 650 0 2 1452 5405 ] [ 12 1 64 1 3 1453 3551 ] [ 0 0 0 0
0 0 0 ] [ 0 0 0 0 0 0 0 ] [ 11 2441 650 0 0 1452 5403 ] [ 12 2442 651 1 3 1
3953 ] [ 0 0 0 0 0 0 0 ] ]


[2015-01-22 09:08:47.105262] E
[client-rpc-fops.c:1533:client3_3_inodelk_cbk]
0-vm_storage_volume-client-2: remote operation failed: Transport endpoint
is not connected
[2015-01-22 09:08:47.105309] E
[client-rpc-fops.c:1533:client3_3_inodelk_cbk]
0-vm_storage_volume-client-3: remote operation 

Re: [Gluster-users] split-brain with quorum enabled

[Pranith Kumar Karampuri]

On 12/25/2014 08:05 PM, Alexey wrote:

Hi all,

We are using glusterfs setup with a quorum turned on and the 
configuration as the follows:


Nodes: 3
Type: Replicate
Number of Bricks: 1 x 3 = 3
cluster.quorum-type: fixed
cluster.quorum-count: 2
cluster.data-self-heal-algorithm: diff
cluster.server-quorum-ratio: 51%
glusterfs version: 3.5.3

Despite on the quorum is turned on sometimes we are still encounter a 
split-brain occurrence after shutting down one node or all nodes together/


Is this is a normal behavior? What conditions could lead to this and 
how to prevent split-brain occurence?

Could you please describe what is the kind of split-brain that happened?

Pranith


BR,
Alexey


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors

[Anirban Ghoshal]

Ok, no problem. The issue is very rare, even with our setup - we have seen it 
only once on one site even though we have been in production for several months 
now. For now, we can live with that IMO. 

And, thanks again. 

Anirban___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors

[Anirban Ghoshal]

It is possible, yes, because these are actually a kind of log files. I suppose, 
like other logging frameworks these files an remain open for a considerable 
period, and then get renamed to support log rotate semantics. 

That said, I might need to check with the team that actually manages the 
logging framework to be sure. I only take care of the file-system stuff. I can 
tell you for sure Monday. 

If it is the same race that you mention, is there a fix for it?

Thanks,
Anirban___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors

[Pranith Kumar Karampuri]

On 10/19/2014 01:36 PM, Anirban Ghoshal wrote:
It is possible, yes, because these are actually a kind of log files. I 
suppose, like other logging frameworks these files an remain open for 
a considerable period, and then get renamed to support log rotate 
semantics.


That said, I might need to check with the team that actually manages 
the logging framework to be sure. I only take care of the file-system 
stuff. I can tell you for sure Monday.


If it is the same race that you mention, is there a fix for it?

Thanks,
Anirban



I am working on the fix.

RCA:
0) Lets say the file 'abc.log' is opened for writing on replica pair 
(brick-0, brick-1)

1) brick-0 went down
2) abc.log is renamed to abc.log.1
3) brick-0 comes back up
4) re-open on old abc.log happens from mount to brick-0
5) self-heal kicks in and deletes old abc.log and creates and syncs 
abc.log.1
6) But the mount is still writing to the deleted 'old abc.log' on 
brick-0 so abc.log.1 file remains at the same size while abc.log.1 file 
keeps increasing on brick-1. This leads to size mismatch split-brain on 
abc.log.1.


Race happens between steps 4), 5). If 5) happens before 4) no 
split-brain will be observed.


Work-around:

0) Take backup of good abc.log.1 file from brick-1. (Just being paranoid)

Do any of the following two steps to make sure the stale file that is 
open is closed
1-a) Take the brick process with bad file down using kill -9 brick-pid 
(In my example brick-0).

1-b) Introduce a temporary disconnect between mount and brick-0.
(I would choose 1-a)
2) Remove the bad file(abc.log.1) and its gfid-backend-file from brick-0
3) Bring the brick back up (gluster volume start volname 
force)/restore the connection and let it heal by doing 'stat' on the 
file abc.log.1 on the mount.


This bug existed from 2012, from the first time I implemented 
rename/hard-link self-heal. It is difficult to re-create. I have to put 
break-points at several places in the process to hit the race.


Pranith


Thanks,
Anirban


*From: * Pranith Kumar Karampuri pkara...@redhat.com;
*To: * Anirban Ghoshal chalcogen_eg_oxy...@yahoo.com; 
gluster-users@gluster.org;
*Subject: * Re: [Gluster-users] Split-brain seen with [0 0] pending 
matrix and io-cache page errors

*Sent: * Sun, Oct 19, 2014 5:42:24 AM


On 10/18/2014 04:36 PM, Anirban Ghoshal wrote:

Hi,

Yes, they do, and considerably. I'd forgotten to mention that on my 
last email. Their mtimes, however, as far as i could tell on separate 
servers, seemed to coincide.


Thanks,
Anirban




Are these files always open? And is it possible that the file could 
have been renamed when one of the bricks was offline? I know of a race 
which can introduce this one. Just trying to find if it is the same case.


Pranith




*From: * Pranith Kumar Karampuri pkara...@redhat.com;
*To: * Anirban Ghoshal chalcogen_eg_oxy...@yahoo.com; 
gluster-users@gluster.org gluster-users@gluster.org;
*Subject: * Re: [Gluster-users] Split-brain seen with [0 0] pending 
matrix and io-cache page errors

*Sent: * Sat, Oct 18, 2014 12:26:08 AM

hi,
  Could you see if the size of the file mismatches?

Pranith

On 10/18/2014 04:20 AM, Anirban Ghoshal wrote:

Hi everyone,

I have this really confusing split-brain here that's bothering me. I 
am running glusterfs 3.4.2 over linux 2.6.34. I have a replica 2 
volume 'testvol' that is It seems I cannot read/stat/edit the file 
in question, and `gluster volume heal testvol info split-brain` 
shows nothing. Here are the logs from the fuse-mount for the volume:


[2014-09-29 07:53:02.867111] W [fuse-bridge.c:1172:fuse_err_cbk] 
0-glusterfs-fuse: 4560969: FLUSH() ERR = -1 (Input/output error)
[2014-09-29 07:54:16.007799] W [page.c:991:__ioc_page_error] 
0-testvol-io-cache: page error for page = 0x7fd5c8529d20  waitq = 
0x7fd5c8067d40
[2014-09-29 07:54:16.007854] W [fuse-bridge.c:2089:fuse_readv_cbk] 
0-glusterfs-fuse: 4561103: READ = -1 (Input/output error)
[2014-09-29 07:54:16.008018] W [page.c:991:__ioc_page_error] 
0-testvol-io-cache: page error for page = 0x7fd5c8607ee0  waitq = 
0x7fd5c8067d40
[2014-09-29 07:54:16.008056] W [fuse-bridge.c:2089:fuse_readv_cbk] 
0-glusterfs-fuse: 4561104: READ = -1 (Input/output error)
[2014-09-29 07:54:16.008233] W [page.c:991:__ioc_page_error] 
0-testvol-io-cache: page error for page = 0x7fd5c8066f30  waitq = 
0x7fd5c8067d40
[2014-09-29 07:54:16.008269] W [fuse-bridge.c:2089:fuse_readv_cbk] 
0-glusterfs-fuse: 4561105: READ = -1 (Input/output error)
[2014-09-29 07:54:16.008800] W [page.c:991:__ioc_page_error] 
0-testvol-io-cache: page error for page = 0x7fd5c860bcf0  waitq = 
0x7fd5c863b1f0
[2014-09-29 07:54:16.008839] W [fuse-bridge.c:2089:fuse_readv_cbk] 
0-glusterfs-fuse: 4561107: READ = -1 (Input/output error)
[2014-09-29 07:54:16.009365] W [page.c:991:__ioc_page_error] 
0-testvol-io-cache: page error

Re: [Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors

[Anirban Ghoshal]

I see. Thanks a tonne for the thorough explanation! :) I can see that our setup 
would be vulnerable here because the logger on one server is not generally 
aware of the state of the replica on the other server. So, it is possible that 
the log files may have been renamed before heal had a chance to kick in. 

Could I also request you for the bug ID (should there be one) against which you 
are coding up the fix, so that we could get a notification once it is passed?

Also, as an aside, is O_DIRECT supposed to prevent this from occurring if one 
were to make allowance for the performance hit? 

Thanks again,
Anirban___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors

[Pranith Kumar Karampuri]

On 10/19/2014 06:05 PM, Anirban Ghoshal wrote:
I see. Thanks a tonne for the thorough explanation! :) I can see that 
our setup would be vulnerable here because the logger on one server is 
not generally aware of the state of the replica on the other server. 
So, it is possible that the log files may have been renamed before 
heal had a chance to kick in.


Could I also request you for the bug ID (should there be one) against 
which you are coding up the fix, so that we could get a notification 
once it is passed?


This bug was reported by Redhat QE and the bug is cloned upstream. I 
copied the relevant content so you would understand the context:

https://bugzilla.redhat.com/show_bug.cgi?id=1154491

Pranith


Also, as an aside, is O_DIRECT supposed to prevent this from occurring 
if one were to make allowance for the performance hit?



Unfortunately no :-(. As far as I understand that was the only work-around.

Pranith


Thanks again,
Anirban



*From: * Pranith Kumar Karampuri pkara...@redhat.com;
*To: * Anirban Ghoshal chalcogen_eg_oxy...@yahoo.com; 
gluster-users@gluster.org;
*Subject: * Re: [Gluster-users] Split-brain seen with [0 0] pending 
matrix and io-cache page errors

*Sent: * Sun, Oct 19, 2014 9:01:58 AM


On 10/19/2014 01:36 PM, Anirban Ghoshal wrote:
It is possible, yes, because these are actually a kind of log files. 
I suppose, like other logging frameworks these files an remain open 
for a considerable period, and then get renamed to support log rotate 
semantics.


That said, I might need to check with the team that actually manages 
the logging framework to be sure. I only take care of the file-system 
stuff. I can tell you for sure Monday.


If it is the same race that you mention, is there a fix for it?

Thanks,
Anirban



I am working on the fix.

RCA:
0) Lets say the file 'abc.log' is opened for writing on replica pair 
(brick-0, brick-1)

1) brick-0 went down
2) abc.log is renamed to abc.log.1
3) brick-0 comes back up
4) re-open on old abc.log happens from mount to brick-0
5) self-heal kicks in and deletes old abc.log and creates and syncs 
abc.log.1
6) But the mount is still writing to the deleted 'old abc.log' on 
brick-0 so abc.log.1 file remains at the same size while abc.log.1 
file keeps increasing on brick-1. This leads to size mismatch 
split-brain on abc.log.1.


Race happens between steps 4), 5). If 5) happens before 4) no 
split-brain will be observed.


Work-around:

0) Take backup of good abc.log.1 file from brick-1. (Just being paranoid)

Do any of the following two steps to make sure the stale file that is 
open is closed
1-a) Take the brick process with bad file down using kill -9 
brick-pid (In my example brick-0).

1-b) Introduce a temporary disconnect between mount and brick-0.
(I would choose 1-a)
2) Remove the bad file(abc.log.1) and its gfid-backend-file from brick-0
3) Bring the brick back up (gluster volume start volname 
force)/restore the connection and let it heal by doing 'stat' on the 
file abc.log.1 on the mount.


This bug existed from 2012, from the first time I implemented 
rename/hard-link self-heal. It is difficult to re-create. I have to 
put break-points at several places in the process to hit the race.


Pranith



Thanks,
Anirban


*From: * Pranith Kumar Karampuri pkara...@redhat.com;
*To: * Anirban Ghoshal chalcogen_eg_oxy...@yahoo.com; 
gluster-users@gluster.org;
*Subject: * Re: [Gluster-users] Split-brain seen with [0 0] pending 
matrix and io-cache page errors

*Sent: * Sun, Oct 19, 2014 5:42:24 AM


On 10/18/2014 04:36 PM, Anirban Ghoshal wrote:

Hi,

Yes, they do, and considerably. I'd forgotten to mention that on my 
last email. Their mtimes, however, as far as i could tell on 
separate servers, seemed to coincide.


Thanks,
Anirban




Are these files always open? And is it possible that the file could 
have been renamed when one of the bricks was offline? I know of a 
race which can introduce this one. Just trying to find if it is the 
same case.


Pranith




*From: * Pranith Kumar Karampuri pkara...@redhat.com;
*To: * Anirban Ghoshal chalcogen_eg_oxy...@yahoo.com; 
gluster-users@gluster.org gluster-users@gluster.org;
*Subject: * Re: [Gluster-users] Split-brain seen with [0 0] pending 
matrix and io-cache page errors

*Sent: * Sat, Oct 18, 2014 12:26:08 AM

hi,
  Could you see if the size of the file mismatches?

Pranith

On 10/18/2014 04:20 AM, Anirban Ghoshal wrote:

Hi everyone,

I have this really confusing split-brain here that's bothering me. 
I am running glusterfs 3.4.2 over linux 2.6.34. I have a replica 2 
volume 'testvol' that is It seems I cannot read/stat/edit the file 
in question, and `gluster volume heal testvol info split-brain` 
shows nothing. Here are the logs from the fuse-mount for the volume

Re: [Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors

[Anirban Ghoshal]

Hi,

Yes, they do, and considerably. I#39;d forgotten to mention that on my last 
email. Their mtimes, however, as far as i could tell on separate servers, 
seemed to coincide. 

Thanks,
Anirban___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors

[Pranith Kumar Karampuri]

On 10/18/2014 04:36 PM, Anirban Ghoshal wrote:

Hi,

Yes, they do, and considerably. I'd forgotten to mention that on my 
last email. Their mtimes, however, as far as i could tell on separate 
servers, seemed to coincide.


Thanks,
Anirban




Are these files always open? And is it possible that the file could have 
been renamed when one of the bricks was offline? I know of a race which 
can introduce this one. Just trying to find if it is the same case.


Pranith



*From: * Pranith Kumar Karampuri pkara...@redhat.com;
*To: * Anirban Ghoshal chalcogen_eg_oxy...@yahoo.com; 
gluster-users@gluster.org gluster-users@gluster.org;
*Subject: * Re: [Gluster-users] Split-brain seen with [0 0] pending 
matrix and io-cache page errors

*Sent: * Sat, Oct 18, 2014 12:26:08 AM

hi,
  Could you see if the size of the file mismatches?

Pranith

On 10/18/2014 04:20 AM, Anirban Ghoshal wrote:

Hi everyone,

I have this really confusing split-brain here that's bothering me. I 
am running glusterfs 3.4.2 over linux 2.6.34. I have a replica 2 
volume 'testvol' that is It seems I cannot read/stat/edit the file in 
question, and `gluster volume heal testvol info split-brain` shows 
nothing. Here are the logs from the fuse-mount for the volume:


[2014-09-29 07:53:02.867111] W [fuse-bridge.c:1172:fuse_err_cbk] 
0-glusterfs-fuse: 4560969: FLUSH() ERR = -1 (Input/output error)
[2014-09-29 07:54:16.007799] W [page.c:991:__ioc_page_error] 
0-testvol-io-cache: page error for page = 0x7fd5c8529d20  waitq = 
0x7fd5c8067d40
[2014-09-29 07:54:16.007854] W [fuse-bridge.c:2089:fuse_readv_cbk] 
0-glusterfs-fuse: 4561103: READ = -1 (Input/output error)
[2014-09-29 07:54:16.008018] W [page.c:991:__ioc_page_error] 
0-testvol-io-cache: page error for page = 0x7fd5c8607ee0  waitq = 
0x7fd5c8067d40
[2014-09-29 07:54:16.008056] W [fuse-bridge.c:2089:fuse_readv_cbk] 
0-glusterfs-fuse: 4561104: READ = -1 (Input/output error)
[2014-09-29 07:54:16.008233] W [page.c:991:__ioc_page_error] 
0-testvol-io-cache: page error for page = 0x7fd5c8066f30  waitq = 
0x7fd5c8067d40
[2014-09-29 07:54:16.008269] W [fuse-bridge.c:2089:fuse_readv_cbk] 
0-glusterfs-fuse: 4561105: READ = -1 (Input/output error)
[2014-09-29 07:54:16.008800] W [page.c:991:__ioc_page_error] 
0-testvol-io-cache: page error for page = 0x7fd5c860bcf0  waitq = 
0x7fd5c863b1f0
[2014-09-29 07:54:16.008839] W [fuse-bridge.c:2089:fuse_readv_cbk] 
0-glusterfs-fuse: 4561107: READ = -1 (Input/output error)
[2014-09-29 07:54:16.009365] W [page.c:991:__ioc_page_error] 
0-testvol-io-cache: page error for page = 0x7fd5c85fd120  waitq = 
0x7fd5c8067d40
[2014-09-29 07:54:16.009413] W [fuse-bridge.c:2089:fuse_readv_cbk] 
0-glusterfs-fuse: 4561109: READ = -1 (Input/output error)
[2014-09-29 07:54:16.040549] W [afr-open.c:213:afr_open] 
0-testvol-replicate-0: failed to open as split brain seen, returning EIO
[2014-09-29 07:54:16.040594] W [fuse-bridge.c:915:fuse_fd_cbk] 
0-glusterfs-fuse: 4561142: OPEN() 
/SECLOG/20140908.d/SECLOG_00427425_.log 
= -1 (Input/output error)


Could somebody please give me some clue on where to begin? I checked 
the xattrs on 
/SECLOG/20140908.d/SECLOG_00427425_.log 
and it seems the changelogs are [0, 0] on both replicas, and the 
gfid's match.


Thank you very much for any help on this.
Anirban





___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users




___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split-brain seen with [0 0] pending matrix and io-cache page errors

[Pranith Kumar Karampuri]

hi,
  Could you see if the size of the file mismatches?

Pranith

On 10/18/2014 04:20 AM, Anirban Ghoshal wrote:

Hi everyone,

I have this really confusing split-brain here that's bothering me. I 
am running glusterfs 3.4.2 over linux 2.6.34. I have a replica 2 
volume 'testvol' that is It seems I cannot read/stat/edit the file in 
question, and `gluster volume heal testvol info split-brain` shows 
nothing. Here are the logs from the fuse-mount for the volume:


[2014-09-29 07:53:02.867111] W [fuse-bridge.c:1172:fuse_err_cbk] 
0-glusterfs-fuse: 4560969: FLUSH() ERR = -1 (Input/output error)
[2014-09-29 07:54:16.007799] W [page.c:991:__ioc_page_error] 
0-testvol-io-cache: page error for page = 0x7fd5c8529d20  waitq = 
0x7fd5c8067d40
[2014-09-29 07:54:16.007854] W [fuse-bridge.c:2089:fuse_readv_cbk] 
0-glusterfs-fuse: 4561103: READ = -1 (Input/output error)
[2014-09-29 07:54:16.008018] W [page.c:991:__ioc_page_error] 
0-testvol-io-cache: page error for page = 0x7fd5c8607ee0  waitq = 
0x7fd5c8067d40
[2014-09-29 07:54:16.008056] W [fuse-bridge.c:2089:fuse_readv_cbk] 
0-glusterfs-fuse: 4561104: READ = -1 (Input/output error)
[2014-09-29 07:54:16.008233] W [page.c:991:__ioc_page_error] 
0-testvol-io-cache: page error for page = 0x7fd5c8066f30  waitq = 
0x7fd5c8067d40
[2014-09-29 07:54:16.008269] W [fuse-bridge.c:2089:fuse_readv_cbk] 
0-glusterfs-fuse: 4561105: READ = -1 (Input/output error)
[2014-09-29 07:54:16.008800] W [page.c:991:__ioc_page_error] 
0-testvol-io-cache: page error for page = 0x7fd5c860bcf0  waitq = 
0x7fd5c863b1f0
[2014-09-29 07:54:16.008839] W [fuse-bridge.c:2089:fuse_readv_cbk] 
0-glusterfs-fuse: 4561107: READ = -1 (Input/output error)
[2014-09-29 07:54:16.009365] W [page.c:991:__ioc_page_error] 
0-testvol-io-cache: page error for page = 0x7fd5c85fd120  waitq = 
0x7fd5c8067d40
[2014-09-29 07:54:16.009413] W [fuse-bridge.c:2089:fuse_readv_cbk] 
0-glusterfs-fuse: 4561109: READ = -1 (Input/output error)
[2014-09-29 07:54:16.040549] W [afr-open.c:213:afr_open] 
0-testvol-replicate-0: failed to open as split brain seen, returning EIO
[2014-09-29 07:54:16.040594] W [fuse-bridge.c:915:fuse_fd_cbk] 
0-glusterfs-fuse: 4561142: OPEN() 
/SECLOG/20140908.d/SECLOG_00427425_.log = 
-1 (Input/output error)


Could somebody please give me some clue on where to begin? I checked 
the xattrs on 
/SECLOG/20140908.d/SECLOG_00427425_.log and 
it seems the changelogs are [0, 0] on both replicas, and the gfid's match.


Thank you very much for any help on this.
Anirban





___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain on glusterfs running with quorum on server and client

[Ramesh Natarajan]

I was able to run another set of tests this week and I was able to
reproduce the issue again. Going by the extended attributes, I think i ran
into the same issue I saw earlier..

 Do you think i need to open up a bug report?

Brick 1:

trusted.afr.PL2-client-0=0x
trusted.afr.PL2-client-1=0x0001
trusted.afr.PL2-client-2=0x0001
trusted.gfid=0x1cea509b07cc49e9bd28560b5f33032c

Brick 2

trusted.afr.PL2-client-0=0x125c
trusted.afr.PL2-client-1=0x
trusted.afr.PL2-client-2=0x
trusted.gfid=0x1cea509b07cc49e9bd28560b5f33032c

Brick 3

trusted.afr.PL2-client-0=0x125c
trusted.afr.PL2-client-1=0x
trusted.afr.PL2-client-2=0x
trusted.gfid=0x1cea509b07cc49e9bd28560b5f33032c


[root@ip-172-31-12-218 ~]# gluster volume info

Volume Name: PL1
Type: Replicate
Volume ID: bd351bae-d467-4e8c-bbd2-6a0fe99c346a
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 172.31.38.189:/data/vol1/gluster-data
Brick2: 172.31.16.220:/data/vol1/gluster-data
Brick3: 172.31.12.218:/data/vol1/gluster-data
Options Reconfigured:
cluster.server-quorum-type: server
network.ping-timeout: 12
nfs.addr-namelookup: off
performance.cache-size: 2147483648
cluster.quorum-type: auto
performance.read-ahead: off
performance.client-io-threads: on
performance.io-thread-count: 64
cluster.eager-lock: on
cluster.server-quorum-ratio: 51%

Volume Name: PL2
Type: Replicate
Volume ID: e6ad8787-05d8-474b-bc78-748f8c13700f
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 172.31.38.189:/data/vol2/gluster-data
Brick2: 172.31.16.220:/data/vol2/gluster-data
Brick3: 172.31.12.218:/data/vol2/gluster-data
Options Reconfigured:
nfs.addr-namelookup: off
cluster.server-quorum-type: server
network.ping-timeout: 12
performance.cache-size: 2147483648
cluster.quorum-type: auto
performance.read-ahead: off
performance.client-io-threads: on
performance.io-thread-count: 64
cluster.eager-lock: on
cluster.server-quorum-ratio: 51%
[root@ip-172-31-12-218 ~]#

*Mount command*

Client

mount -t glusterfs -o
defaults,enable-ino32,direct-io-mode=disable,log-level=WARNING,log-file=/var/log/gluster.log,backupvolfile-server=172.31.38.189,backupvolfile-server=172.31.12.218,background-qlen=256
172.31.16.220:/PL2  /mnt/vm

Server

/dev/xvdf/data/vol1 xfs defaults,inode64,noatime 1 2
/dev/xvdg   /data/vol2 xfs defaults,inode64,noatime 1 2

*Packages*

Client

rpm -qa | grep gluster
glusterfs-fuse-3.5.2-1.el6.x86_64
glusterfs-3.5.2-1.el6.x86_64
glusterfs-libs-3.5.2-1.el6.x86_64

Server

[root@ip-172-31-12-218 ~]# rpm -qa | grep gluster
glusterfs-3.5.2-1.el6.x86_64
glusterfs-fuse-3.5.2-1.el6.x86_64
glusterfs-api-3.5.2-1.el6.x86_64
glusterfs-server-3.5.2-1.el6.x86_64
glusterfs-libs-3.5.2-1.el6.x86_64
glusterfs-cli-3.5.2-1.el6.x86_64
[root@ip-172-31-12-218 ~]#


On Sat, Sep 6, 2014 at 9:01 AM, Pranith Kumar Karampuri pkara...@redhat.com
 wrote:


 On 09/06/2014 04:53 AM, Jeff Darcy wrote:

 I have a replicate glusterfs setup on 3 Bricks ( replicate = 3 ). I have
 client and server quorum turned on. I rebooted one of the 3 bricks. When
 it
 came back up, the client started throwing error messages that one of the
 files went into split brain.

 This is a good example of how split brain can happen even with all kinds
 of
 quorum enabled.  Let's look at those xattrs.  BTW, thank you for a very
 nicely detailed bug report which includes those.

  BRICK1
 
 [root@ip-172-31-38-189 ~]# getfattr -d -m . -e hex
 /data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17
 _00_00
 getfattr: Removing leading '/' from absolute path names
 # file:
 data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
 trusted.afr.PL2-client-0=0x
 trusted.afr.PL2-client-1=0x0001
 trusted.afr.PL2-client-2=0x0001
 trusted.gfid=0xea950263977e46bf89a0ef631ca139c2

 BRICK 2
 ===
 [root@ip-172-31-16-220 ~]# getfattr -d -m . -e hex
 /data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17
 _00_00
 getfattr: Removing leading '/' from absolute path names
 # file:
 data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
 trusted.afr.PL2-client-0=0x0d46
 trusted.afr.PL2-client-1=0x
 trusted.afr.PL2-client-2=0x
 trusted.gfid=0xea950263977e46bf89a0ef631ca139c2
 BRICK 3
 =
 [root@ip-172-31-12-218 ~]# getfattr -d -m . -e hex
 /data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17
 _00_00
 getfattr: Removing leading '/' from absolute path names
 # file:
 data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
 trusted.afr.PL2-client-0=0x0d46
 trusted.afr.PL2-client-1=0x
 

Re: [Gluster-users] split-brain on glusterfs running with quorum on server and client

[Pranith Kumar Karampuri]

On 09/19/2014 09:58 PM, Ramesh Natarajan wrote:
I was able to run another set of tests this week and I was able to 
reproduce the issue again. Going by the extended attributes, I think i 
ran into the same issue I saw earlier..


 Do you think i need to open up a bug report?

hi Ramesh,
 I already fixed this bug. http://review.gluster.org/8757. We 
should have the fix in next 3.5.x release I believe.


Pranith


Brick 1:

trusted.afr.PL2-client-0=0x
trusted.afr.PL2-client-1=0x0001
trusted.afr.PL2-client-2=0x0001
trusted.gfid=0x1cea509b07cc49e9bd28560b5f33032c

Brick 2

trusted.afr.PL2-client-0=0x125c
trusted.afr.PL2-client-1=0x
trusted.afr.PL2-client-2=0x
trusted.gfid=0x1cea509b07cc49e9bd28560b5f33032c

Brick 3

trusted.afr.PL2-client-0=0x125c
trusted.afr.PL2-client-1=0x
trusted.afr.PL2-client-2=0x
trusted.gfid=0x1cea509b07cc49e9bd28560b5f33032c


[root@ip-172-31-12-218 ~]# gluster volume info
Volume Name: PL1
Type: Replicate
Volume ID: bd351bae-d467-4e8c-bbd2-6a0fe99c346a
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 172.31.38.189:/data/vol1/gluster-data
Brick2: 172.31.16.220:/data/vol1/gluster-data
Brick3: 172.31.12.218:/data/vol1/gluster-data
Options Reconfigured:
cluster.server-quorum-type: server
network.ping-timeout: 12
nfs.addr-namelookup: off
performance.cache-size: 2147483648
cluster.quorum-type: auto
performance.read-ahead: off
performance.client-io-threads: on
performance.io-thread-count: 64
cluster.eager-lock: on
cluster.server-quorum-ratio: 51%
Volume Name: PL2
Type: Replicate
Volume ID: e6ad8787-05d8-474b-bc78-748f8c13700f
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 172.31.38.189:/data/vol2/gluster-data
Brick2: 172.31.16.220:/data/vol2/gluster-data
Brick3: 172.31.12.218:/data/vol2/gluster-data
Options Reconfigured:
nfs.addr-namelookup: off
cluster.server-quorum-type: server
network.ping-timeout: 12
performance.cache-size: 2147483648
cluster.quorum-type: auto
performance.read-ahead: off
performance.client-io-threads: on
performance.io-thread-count: 64
cluster.eager-lock: on
cluster.server-quorum-ratio: 51%
[root@ip-172-31-12-218 ~]#

*Mount command*

Client

mount -t glusterfs -o 
defaults,enable-ino32,direct-io-mode=disable,log-level=WARNING,log-file=/var/log/gluster.log,backupvolfile-server=172.31.38.189,backupvolfile-server=172.31.12.218,background-qlen=256 
172.31.16.220:/PL2  /mnt/vm


Server

/dev/xvdf/data/vol1 xfs defaults,inode64,noatime 1 2
/dev/xvdg   /data/vol2 xfs defaults,inode64,noatime 1 2

*Packages*

Client

rpm -qa | grep gluster
glusterfs-fuse-3.5.2-1.el6.x86_64
glusterfs-3.5.2-1.el6.x86_64
glusterfs-libs-3.5.2-1.el6.x86_64

Server

[root@ip-172-31-12-218 ~]# rpm -qa | grep gluster
glusterfs-3.5.2-1.el6.x86_64
glusterfs-fuse-3.5.2-1.el6.x86_64
glusterfs-api-3.5.2-1.el6.x86_64
glusterfs-server-3.5.2-1.el6.x86_64
glusterfs-libs-3.5.2-1.el6.x86_64
glusterfs-cli-3.5.2-1.el6.x86_64
[root@ip-172-31-12-218 ~]#


On Sat, Sep 6, 2014 at 9:01 AM, Pranith Kumar Karampuri 
pkara...@redhat.com mailto:pkara...@redhat.com wrote:



On 09/06/2014 04:53 AM, Jeff Darcy wrote:

I have a replicate glusterfs setup on 3 Bricks ( replicate
= 3 ). I have
client and server quorum turned on. I rebooted one of the
3 bricks. When it
came back up, the client started throwing error messages
that one of the
files went into split brain.

This is a good example of how split brain can happen even with
all kinds of
quorum enabled.  Let's look at those xattrs.  BTW, thank you
for a very
nicely detailed bug report which includes those.

BRICK1

[root@ip-172-31-38-189 ~]# getfattr -d -m . -e hex
/data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17
tel:2014-09-05-17_00_00
getfattr: Removing leading '/' from absolute path names
# file:
data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17
tel:2014-09-05-17_00_00
trusted.afr.PL2-client-0=0x
trusted.afr.PL2-client-1=0x0001
trusted.afr.PL2-client-2=0x0001
trusted.gfid=0xea950263977e46bf89a0ef631ca139c2

BRICK 2
===
[root@ip-172-31-16-220 ~]# getfattr -d -m . -e hex
/data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17
tel:2014-09-05-17_00_00
getfattr: Removing leading '/' from absolute path names
# file:
data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17

Re: [Gluster-users] Split brain that is not split brain

[Ilya Ivanov]

I don't understand why there's such a complicated process to recover when I
can just look at both files, decide which one I need and delete another one.


On Thu, Sep 11, 2014 at 7:56 AM, Pranith Kumar Karampuri 
pkara...@redhat.com wrote:


 On 09/11/2014 09:29 AM, Ilya Ivanov wrote:

  Right... I deleted it and now all appears to be fine.

  Still, could you please elaborate on gfid split-brain?

 Could you go through
 https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md
 Let us know if you would like something to be more clearer and we can add
 that and improve the document.

 Pranith



 On Thu, Sep 11, 2014 at 5:32 AM, Pranith Kumar Karampuri 
 pkara...@redhat.com wrote:


 On 09/11/2014 12:16 AM, Ilya Ivanov wrote:

 Any insight?

 Was the other file's gfid d3def9e1-c6d0-4b7d-a322-b5019305182e?
 Could you check if this file exists in brick/.glusterfs/d3/de/
 When a file is deleted this file also needs to be deleted if there are no
 more hardlinks to the file

 Pranith


 On Tue, Sep 9, 2014 at 8:35 AM, Ilya Ivanov bearw...@gmail.com wrote:

 What's a gfid split-brain and how is it different from normal
 split-brain?

  I accessed the file with stat, but heal info still shows Number of
 entries: 1

 [root@gluster1 gluster]# getfattr -d -m. -e hex gv01/123
 # getfattr -d -m. -e hex gv01/123
 # file: gv01/123
 trusted.afr.gv01-client-0=0x
 trusted.afr.gv01-client-1=0x
 trusted.gfid=0x35f86f4561134ba0bd1b94ef70179d4d

 [root@gluster1 gluster]# getfattr -d -m. -e hex gv01
 # file: gv01
 trusted.afr.gv01-client-0=0x
 trusted.afr.gv01-client-1=0x
 trusted.gfid=0x0001
 trusted.glusterfs.dht=0x0001
 trusted.glusterfs.volume-id=0x31a2c4c486ca4344b838d2c2e6c716c1



 On Tue, Sep 9, 2014 at 8:19 AM, Pranith Kumar Karampuri 
 pkara...@redhat.com wrote:


 On 09/09/2014 11:35 AM, Ilya Ivanov wrote:

  Ahh, thank you, now I get it. I deleted it on one node and it
 replicated to another one. Now I get the following output:

 [root@gluster1 var]# gluster volume heal gv01 info
 Brick gluster1:/home/gluster/gv01/
 gfid:d3def9e1-c6d0-4b7d-a322-b5019305182e
 Number of entries: 1

 Brick gluster2:/home/gluster/gv01/
 Number of entries: 0

  Is it normal? Why the number of entries isn't reset to 0?

  If you access the file using ls/stat etc, it will be fixed. But before
 that could you please post the output of 'getfattr -d -m. -e hex
 file/path/in/backend/brick' and 'getfattr -d -m. -e hex
 parent/dir/to/file/path/in/backend/brick'

 Pranith



 And why wouldn't the file show up in split-brain before, anyway?

  Gfid split-brains are not shown in heal-info-split-brain yet.

 Pranith



 On Tue, Sep 9, 2014 at 7:46 AM, Pranith Kumar Karampuri 
 pkara...@redhat.com wrote:


 On 09/09/2014 01:54 AM, Ilya Ivanov wrote:

  Hello.

  I've Gluster 3.5.2 on Centos 6. A primitive replicated volume, as
 describe here
 https://www.digitalocean.com/community/tutorials/how-to-create-a-redundant-storage-pool-using-glusterfs-on-ubuntu-servers.
 I tried to simulate split-brain by temporarily disconnecting the nodes and
 creating a file with the same name and different contents. That worked.

  The question is, how do I fix it now? All the tutorials suggest
 deleting the file from one of the nodes. I can't do that, it reports
 Input/output error. The file won't even show up in gluster volume heal
 gv00 info split-brain. That shows 0 entries.

  The deletion needs to happen on one of the bricks, not from the mount
 point.

 Pranith

  I can see the file in gluster volume heal gv00 info heal-failed,
 though.


  --
 Ilya.


  ___
 Gluster-users mailing 
 listGluster-users@gluster.orghttp://supercolony.gluster.org/mailman/listinfo/gluster-users





 --
 Ilya.





  --
 Ilya.




 --
 Ilya.





 --
 Ilya.





-- 
Ilya.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain that is not split brain

[Pranith Kumar Karampuri]

On 09/11/2014 11:37 AM, Ilya Ivanov wrote:
I don't understand why there's such a complicated process to recover 
when I can just look at both files, decide which one I need and delete 
another one.
If the file needs to be deleted the whole file needs to be copied which 
is fine for small files but for big files like VM images it takes less 
time if the file already exists and it syncs only the parts of files 
that are different from the good copy.
One more reason is if the parent directory from which the file is 
deleted from is the source then self-heal will delete the file from 
other directory rather than creating it. SO instead of deleting the file 
may be it is a better practise to make a copy of the file somewhere and 
delete it. We shall update the document as well with this new 
information. Thanks for the feedback. In 3.7 it is going to be 
simplified. We are giving a command to fix the split-brains where the 
user gets to choose the file and it will do the rest.


Pranith



On Thu, Sep 11, 2014 at 7:56 AM, Pranith Kumar Karampuri 
pkara...@redhat.com mailto:pkara...@redhat.com wrote:



On 09/11/2014 09:29 AM, Ilya Ivanov wrote:

Right... I deleted it and now all appears to be fine.

Still, could you please elaborate on gfid split-brain?

Could you go through

https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md
Let us know if you would like something to be more clearer and we
can add that and improve the document.

Pranith




On Thu, Sep 11, 2014 at 5:32 AM, Pranith Kumar Karampuri
pkara...@redhat.com mailto:pkara...@redhat.com wrote:


On 09/11/2014 12:16 AM, Ilya Ivanov wrote:

Any insight?

Was the other file's gfid d3def9e1-c6d0-4b7d-a322-b5019305182e?
Could you check if this file exists in brick/.glusterfs/d3/de/
When a file is deleted this file also needs to be deleted if
there are no more hardlinks to the file

Pranith


On Tue, Sep 9, 2014 at 8:35 AM, Ilya Ivanov
bearw...@gmail.com mailto:bearw...@gmail.com wrote:

What's a gfid split-brain and how is it different from
normal split-brain?

I accessed the file with stat, but heal info still
shows Number of entries: 1

[root@gluster1 gluster]# getfattr -d -m. -e hex gv01/123
# getfattr -d -m. -e hex gv01/123
# file: gv01/123
trusted.afr.gv01-client-0=0x
trusted.afr.gv01-client-1=0x
trusted.gfid=0x35f86f4561134ba0bd1b94ef70179d4d

[root@gluster1 gluster]# getfattr -d -m. -e hex gv01
# file: gv01
trusted.afr.gv01-client-0=0x
trusted.afr.gv01-client-1=0x
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x31a2c4c486ca4344b838d2c2e6c716c1



On Tue, Sep 9, 2014 at 8:19 AM, Pranith Kumar Karampuri
pkara...@redhat.com mailto:pkara...@redhat.com wrote:


On 09/09/2014 11:35 AM, Ilya Ivanov wrote:

Ahh, thank you, now I get it. I deleted it on one
node and it replicated to another one. Now I get
the following output:

[root@gluster1 var]# gluster volume heal gv01 info
Brick gluster1:/home/gluster/gv01/
gfid:d3def9e1-c6d0-4b7d-a322-b5019305182e
Number of entries: 1

Brick gluster2:/home/gluster/gv01/
Number of entries: 0

Is it normal? Why the number of entries isn't reset
to 0?

If you access the file using ls/stat etc, it will be
fixed. But before that could you please post the
output of 'getfattr -d -m. -e hex
file/path/in/backend/brick' and 'getfattr -d -m. -e
hex parent/dir/to/file/path/in/backend/brick'

Pranith



And why wouldn't the file show up in split-brain
before, anyway?

Gfid split-brains are not shown in
heal-info-split-brain yet.

Pranith



On Tue, Sep 9, 2014 at 7:46 AM, Pranith Kumar
Karampuri pkara...@redhat.com
mailto:pkara...@redhat.com wrote:


On 09/09/2014 01:54 AM, Ilya Ivanov wrote:

Hello.

I've Gluster 3.5.2 on Centos 6. A primitive
replicated volume, as describe here

https://www.digitalocean.com/community/tutorials/how-to-create-a-redundant-storage-pool-using-glusterfs-on-ubuntu-servers.
I tried to simulate split-brain by temporarily

Re: [Gluster-users] Split brain that is not split brain

[Ilya Ivanov]

Makes some sense. Yes, I meant make a backup and delete, rather than just
delete.

If I may suggest, putting that debug link somewhere more visible would be
be good, too. I wouldn't find without your help.

Thank you for the assistance.




On Thu, Sep 11, 2014 at 9:14 AM, Pranith Kumar Karampuri 
pkara...@redhat.com wrote:


 On 09/11/2014 11:37 AM, Ilya Ivanov wrote:

 I don't understand why there's such a complicated process to recover when
 I can just look at both files, decide which one I need and delete another
 one.

 If the file needs to be deleted the whole file needs to be copied which is
 fine for small files but for big files like VM images it takes less time if
 the file already exists and it syncs only the parts of files that are
 different from the good copy.
 One more reason is if the parent directory from which the file is deleted
 from is the source then self-heal will delete the file from other directory
 rather than creating it. SO instead of deleting the file may be it is a
 better practise to make a copy of the file somewhere and delete it. We
 shall update the document as well with this new information. Thanks for the
 feedback. In 3.7 it is going to be simplified. We are giving a command to
 fix the split-brains where the user gets to choose the file and it will do
 the rest.


 Pranith



 On Thu, Sep 11, 2014 at 7:56 AM, Pranith Kumar Karampuri 
 pkara...@redhat.com wrote:


 On 09/11/2014 09:29 AM, Ilya Ivanov wrote:

  Right... I deleted it and now all appears to be fine.

  Still, could you please elaborate on gfid split-brain?

  Could you go through
 https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md
 Let us know if you would like something to be more clearer and we can add
 that and improve the document.

 Pranith



 On Thu, Sep 11, 2014 at 5:32 AM, Pranith Kumar Karampuri 
 pkara...@redhat.com wrote:


 On 09/11/2014 12:16 AM, Ilya Ivanov wrote:

 Any insight?

 Was the other file's gfid d3def9e1-c6d0-4b7d-a322-b5019305182e?
 Could you check if this file exists in brick/.glusterfs/d3/de/
 When a file is deleted this file also needs to be deleted if there are
 no more hardlinks to the file

 Pranith


 On Tue, Sep 9, 2014 at 8:35 AM, Ilya Ivanov bearw...@gmail.com wrote:

 What's a gfid split-brain and how is it different from normal
 split-brain?

  I accessed the file with stat, but heal info still shows Number of
 entries: 1

 [root@gluster1 gluster]# getfattr -d -m. -e hex gv01/123
 # getfattr -d -m. -e hex gv01/123
 # file: gv01/123
 trusted.afr.gv01-client-0=0x
 trusted.afr.gv01-client-1=0x
 trusted.gfid=0x35f86f4561134ba0bd1b94ef70179d4d

 [root@gluster1 gluster]# getfattr -d -m. -e hex gv01
 # file: gv01
 trusted.afr.gv01-client-0=0x
 trusted.afr.gv01-client-1=0x
 trusted.gfid=0x0001
 trusted.glusterfs.dht=0x0001
 trusted.glusterfs.volume-id=0x31a2c4c486ca4344b838d2c2e6c716c1



 On Tue, Sep 9, 2014 at 8:19 AM, Pranith Kumar Karampuri 
 pkara...@redhat.com wrote:


 On 09/09/2014 11:35 AM, Ilya Ivanov wrote:

  Ahh, thank you, now I get it. I deleted it on one node and it
 replicated to another one. Now I get the following output:

 [root@gluster1 var]# gluster volume heal gv01 info
 Brick gluster1:/home/gluster/gv01/
 gfid:d3def9e1-c6d0-4b7d-a322-b5019305182e
 Number of entries: 1

 Brick gluster2:/home/gluster/gv01/
 Number of entries: 0

  Is it normal? Why the number of entries isn't reset to 0?

  If you access the file using ls/stat etc, it will be fixed. But
 before that could you please post the output of 'getfattr -d -m. -e hex
 file/path/in/backend/brick' and 'getfattr -d -m. -e hex
 parent/dir/to/file/path/in/backend/brick'

 Pranith



 And why wouldn't the file show up in split-brain before, anyway?

  Gfid split-brains are not shown in heal-info-split-brain yet.

 Pranith



 On Tue, Sep 9, 2014 at 7:46 AM, Pranith Kumar Karampuri 
 pkara...@redhat.com wrote:


 On 09/09/2014 01:54 AM, Ilya Ivanov wrote:

  Hello.

  I've Gluster 3.5.2 on Centos 6. A primitive replicated volume, as
 describe here
 https://www.digitalocean.com/community/tutorials/how-to-create-a-redundant-storage-pool-using-glusterfs-on-ubuntu-servers.
 I tried to simulate split-brain by temporarily disconnecting the nodes 
 and
 creating a file with the same name and different contents. That worked.

  The question is, how do I fix it now? All the tutorials suggest
 deleting the file from one of the nodes. I can't do that, it reports
 Input/output error. The file won't even show up in gluster volume heal
 gv00 info split-brain. That shows 0 entries.

  The deletion needs to happen on one of the bricks, not from the
 mount point.

 Pranith

  I can see the file in gluster volume heal gv00 info heal-failed,
 though.


  --
 Ilya.


  ___
 Gluster-users mailing 
 

Re: [Gluster-users] Split brain that is not split brain

[Pranith Kumar Karampuri]

On 09/11/2014 01:13 PM, Ilya Ivanov wrote:
Makes some sense. Yes, I meant make a backup and delete, rather than 
just delete.


If I may suggest, putting that debug link somewhere more visible would 
be be good, too. I wouldn't find without your help.

Justin, where shall we put the doc?

Pranith


Thank you for the assistance.




On Thu, Sep 11, 2014 at 9:14 AM, Pranith Kumar Karampuri 
pkara...@redhat.com mailto:pkara...@redhat.com wrote:



On 09/11/2014 11:37 AM, Ilya Ivanov wrote:

I don't understand why there's such a complicated process to
recover when I can just look at both files, decide which one I
need and delete another one.

If the file needs to be deleted the whole file needs to be copied
which is fine for small files but for big files like VM images it
takes less time if the file already exists and it syncs only the
parts of files that are different from the good copy.
One more reason is if the parent directory from which the file is
deleted from is the source then self-heal will delete the file
from other directory rather than creating it. SO instead of
deleting the file may be it is a better practise to make a copy of
the file somewhere and delete it. We shall update the document as
well with this new information. Thanks for the feedback. In 3.7 it
is going to be simplified. We are giving a command to fix the
split-brains where the user gets to choose the file and it will do
the rest.


Pranith



On Thu, Sep 11, 2014 at 7:56 AM, Pranith Kumar Karampuri
pkara...@redhat.com mailto:pkara...@redhat.com wrote:


On 09/11/2014 09:29 AM, Ilya Ivanov wrote:

Right... I deleted it and now all appears to be fine.

Still, could you please elaborate on gfid split-brain?

Could you go through

https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md
Let us know if you would like something to be more clearer
and we can add that and improve the document.

Pranith




On Thu, Sep 11, 2014 at 5:32 AM, Pranith Kumar Karampuri
pkara...@redhat.com mailto:pkara...@redhat.com wrote:


On 09/11/2014 12:16 AM, Ilya Ivanov wrote:

Any insight?

Was the other file's gfid
d3def9e1-c6d0-4b7d-a322-b5019305182e?
Could you check if this file exists in
brick/.glusterfs/d3/de/
When a file is deleted this file also needs to be
deleted if there are no more hardlinks to the file

Pranith


On Tue, Sep 9, 2014 at 8:35 AM, Ilya Ivanov
bearw...@gmail.com mailto:bearw...@gmail.com wrote:

What's a gfid split-brain and how is it different
from normal split-brain?

I accessed the file with stat, but heal info
still shows Number of entries: 1

[root@gluster1 gluster]# getfattr -d -m. -e hex
gv01/123
# getfattr -d -m. -e hex gv01/123
# file: gv01/123
trusted.afr.gv01-client-0=0x
trusted.afr.gv01-client-1=0x
trusted.gfid=0x35f86f4561134ba0bd1b94ef70179d4d

[root@gluster1 gluster]# getfattr -d -m. -e hex gv01
# file: gv01
trusted.afr.gv01-client-0=0x
trusted.afr.gv01-client-1=0x
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x31a2c4c486ca4344b838d2c2e6c716c1



On Tue, Sep 9, 2014 at 8:19 AM, Pranith Kumar
Karampuri pkara...@redhat.com
mailto:pkara...@redhat.com wrote:


On 09/09/2014 11:35 AM, Ilya Ivanov wrote:

Ahh, thank you, now I get it. I deleted it on
one node and it replicated to another one. Now
I get the following output:

[root@gluster1 var]# gluster volume heal gv01 info
Brick gluster1:/home/gluster/gv01/
gfid:d3def9e1-c6d0-4b7d-a322-b5019305182e
Number of entries: 1

Brick gluster2:/home/gluster/gv01/
Number of entries: 0

Is it normal? Why the number of entries isn't
reset to 0?

If you access the file using ls/stat etc, it
will be fixed. But before that could you please
post the output of 'getfattr -d -m. -e hex
file/path/in/backend/brick' and 'getfattr -d
-m. -e hex
parent/dir/to/file/path/in/backend/brick'

Pranith



  

Re: [Gluster-users] Split brain that is not split brain

[Justin Clift]

On 11/09/2014, at 9:44 AM, Pranith Kumar Karampuri wrote:
 On 09/11/2014 01:13 PM, Ilya Ivanov wrote:
 Makes some sense. Yes, I meant make a backup and delete, rather than just 
 delete.
 
 If I may suggest, putting that debug link somewhere more visible would be be 
 good, too. I wouldn't find without your help.
 Justin, where shall we put the doc?

In theory, the .md files should be pulled into the main documentation
part of the static site, somewhere under here:

  http://www.gluster.org/documentation/

I'm not sure if we've yet got that happening, but we definitely need
to in the near future.

+ Justin

--
GlusterFS - http://www.gluster.org

An open source, distributed file system scaling to several
petabytes, and handling thousands of clients.

My personal twitter: twitter.com/realjustinclift

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain that is not split brain

[Ilya Ivanov]

Any insight?

On Tue, Sep 9, 2014 at 8:35 AM, Ilya Ivanov bearw...@gmail.com wrote:

 What's a gfid split-brain and how is it different from normal
 split-brain?

 I accessed the file with stat, but heal info still shows Number of
 entries: 1

 [root@gluster1 gluster]# getfattr -d -m. -e hex gv01/123
 # getfattr -d -m. -e hex gv01/123
 # file: gv01/123
 trusted.afr.gv01-client-0=0x
 trusted.afr.gv01-client-1=0x
 trusted.gfid=0x35f86f4561134ba0bd1b94ef70179d4d

 [root@gluster1 gluster]# getfattr -d -m. -e hex gv01
 # file: gv01
 trusted.afr.gv01-client-0=0x
 trusted.afr.gv01-client-1=0x
 trusted.gfid=0x0001
 trusted.glusterfs.dht=0x0001
 trusted.glusterfs.volume-id=0x31a2c4c486ca4344b838d2c2e6c716c1



 On Tue, Sep 9, 2014 at 8:19 AM, Pranith Kumar Karampuri 
 pkara...@redhat.com wrote:


 On 09/09/2014 11:35 AM, Ilya Ivanov wrote:

  Ahh, thank you, now I get it. I deleted it on one node and it
 replicated to another one. Now I get the following output:

 [root@gluster1 var]# gluster volume heal gv01 info
 Brick gluster1:/home/gluster/gv01/
 gfid:d3def9e1-c6d0-4b7d-a322-b5019305182e
 Number of entries: 1

 Brick gluster2:/home/gluster/gv01/
 Number of entries: 0

  Is it normal? Why the number of entries isn't reset to 0?

 If you access the file using ls/stat etc, it will be fixed. But before
 that could you please post the output of 'getfattr -d -m. -e hex
 file/path/in/backend/brick' and 'getfattr -d -m. -e hex
 parent/dir/to/file/path/in/backend/brick'

 Pranith



 And why wouldn't the file show up in split-brain before, anyway?

 Gfid split-brains are not shown in heal-info-split-brain yet.

 Pranith



 On Tue, Sep 9, 2014 at 7:46 AM, Pranith Kumar Karampuri 
 pkara...@redhat.com wrote:


 On 09/09/2014 01:54 AM, Ilya Ivanov wrote:

  Hello.

  I've Gluster 3.5.2 on Centos 6. A primitive replicated volume, as
 describe here
 https://www.digitalocean.com/community/tutorials/how-to-create-a-redundant-storage-pool-using-glusterfs-on-ubuntu-servers.
 I tried to simulate split-brain by temporarily disconnecting the nodes and
 creating a file with the same name and different contents. That worked.

  The question is, how do I fix it now? All the tutorials suggest
 deleting the file from one of the nodes. I can't do that, it reports
 Input/output error. The file won't even show up in gluster volume heal
 gv00 info split-brain. That shows 0 entries.

  The deletion needs to happen on one of the bricks, not from the mount
 point.

 Pranith

  I can see the file in gluster volume heal gv00 info heal-failed,
 though.


  --
 Ilya.


  ___
 Gluster-users mailing 
 listGluster-users@gluster.orghttp://supercolony.gluster.org/mailman/listinfo/gluster-users





 --
 Ilya.





 --
 Ilya.




-- 
Ilya.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain that is not split brain

[Pranith Kumar Karampuri]

On 09/11/2014 12:16 AM, Ilya Ivanov wrote:

Any insight?

Was the other file's gfid d3def9e1-c6d0-4b7d-a322-b5019305182e?
Could you check if this file exists in brick/.glusterfs/d3/de/
When a file is deleted this file also needs to be deleted if there are 
no more hardlinks to the file


Pranith


On Tue, Sep 9, 2014 at 8:35 AM, Ilya Ivanov bearw...@gmail.com 
mailto:bearw...@gmail.com wrote:


What's a gfid split-brain and how is it different from normal
split-brain?

I accessed the file with stat, but heal info still shows Number
of entries: 1

[root@gluster1 gluster]# getfattr -d -m. -e hex gv01/123
# getfattr -d -m. -e hex gv01/123
# file: gv01/123
trusted.afr.gv01-client-0=0x
trusted.afr.gv01-client-1=0x
trusted.gfid=0x35f86f4561134ba0bd1b94ef70179d4d

[root@gluster1 gluster]# getfattr -d -m. -e hex gv01
# file: gv01
trusted.afr.gv01-client-0=0x
trusted.afr.gv01-client-1=0x
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x31a2c4c486ca4344b838d2c2e6c716c1



On Tue, Sep 9, 2014 at 8:19 AM, Pranith Kumar Karampuri
pkara...@redhat.com mailto:pkara...@redhat.com wrote:


On 09/09/2014 11:35 AM, Ilya Ivanov wrote:

Ahh, thank you, now I get it. I deleted it on one node and it
replicated to another one. Now I get the following output:

[root@gluster1 var]# gluster volume heal gv01 info
Brick gluster1:/home/gluster/gv01/
gfid:d3def9e1-c6d0-4b7d-a322-b5019305182e
Number of entries: 1

Brick gluster2:/home/gluster/gv01/
Number of entries: 0

Is it normal? Why the number of entries isn't reset to 0?

If you access the file using ls/stat etc, it will be fixed.
But before that could you please post the output of 'getfattr
-d -m. -e hex file/path/in/backend/brick' and 'getfattr -d -m.
-e hex parent/dir/to/file/path/in/backend/brick'

Pranith



And why wouldn't the file show up in split-brain before, anyway?

Gfid split-brains are not shown in heal-info-split-brain yet.

Pranith



On Tue, Sep 9, 2014 at 7:46 AM, Pranith Kumar Karampuri
pkara...@redhat.com mailto:pkara...@redhat.com wrote:


On 09/09/2014 01:54 AM, Ilya Ivanov wrote:

Hello.

I've Gluster 3.5.2 on Centos 6. A primitive replicated
volume, as describe here

https://www.digitalocean.com/community/tutorials/how-to-create-a-redundant-storage-pool-using-glusterfs-on-ubuntu-servers.
I tried to simulate split-brain by temporarily
disconnecting the nodes and creating a file with the
same name and different contents. That worked.

The question is, how do I fix it now? All the tutorials
suggest deleting the file from one of the nodes. I can't
do that, it reports Input/output error. The file won't
even show up in gluster volume heal gv00 info
split-brain. That shows 0 entries.

The deletion needs to happen on one of the bricks, not
from the mount point.

Pranith

I can see the file in gluster volume heal gv00 info
heal-failed, though.


-- 
Ilya.



___
Gluster-users mailing list
Gluster-users@gluster.org  mailto:Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users





-- 
Ilya.





-- 
Ilya.





--
Ilya.


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain that is not split brain

[Ilya Ivanov]

Right... I deleted it and now all appears to be fine.

Still, could you please elaborate on gfid split-brain?


On Thu, Sep 11, 2014 at 5:32 AM, Pranith Kumar Karampuri 
pkara...@redhat.com wrote:


 On 09/11/2014 12:16 AM, Ilya Ivanov wrote:

 Any insight?

 Was the other file's gfid d3def9e1-c6d0-4b7d-a322-b5019305182e?
 Could you check if this file exists in brick/.glusterfs/d3/de/
 When a file is deleted this file also needs to be deleted if there are no
 more hardlinks to the file

 Pranith


 On Tue, Sep 9, 2014 at 8:35 AM, Ilya Ivanov bearw...@gmail.com wrote:

 What's a gfid split-brain and how is it different from normal
 split-brain?

  I accessed the file with stat, but heal info still shows Number of
 entries: 1

 [root@gluster1 gluster]# getfattr -d -m. -e hex gv01/123
 # getfattr -d -m. -e hex gv01/123
 # file: gv01/123
 trusted.afr.gv01-client-0=0x
 trusted.afr.gv01-client-1=0x
 trusted.gfid=0x35f86f4561134ba0bd1b94ef70179d4d

 [root@gluster1 gluster]# getfattr -d -m. -e hex gv01
 # file: gv01
 trusted.afr.gv01-client-0=0x
 trusted.afr.gv01-client-1=0x
 trusted.gfid=0x0001
 trusted.glusterfs.dht=0x0001
 trusted.glusterfs.volume-id=0x31a2c4c486ca4344b838d2c2e6c716c1



 On Tue, Sep 9, 2014 at 8:19 AM, Pranith Kumar Karampuri 
 pkara...@redhat.com wrote:


 On 09/09/2014 11:35 AM, Ilya Ivanov wrote:

  Ahh, thank you, now I get it. I deleted it on one node and it
 replicated to another one. Now I get the following output:

 [root@gluster1 var]# gluster volume heal gv01 info
 Brick gluster1:/home/gluster/gv01/
 gfid:d3def9e1-c6d0-4b7d-a322-b5019305182e
 Number of entries: 1

 Brick gluster2:/home/gluster/gv01/
 Number of entries: 0

  Is it normal? Why the number of entries isn't reset to 0?

  If you access the file using ls/stat etc, it will be fixed. But before
 that could you please post the output of 'getfattr -d -m. -e hex
 file/path/in/backend/brick' and 'getfattr -d -m. -e hex
 parent/dir/to/file/path/in/backend/brick'

 Pranith



 And why wouldn't the file show up in split-brain before, anyway?

  Gfid split-brains are not shown in heal-info-split-brain yet.

 Pranith



 On Tue, Sep 9, 2014 at 7:46 AM, Pranith Kumar Karampuri 
 pkara...@redhat.com wrote:


 On 09/09/2014 01:54 AM, Ilya Ivanov wrote:

  Hello.

  I've Gluster 3.5.2 on Centos 6. A primitive replicated volume, as
 describe here
 https://www.digitalocean.com/community/tutorials/how-to-create-a-redundant-storage-pool-using-glusterfs-on-ubuntu-servers.
 I tried to simulate split-brain by temporarily disconnecting the nodes and
 creating a file with the same name and different contents. That worked.

  The question is, how do I fix it now? All the tutorials suggest
 deleting the file from one of the nodes. I can't do that, it reports
 Input/output error. The file won't even show up in gluster volume heal
 gv00 info split-brain. That shows 0 entries.

  The deletion needs to happen on one of the bricks, not from the mount
 point.

 Pranith

  I can see the file in gluster volume heal gv00 info heal-failed,
 though.


  --
 Ilya.


  ___
 Gluster-users mailing 
 listGluster-users@gluster.orghttp://supercolony.gluster.org/mailman/listinfo/gluster-users





 --
 Ilya.





  --
 Ilya.




 --
 Ilya.





-- 
Ilya.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain that is not split brain

[Pranith Kumar Karampuri]

On 09/11/2014 09:29 AM, Ilya Ivanov wrote:

Right... I deleted it and now all appears to be fine.

Still, could you please elaborate on gfid split-brain?
Could you go through 
https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md
Let us know if you would like something to be more clearer and we can 
add that and improve the document.


Pranith



On Thu, Sep 11, 2014 at 5:32 AM, Pranith Kumar Karampuri 
pkara...@redhat.com mailto:pkara...@redhat.com wrote:



On 09/11/2014 12:16 AM, Ilya Ivanov wrote:

Any insight?

Was the other file's gfid d3def9e1-c6d0-4b7d-a322-b5019305182e?
Could you check if this file exists in brick/.glusterfs/d3/de/
When a file is deleted this file also needs to be deleted if there
are no more hardlinks to the file

Pranith


On Tue, Sep 9, 2014 at 8:35 AM, Ilya Ivanov bearw...@gmail.com
mailto:bearw...@gmail.com wrote:

What's a gfid split-brain and how is it different from
normal split-brain?

I accessed the file with stat, but heal info still shows
Number of entries: 1

[root@gluster1 gluster]# getfattr -d -m. -e hex gv01/123
# getfattr -d -m. -e hex gv01/123
# file: gv01/123
trusted.afr.gv01-client-0=0x
trusted.afr.gv01-client-1=0x
trusted.gfid=0x35f86f4561134ba0bd1b94ef70179d4d

[root@gluster1 gluster]# getfattr -d -m. -e hex gv01
# file: gv01
trusted.afr.gv01-client-0=0x
trusted.afr.gv01-client-1=0x
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x31a2c4c486ca4344b838d2c2e6c716c1



On Tue, Sep 9, 2014 at 8:19 AM, Pranith Kumar Karampuri
pkara...@redhat.com mailto:pkara...@redhat.com wrote:


On 09/09/2014 11:35 AM, Ilya Ivanov wrote:

Ahh, thank you, now I get it. I deleted it on one node
and it replicated to another one. Now I get the
following output:

[root@gluster1 var]# gluster volume heal gv01 info
Brick gluster1:/home/gluster/gv01/
gfid:d3def9e1-c6d0-4b7d-a322-b5019305182e
Number of entries: 1

Brick gluster2:/home/gluster/gv01/
Number of entries: 0

Is it normal? Why the number of entries isn't reset to 0?

If you access the file using ls/stat etc, it will be
fixed. But before that could you please post the output
of 'getfattr -d -m. -e hex file/path/in/backend/brick'
and 'getfattr -d -m. -e hex
parent/dir/to/file/path/in/backend/brick'

Pranith



And why wouldn't the file show up in split-brain before,
anyway?

Gfid split-brains are not shown in heal-info-split-brain yet.

Pranith



On Tue, Sep 9, 2014 at 7:46 AM, Pranith Kumar Karampuri
pkara...@redhat.com mailto:pkara...@redhat.com wrote:


On 09/09/2014 01:54 AM, Ilya Ivanov wrote:

Hello.

I've Gluster 3.5.2 on Centos 6. A primitive
replicated volume, as describe here

https://www.digitalocean.com/community/tutorials/how-to-create-a-redundant-storage-pool-using-glusterfs-on-ubuntu-servers.
I tried to simulate split-brain by temporarily
disconnecting the nodes and creating a file with
the same name and different contents. That worked.

The question is, how do I fix it now? All the
tutorials suggest deleting the file from one of the
nodes. I can't do that, it reports Input/output
error. The file won't even show up in gluster
volume heal gv00 info split-brain. That shows 0
entries.

The deletion needs to happen on one of the bricks,
not from the mount point.

Pranith

I can see the file in gluster volume heal gv00
info heal-failed, though.


-- 
Ilya.



___
Gluster-users mailing list
Gluster-users@gluster.org  mailto:Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users





-- 
Ilya.





-- 
Ilya.





-- 
Ilya.





--
Ilya.


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain that is not split brain

[Pranith Kumar Karampuri]

On 09/09/2014 11:35 AM, Ilya Ivanov wrote:
Ahh, thank you, now I get it. I deleted it on one node and it 
replicated to another one. Now I get the following output:


[root@gluster1 var]# gluster volume heal gv01 info
Brick gluster1:/home/gluster/gv01/
gfid:d3def9e1-c6d0-4b7d-a322-b5019305182e
Number of entries: 1

Brick gluster2:/home/gluster/gv01/
Number of entries: 0

Is it normal? Why the number of entries isn't reset to 0?
If you access the file using ls/stat etc, it will be fixed. But before 
that could you please post the output of 'getfattr -d -m. -e hex 
file/path/in/backend/brick' and 'getfattr -d -m. -e hex 
parent/dir/to/file/path/in/backend/brick'


Pranith



And why wouldn't the file show up in split-brain before, anyway?

Gfid split-brains are not shown in heal-info-split-brain yet.

Pranith



On Tue, Sep 9, 2014 at 7:46 AM, Pranith Kumar Karampuri 
pkara...@redhat.com mailto:pkara...@redhat.com wrote:



On 09/09/2014 01:54 AM, Ilya Ivanov wrote:

Hello.

I've Gluster 3.5.2 on Centos 6. A primitive replicated volume, as
describe here

https://www.digitalocean.com/community/tutorials/how-to-create-a-redundant-storage-pool-using-glusterfs-on-ubuntu-servers.
I tried to simulate split-brain by temporarily disconnecting the
nodes and creating a file with the same name and different
contents. That worked.

The question is, how do I fix it now? All the tutorials suggest
deleting the file from one of the nodes. I can't do that, it
reports Input/output error. The file won't even show up in
gluster volume heal gv00 info split-brain. That shows 0 entries.

The deletion needs to happen on one of the bricks, not from the
mount point.

Pranith

I can see the file in gluster volume heal gv00 info
heal-failed, though.


-- 
Ilya.



___
Gluster-users mailing list
Gluster-users@gluster.org  mailto:Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users





--
Ilya.


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain that is not split brain

[Ilya Ivanov]

What's a gfid split-brain and how is it different from normal split-brain?

I accessed the file with stat, but heal info still shows Number of
entries: 1

[root@gluster1 gluster]# getfattr -d -m. -e hex gv01/123
# getfattr -d -m. -e hex gv01/123
# file: gv01/123
trusted.afr.gv01-client-0=0x
trusted.afr.gv01-client-1=0x
trusted.gfid=0x35f86f4561134ba0bd1b94ef70179d4d

[root@gluster1 gluster]# getfattr -d -m. -e hex gv01
# file: gv01
trusted.afr.gv01-client-0=0x
trusted.afr.gv01-client-1=0x
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x31a2c4c486ca4344b838d2c2e6c716c1



On Tue, Sep 9, 2014 at 8:19 AM, Pranith Kumar Karampuri pkara...@redhat.com
 wrote:


 On 09/09/2014 11:35 AM, Ilya Ivanov wrote:

  Ahh, thank you, now I get it. I deleted it on one node and it replicated
 to another one. Now I get the following output:

 [root@gluster1 var]# gluster volume heal gv01 info
 Brick gluster1:/home/gluster/gv01/
 gfid:d3def9e1-c6d0-4b7d-a322-b5019305182e
 Number of entries: 1

 Brick gluster2:/home/gluster/gv01/
 Number of entries: 0

  Is it normal? Why the number of entries isn't reset to 0?

 If you access the file using ls/stat etc, it will be fixed. But before
 that could you please post the output of 'getfattr -d -m. -e hex
 file/path/in/backend/brick' and 'getfattr -d -m. -e hex
 parent/dir/to/file/path/in/backend/brick'

 Pranith



 And why wouldn't the file show up in split-brain before, anyway?

 Gfid split-brains are not shown in heal-info-split-brain yet.

 Pranith



 On Tue, Sep 9, 2014 at 7:46 AM, Pranith Kumar Karampuri 
 pkara...@redhat.com wrote:


 On 09/09/2014 01:54 AM, Ilya Ivanov wrote:

  Hello.

  I've Gluster 3.5.2 on Centos 6. A primitive replicated volume, as
 describe here
 https://www.digitalocean.com/community/tutorials/how-to-create-a-redundant-storage-pool-using-glusterfs-on-ubuntu-servers.
 I tried to simulate split-brain by temporarily disconnecting the nodes and
 creating a file with the same name and different contents. That worked.

  The question is, how do I fix it now? All the tutorials suggest
 deleting the file from one of the nodes. I can't do that, it reports
 Input/output error. The file won't even show up in gluster volume heal
 gv00 info split-brain. That shows 0 entries.

  The deletion needs to happen on one of the bricks, not from the mount
 point.

 Pranith

  I can see the file in gluster volume heal gv00 info heal-failed,
 though.


  --
 Ilya.


  ___
 Gluster-users mailing 
 listGluster-users@gluster.orghttp://supercolony.gluster.org/mailman/listinfo/gluster-users





 --
 Ilya.





-- 
Ilya.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain that is not split brain

[Ilya Ivanov]

Ahh, thank you, now I get it. I deleted it on one node and it replicated to
another one. Now I get the following output:

[root@gluster1 var]# gluster volume heal gv01 info
Brick gluster1:/home/gluster/gv01/
gfid:d3def9e1-c6d0-4b7d-a322-b5019305182e
Number of entries: 1

Brick gluster2:/home/gluster/gv01/
Number of entries: 0

Is it normal? Why the number of entries isn't reset to 0?


And why wouldn't the file show up in split-brain before, anyway?


On Tue, Sep 9, 2014 at 7:46 AM, Pranith Kumar Karampuri pkara...@redhat.com
 wrote:


 On 09/09/2014 01:54 AM, Ilya Ivanov wrote:

  Hello.

  I've Gluster 3.5.2 on Centos 6. A primitive replicated volume, as
 describe here
 https://www.digitalocean.com/community/tutorials/how-to-create-a-redundant-storage-pool-using-glusterfs-on-ubuntu-servers.
 I tried to simulate split-brain by temporarily disconnecting the nodes and
 creating a file with the same name and different contents. That worked.

  The question is, how do I fix it now? All the tutorials suggest deleting
 the file from one of the nodes. I can't do that, it reports Input/output
 error. The file won't even show up in gluster volume heal gv00 info
 split-brain. That shows 0 entries.

 The deletion needs to happen on one of the bricks, not from the mount
 point.

 Pranith

  I can see the file in gluster volume heal gv00 info heal-failed,
 though.


  --
 Ilya.


 ___
 Gluster-users mailing 
 listGluster-users@gluster.orghttp://supercolony.gluster.org/mailman/listinfo/gluster-users





-- 
Ilya.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain that is not split brain

[Pranith Kumar Karampuri]

On 09/09/2014 01:54 AM, Ilya Ivanov wrote:

Hello.

I've Gluster 3.5.2 on Centos 6. A primitive replicated volume, as 
describe here 
https://www.digitalocean.com/community/tutorials/how-to-create-a-redundant-storage-pool-using-glusterfs-on-ubuntu-servers. 
I tried to simulate split-brain by temporarily disconnecting the nodes 
and creating a file with the same name and different contents. That 
worked.


The question is, how do I fix it now? All the tutorials suggest 
deleting the file from one of the nodes. I can't do that, it reports 
Input/output error. The file won't even show up in gluster volume 
heal gv00 info split-brain. That shows 0 entries.

The deletion needs to happen on one of the bricks, not from the mount point.

Pranith

I can see the file in gluster volume heal gv00 info heal-failed, though.


--
Ilya.


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain on glusterfs running with quorum on server and client

[Pranith Kumar Karampuri]

What is the glusterfs version where you ran into this issue?

Pranith
On 09/05/2014 11:52 PM, Ramesh Natarajan wrote:
I have a replicate glusterfs setup on 3 Bricks ( replicate = 3 ). I 
have client and server quorum turned on.  I rebooted one of the 3 
bricks. When it came back up, the client started throwing error 
messages that one of the files went into split brain.


When i check the file sizes and sha1sum on the bricks, 2 of the 3 
bricks have the same value. So by quorum logic the first brick should 
have healed with this information. But i don't see that happening. Can 
someone please tell me if this is expected behavior?



Can someone please tell me if i have things misconfigured...

thanks
Ramesh

My config is as below.

[root@ip-172-31-12-218 ~]# gluster volume info
Volume Name: PL1
Type: Replicate
Volume ID: a7aabae0-c6bc-40a9-8b26-0498d488ee39
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 172.31.38.189:/data/vol1/gluster-data
Brick2: 172.31.16.220:/data/vol1/gluster-data
Brick3: 172.31.12.218:/data/vol1/gluster-data
Options Reconfigured:
performance.cache-size: 2147483648
nfs.addr-namelookup: off
network.ping-timeout: 12
cluster.server-quorum-type: server
nfs.enable-ino32: on
cluster.quorum-type: auto
cluster.server-quorum-ratio: 51%
Volume Name: PL2
Type: Replicate
Volume ID: fadb3671-7a92-40b7-bccd-fbacf672f6dc
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 172.31.38.189:/data/vol2/gluster-data
Brick2: 172.31.16.220:/data/vol2/gluster-data
Brick3: 172.31.12.218:/data/vol2/gluster-data
Options Reconfigured:
performance.cache-size: 2147483648
nfs.addr-namelookup: off
network.ping-timeout: 12
cluster.server-quorum-type: server
nfs.enable-ino32: on
cluster.quorum-type: auto
cluster.server-quorum-ratio: 51%
[root@ip-172-31-12-218 ~]#


I have 2 clients each mounting one of the volumes. At no time the same 
volume is mounted by more than 1 client.


mount -t glusterfs -o 
defaults,enable-ino32,direct-io-mode=disable,log-level=WARNING,log-file=/var/log/gluster.log,backupvolfile-server=172.31.38.189,backupvolfile-server=172.31.12.218,background-qlen=256 
172.31.16.220:/PL2  /mnt/vm



I restarted the Brick 1 172.31.38.189 and when it came up, one of the 
file on PL2 volume went into split mode..



[2014-09-05 17:59:42.997308] W [afr-open.c:209:afr_open] 
0-PL2-replicate-0: failed to open as split brain seen, returning EIO
[2014-09-05 17:59:42.997350] W [fuse-bridge.c:2209:fuse_writev_cbk] 
0-glusterfs-fuse: 3359683: WRITE = -1 (Input/output error)
[2014-09-05 17:59:42.997476] W [fuse-bridge.c:690:fuse_truncate_cbk] 
0-glusterfs-fuse: 3359684: FTRUNCATE() ERR = -1 (Input/
output error)[2014-09-05 17:59:42.997647] W 
[fuse-bridge.c:2209:fuse_writev_cbk] 0-glusterfs-fuse: 3359686: WRITE 
= -1 (Input/output erro
r)[2014-09-05 17:59:42.997783] W [fuse-bridge.c:1214:fuse_err_cbk] 
0-glusterfs-fuse: 3359687: FLUSH() ERR = -1 (Input/output e
rror)[2014-09-05 17:59:44.009187] E 
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 
0-PL2-replicate-0: Unable to self-he
al contents of '/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00' 
(possible split-brain). Please delete the file from all but the 
preferred subvolume.- Pending matrix:  [ [ 0 1 1 ] [ 3398 0 0 ] [ 3398 
0 0 ] ]
[2014-09-05 17:59:44.06] E 
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 
0-PL2-replicate-0:  backgroung data self heal  failed, on 
/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
[2014-09-05 17:59:44.011480] W [afr-open.c:209:afr_open] 
0-PL2-replicate-0: failed to open as split brain seen, returning EIO


Starting time of crawl: Fri Sep  5 17:55:32 2014

Ending time of crawl: Fri Sep  5 17:55:33 2014

Type of crawl: INDEX
No. of entries healed: 4
No. of entries in split-brain: 1
No. of heal failed entries: 0
[root@ip-172-31-16-220 ~]# gluster volume heal PL2 info
Brick ip-172-31-38-189:/data/vol2/gluster-data/
/apache_cp_mm1/logs/mm1.access_log.2014-09-05-17_00_00
Number of entries: 1

Brick ip-172-31-16-220:/data/vol2/gluster-data/
/apache_cp_mm1/logs/mm1.access_log.2014-09-05-17_00_00
Number of entries: 1

Brick ip-172-31-12-218:/data/vol2/gluster-data/
/apache_cp_mm1/logs/mm1.access_log.2014-09-05-17_00_00
Number of entries: 1


BRICK1


[root@ip-172-31-38-189 ~]# sha1sum access_log.2014-09-05-17_00_00
aa72d0f3949700f67b61d3c58fdbc75b772d607b  access_log.2014-09-05-17_00_00

[root@ip-172-31-38-189 ~]# ls -al
total 12760
dr-xr-x---  3 root root 4096 Sep  5 17:42 .
dr-xr-xr-x 24 root root 4096 Sep  5 17:34 ..
-rw-r-  1 root root 13019808 Sep  5 17:42 
access_log.2014-09-05-17_00_00


[root@ip-172-31-38-189 ~]# getfattr -d -m . -e hex 
 /data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00 


getfattr: Removing leading '/' from absolute path names
# file: 
data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00

Re: [Gluster-users] split-brain on glusterfs running with quorum on server and client

[Ramesh Natarajan]

3.5.2 installed via rpm from official gluster repo, running on amazon ami.

Thanks
Ramesh

 On Sep 6, 2014, at 7:29 AM, Pranith Kumar Karampuri pkara...@redhat.com 
 wrote:
 
 What is the glusterfs version where you ran into this issue?
 
 Pranith
 On 09/05/2014 11:52 PM, Ramesh Natarajan wrote:
 I have a replicate glusterfs setup on 3 Bricks ( replicate = 3 ). I have 
 client and server quorum turned on.  I rebooted one of the 3 bricks. When it 
 came back up, the client started throwing error messages that one of the 
 files went into split brain. 
 
 When i check the file sizes and sha1sum on the bricks, 2 of the 3 bricks 
 have the same value. So by quorum logic the first brick should have healed 
 with this information. But i don't see that happening. Can someone please 
 tell me if this is expected behavior?
 
 
 Can someone please tell me if i have things misconfigured...
 
 thanks
 Ramesh
 
 My config is as below.
 
 [root@ip-172-31-12-218 ~]# gluster volume info
  
 Volume Name: PL1
 Type: Replicate
 Volume ID: a7aabae0-c6bc-40a9-8b26-0498d488ee39
 Status: Started
 Number of Bricks: 1 x 3 = 3
 Transport-type: tcp
 Bricks:
 Brick1: 172.31.38.189:/data/vol1/gluster-data
 Brick2: 172.31.16.220:/data/vol1/gluster-data
 Brick3: 172.31.12.218:/data/vol1/gluster-data
 Options Reconfigured:
 performance.cache-size: 2147483648
 nfs.addr-namelookup: off
 network.ping-timeout: 12
 cluster.server-quorum-type: server
 nfs.enable-ino32: on
 cluster.quorum-type: auto
 cluster.server-quorum-ratio: 51%
  
 Volume Name: PL2
 Type: Replicate
 Volume ID: fadb3671-7a92-40b7-bccd-fbacf672f6dc
 Status: Started
 Number of Bricks: 1 x 3 = 3
 Transport-type: tcp
 Bricks:
 Brick1: 172.31.38.189:/data/vol2/gluster-data
 Brick2: 172.31.16.220:/data/vol2/gluster-data
 Brick3: 172.31.12.218:/data/vol2/gluster-data
 Options Reconfigured:
 performance.cache-size: 2147483648
 nfs.addr-namelookup: off
 network.ping-timeout: 12
 cluster.server-quorum-type: server
 nfs.enable-ino32: on
 cluster.quorum-type: auto
 cluster.server-quorum-ratio: 51%
 [root@ip-172-31-12-218 ~]# 
 
 
 I have 2 clients each mounting one of the volumes. At no time the same 
 volume is mounted by more than 1 client.
 
 mount -t glusterfs -o 
 defaults,enable-ino32,direct-io-mode=disable,log-level=WARNING,log-file=/var/log/gluster.log,backupvolfile-server=172.31.38.189,backupvolfile-server=172.31.12.218,background-qlen=256
  172.31.16.220:/PL2  /mnt/vm
 
 
 I restarted the Brick 1 172.31.38.189 and when it came up, one of the file 
 on PL2 volume went into split mode..
 
 
 [2014-09-05 17:59:42.997308] W [afr-open.c:209:afr_open] 0-PL2-replicate-0: 
 failed to open as split brain seen, returning EIO
 [2014-09-05 17:59:42.997350] W [fuse-bridge.c:2209:fuse_writev_cbk] 
 0-glusterfs-fuse: 3359683: WRITE = -1 (Input/output error)
 [2014-09-05 17:59:42.997476] W [fuse-bridge.c:690:fuse_truncate_cbk] 
 0-glusterfs-fuse: 3359684: FTRUNCATE() ERR = -1 (Input/
 output error)[2014-09-05 17:59:42.997647] W 
 [fuse-bridge.c:2209:fuse_writev_cbk] 0-glusterfs-fuse: 3359686: WRITE = -1 
 (Input/output erro
 r)[2014-09-05 17:59:42.997783] W [fuse-bridge.c:1214:fuse_err_cbk] 
 0-glusterfs-fuse: 3359687: FLUSH() ERR = -1 (Input/output e
 rror)[2014-09-05 17:59:44.009187] E 
 [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-PL2-replicate-0: 
 Unable to self-he
 al contents of '/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00' 
 (possible split-brain). Please delete the file from all but the preferred 
 subvolume.- Pending matrix:  [ [ 0 1 1 ] [ 3398 0 0 ] [ 3398 0 0 ] ]
 [2014-09-05 17:59:44.06] E 
 [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 
 0-PL2-replicate-0:  backgroung data self heal  failed,   on 
 /apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
 [2014-09-05 17:59:44.011480] W [afr-open.c:209:afr_open] 0-PL2-replicate-0: 
 failed to open as split brain seen, returning EIO
 
 Starting time of crawl: Fri Sep  5 17:55:32 2014
 
 Ending time of crawl: Fri Sep  5 17:55:33 2014
 
 Type of crawl: INDEX
 No. of entries healed: 4
 No. of entries in split-brain: 1
 No. of heal failed entries: 0
 [root@ip-172-31-16-220 ~]# gluster volume heal PL2 info
 Brick ip-172-31-38-189:/data/vol2/gluster-data/
 /apache_cp_mm1/logs/mm1.access_log.2014-09-05-17_00_00
 Number of entries: 1
 
 Brick ip-172-31-16-220:/data/vol2/gluster-data/
 /apache_cp_mm1/logs/mm1.access_log.2014-09-05-17_00_00
 Number of entries: 1
 
 Brick ip-172-31-12-218:/data/vol2/gluster-data/
 /apache_cp_mm1/logs/mm1.access_log.2014-09-05-17_00_00
 Number of entries: 1
 
 
 BRICK1
 
 
 [root@ip-172-31-38-189 ~]# sha1sum access_log.2014-09-05-17_00_00
 aa72d0f3949700f67b61d3c58fdbc75b772d607b  access_log.2014-09-05-17_00_00
 
 [root@ip-172-31-38-189 ~]# ls -al 
 total 12760
 dr-xr-x---  3 root root 4096 Sep  5 17:42 .
 dr-xr-xr-x 24 root root 4096 Sep  5 17:34 ..
 -rw-r-  1 root root 13019808 Sep  5 17:42 
 access_log.2014-09-05-17_00_00

Re: [Gluster-users] split-brain on glusterfs running with quorum on server and client

[Pranith Kumar Karampuri]

On 09/06/2014 06:17 PM, Ramesh Natarajan wrote:

3.5.2 installed via rpm from official gluster repo, running on amazon ami.

Including the client i.e. mount bits?

Pranith


Thanks
Ramesh

On Sep 6, 2014, at 7:29 AM, Pranith Kumar Karampuri 
pkara...@redhat.com mailto:pkara...@redhat.com wrote:



What is the glusterfs version where you ran into this issue?

Pranith
On 09/05/2014 11:52 PM, Ramesh Natarajan wrote:
I have a replicate glusterfs setup on 3 Bricks ( replicate = 3 ). I 
have client and server quorum turned on.  I rebooted one of the 3 
bricks. When it came back up, the client started throwing error 
messages that one of the files went into split brain.


When i check the file sizes and sha1sum on the bricks, 2 of the 3 
bricks have the same value. So by quorum logic the first brick 
should have healed with this information. But i don't see that 
happening. Can someone please tell me if this is expected behavior?



Can someone please tell me if i have things misconfigured...

thanks
Ramesh

My config is as below.

[root@ip-172-31-12-218 ~]# gluster volume info
Volume Name: PL1
Type: Replicate
Volume ID: a7aabae0-c6bc-40a9-8b26-0498d488ee39
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 172.31.38.189:/data/vol1/gluster-data
Brick2: 172.31.16.220:/data/vol1/gluster-data
Brick3: 172.31.12.218:/data/vol1/gluster-data
Options Reconfigured:
performance.cache-size: 2147483648
nfs.addr-namelookup: off
network.ping-timeout: 12
cluster.server-quorum-type: server
nfs.enable-ino32: on
cluster.quorum-type: auto
cluster.server-quorum-ratio: 51%
Volume Name: PL2
Type: Replicate
Volume ID: fadb3671-7a92-40b7-bccd-fbacf672f6dc
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 172.31.38.189:/data/vol2/gluster-data
Brick2: 172.31.16.220:/data/vol2/gluster-data
Brick3: 172.31.12.218:/data/vol2/gluster-data
Options Reconfigured:
performance.cache-size: 2147483648
nfs.addr-namelookup: off
network.ping-timeout: 12
cluster.server-quorum-type: server
nfs.enable-ino32: on
cluster.quorum-type: auto
cluster.server-quorum-ratio: 51%
[root@ip-172-31-12-218 ~]#


I have 2 clients each mounting one of the volumes. At no time the 
same volume is mounted by more than 1 client.


mount -t glusterfs -o 
defaults,enable-ino32,direct-io-mode=disable,log-level=WARNING,log-file=/var/log/gluster.log,backupvolfile-server=172.31.38.189,backupvolfile-server=172.31.12.218,background-qlen=256 
172.31.16.220:/PL2  /mnt/vm



I restarted the Brick 1 172.31.38.189 and when it came up, one of 
the file on PL2 volume went into split mode..



[2014-09-05 17:59:42.997308] W [afr-open.c:209:afr_open] 
0-PL2-replicate-0: failed to open as split brain seen, returning EIO
[2014-09-05 17:59:42.997350] W [fuse-bridge.c:2209:fuse_writev_cbk] 
0-glusterfs-fuse: 3359683: WRITE = -1 (Input/output error)
[2014-09-05 17:59:42.997476] W [fuse-bridge.c:690:fuse_truncate_cbk] 
0-glusterfs-fuse: 3359684: FTRUNCATE() ERR = -1 (Input/
output error)[2014-09-05 17:59:42.997647] W 
[fuse-bridge.c:2209:fuse_writev_cbk] 0-glusterfs-fuse: 3359686: 
WRITE = -1 (Input/output erro
r)[2014-09-05 17:59:42.997783] W [fuse-bridge.c:1214:fuse_err_cbk] 
0-glusterfs-fuse: 3359687: FLUSH() ERR = -1 (Input/output e
rror)[2014-09-05 17:59:44.009187] E 
[afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 
0-PL2-replicate-0: Unable to self-he
al contents of '/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00' 
(possible split-brain). Please delete the file from all but the 
preferred subvolume.- Pending matrix:  [ [ 0 1 1 ] [ 3398 0 0 ] [ 
3398 0 0 ] ]
[2014-09-05 17:59:44.06] E 
[afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 
0-PL2-replicate-0:  backgroung data self heal  failed,   on 
/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
[2014-09-05 17:59:44.011480] W [afr-open.c:209:afr_open] 
0-PL2-replicate-0: failed to open as split brain seen, returning EIO


Starting time of crawl: Fri Sep  5 17:55:32 2014

Ending time of crawl: Fri Sep  5 17:55:33 2014

Type of crawl: INDEX
No. of entries healed: 4
No. of entries in split-brain: 1
No. of heal failed entries: 0
[root@ip-172-31-16-220 ~]# gluster volume heal PL2 info
Brick ip-172-31-38-189:/data/vol2/gluster-data/
/apache_cp_mm1/logs/mm1.access_log.2014-09-05-17_00_00
Number of entries: 1

Brick ip-172-31-16-220:/data/vol2/gluster-data/
/apache_cp_mm1/logs/mm1.access_log.2014-09-05-17_00_00
Number of entries: 1

Brick ip-172-31-12-218:/data/vol2/gluster-data/
/apache_cp_mm1/logs/mm1.access_log.2014-09-05-17_00_00
Number of entries: 1


BRICK1


[root@ip-172-31-38-189 ~]# sha1sum access_log.2014-09-05-17_00_00
aa72d0f3949700f67b61d3c58fdbc75b772d607b  access_log.2014-09-05-17_00_00

[root@ip-172-31-38-189 ~]# ls -al
total 12760
dr-xr-x---  3 root root 4096 Sep  5 17:42 .
dr-xr-xr-x 24 root root 4096 Sep  5 17:34 ..
-rw-r-  1 root root 13019808 Sep  5 17:42 
access_log.2014-09-05-17_00_00

Re: [Gluster-users] split-brain on glusterfs running with quorum on server and client

[Ramesh Natarajan]

Client is also running the same version. It is running RHEL 6.5. 

The mount command used is

 mount -t glusterfs -o 
 defaults,enable-ino32,direct-io-mode=disable,log-level=WARNING,log-file=/var/log/gluster.log,backupvolfile-server=172.31.38.189,backupvolfile-server=172.31.12.218,background-qlen=256
  172.31.16.220:/PL2  /mnt/vm


 On Sep 6, 2014, at 7:59 AM, Pranith Kumar Karampuri pkara...@redhat.com 
 wrote:
 
 mount -t glusterfs -o 
 defaults,enable-ino32,direct-io-mode=disable,log-level=WARNING,log-file=/var/log/gluster.log,backupvolfile-server=172.31.38.189,backupvolfile-server=172.31.12.218,background-qlen=256
  172.31.16.220:/PL2  /mnt/vm
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain on glusterfs running with quorum on server and client

[Pranith Kumar Karampuri]

On 09/06/2014 04:53 AM, Jeff Darcy wrote:

I have a replicate glusterfs setup on 3 Bricks ( replicate = 3 ). I have
client and server quorum turned on. I rebooted one of the 3 bricks. When it
came back up, the client started throwing error messages that one of the
files went into split brain.

This is a good example of how split brain can happen even with all kinds of
quorum enabled.  Let's look at those xattrs.  BTW, thank you for a very
nicely detailed bug report which includes those.


BRICK1

[root@ip-172-31-38-189 ~]# getfattr -d -m . -e hex
/data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
getfattr: Removing leading '/' from absolute path names
# file:
data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
trusted.afr.PL2-client-0=0x
trusted.afr.PL2-client-1=0x0001
trusted.afr.PL2-client-2=0x0001
trusted.gfid=0xea950263977e46bf89a0ef631ca139c2

BRICK 2
===
[root@ip-172-31-16-220 ~]# getfattr -d -m . -e hex
/data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
getfattr: Removing leading '/' from absolute path names
# file:
data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
trusted.afr.PL2-client-0=0x0d46
trusted.afr.PL2-client-1=0x
trusted.afr.PL2-client-2=0x
trusted.gfid=0xea950263977e46bf89a0ef631ca139c2
BRICK 3
=
[root@ip-172-31-12-218 ~]# getfattr -d -m . -e hex
/data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
getfattr: Removing leading '/' from absolute path names
# file:
data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
trusted.afr.PL2-client-0=0x0d46
trusted.afr.PL2-client-1=0x
trusted.afr.PL2-client-2=0x
trusted.gfid=0xea950263977e46bf89a0ef631ca139c2

Here, we see that brick 1 shows a single pending operation for the other
two, while they show 0xd46 (3398) pending operations for brick 1.
Here's how this can happen.

(1) There is exactly one pending operation.

(2) Brick1 completes the write first, and says so.

(3) Client sends messages to all three, saying to decrement brick1's
count.

(4) All three bricks receive and process that message.

(5) Brick1 fails.

(6) Brick2 and brick3 complete the write, and say so.

(7) Client tells all bricks to decrement remaining counts.

(8) Brick2 and brick3 receive and process that message.

(9) Brick1 is dead, so its counts for brick2/3 stay at one.

(10) Brick2 and brick3 have quorum, with all-zero pending counters.

(11) Client sends 0xd46 more writes to brick2 and brick3.

Note that at no point did we lose quorum. Note also the tight timing
required.  If brick1 had failed an instant earlier, it would not have
decremented its own counter.  If it had failed an instant later, it
would have decremented brick2's and brick3's as well.  If brick1 had not
finished first, we'd be in yet another scenario.  If delayed changelog
had been operative, the messages at (3) and (7) would have been combined
to leave us in yet another scenario.  As far as I can tell, we would
have been able to resolve the conflict in all those cases.
*** Key point: quorum enforcement does not totally eliminate split
brain.  It only makes the frequency a few orders of magnitude lower. ***


Not quite right. After we fixed the bug 
https://bugzilla.redhat.com/show_bug.cgi?id=1066996, the only two 
possible ways to introduce split-brain are
1) if we have an implementation bug in changelog xattr marking, I 
believe that to be the case here.

2) Keep writing to the file from the mount then
a) take brick 1 down, wait until at least one write is successful
b) bring brick1 back up and take brick 2 down (self-heal should not 
happen) wait until at least one write is successful
c) bring brick2 back up and take brick 3 down (self-heal should not 
happen) wait until at least one write is successful


With outcast implementation case-2 will also be immune to split-brain 
errors.


Then the only way we have split-brains in afr is implementation errors 
of changelog marking. If we test it thoroughly and fix such problems we 
can get it to be immune to split-brain :-).


Pranith

So, is there any way to prevent this completely?  Some AFR enhancements,
such as the oft-promised outcast feature[1], might have helped.
NSR[2] is immune to this particular problem.  Policy based split brain
resolution[3] might have resolved it automatically instead of merely
flagging it.  Unfortunately, those are all in the future.  For now, I'd
say the best approach is to resolve the conflict manually and try to
move on.  Unless there's more going on than meets the eye, recurrence
should be very unlikely.

[1] http://www.gluster.org/community/documentation/index.php/Features/outcast

[2] 
http://www.gluster.org/community/documentation/index.php/Features/new-style-replication

Re: [Gluster-users] split-brain on glusterfs running with quorum on server and client

[Jeff Darcy]

 I have a replicate glusterfs setup on 3 Bricks ( replicate = 3 ). I have
 client and server quorum turned on. I rebooted one of the 3 bricks. When it
 came back up, the client started throwing error messages that one of the
 files went into split brain.

This is a good example of how split brain can happen even with all kinds of
quorum enabled.  Let's look at those xattrs.  BTW, thank you for a very
nicely detailed bug report which includes those.

 BRICK1
 
 [root@ip-172-31-38-189 ~]# getfattr -d -m . -e hex
 /data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
 getfattr: Removing leading '/' from absolute path names
 # file:
 data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
 trusted.afr.PL2-client-0=0x
 trusted.afr.PL2-client-1=0x0001
 trusted.afr.PL2-client-2=0x0001
 trusted.gfid=0xea950263977e46bf89a0ef631ca139c2

 BRICK 2
 ===
 [root@ip-172-31-16-220 ~]# getfattr -d -m . -e hex
 /data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
 getfattr: Removing leading '/' from absolute path names
 # file:
 data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
 trusted.afr.PL2-client-0=0x0d46
 trusted.afr.PL2-client-1=0x
 trusted.afr.PL2-client-2=0x
 trusted.gfid=0xea950263977e46bf89a0ef631ca139c2

 BRICK 3
 =
 [root@ip-172-31-12-218 ~]# getfattr -d -m . -e hex
 /data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
 getfattr: Removing leading '/' from absolute path names
 # file:
 data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
 trusted.afr.PL2-client-0=0x0d46
 trusted.afr.PL2-client-1=0x
 trusted.afr.PL2-client-2=0x
 trusted.gfid=0xea950263977e46bf89a0ef631ca139c2

Here, we see that brick 1 shows a single pending operation for the other
two, while they show 0xd46 (3398) pending operations for brick 1.
Here's how this can happen.

(1) There is exactly one pending operation.

(2) Brick1 completes the write first, and says so.

(3) Client sends messages to all three, saying to decrement brick1's
count.

(4) All three bricks receive and process that message.

(5) Brick1 fails.

(6) Brick2 and brick3 complete the write, and say so.

(7) Client tells all bricks to decrement remaining counts.

(8) Brick2 and brick3 receive and process that message.

(9) Brick1 is dead, so its counts for brick2/3 stay at one.

(10) Brick2 and brick3 have quorum, with all-zero pending counters.

(11) Client sends 0xd46 more writes to brick2 and brick3.

Note that at no point did we lose quorum. Note also the tight timing
required.  If brick1 had failed an instant earlier, it would not have
decremented its own counter.  If it had failed an instant later, it
would have decremented brick2's and brick3's as well.  If brick1 had not
finished first, we'd be in yet another scenario.  If delayed changelog
had been operative, the messages at (3) and (7) would have been combined
to leave us in yet another scenario.  As far as I can tell, we would
have been able to resolve the conflict in all those cases.

*** Key point: quorum enforcement does not totally eliminate split
brain.  It only makes the frequency a few orders of magnitude lower. ***

So, is there any way to prevent this completely?  Some AFR enhancements,
such as the oft-promised outcast feature[1], might have helped.
NSR[2] is immune to this particular problem.  Policy based split brain
resolution[3] might have resolved it automatically instead of merely
flagging it.  Unfortunately, those are all in the future.  For now, I'd
say the best approach is to resolve the conflict manually and try to
move on.  Unless there's more going on than meets the eye, recurrence
should be very unlikely.

[1] http://www.gluster.org/community/documentation/index.php/Features/outcast

[2] 
http://www.gluster.org/community/documentation/index.php/Features/new-style-replication

[3] http://www.gluster.org/community/documentation/index.php/Features/pbspbr
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split-brain on glusterfs running with quorum on server and client

[Ramesh Natarajan]

Thanks Jeff for the detailed explanation. You mentioned delayed changelog may 
have prevented this issue.  Can you please tell me how to enable it?


Thanks
Ramesh

On Sep 5, 2014, at 6:23 PM, Jeff Darcy jda...@redhat.com wrote:

 I have a replicate glusterfs setup on 3 Bricks ( replicate = 3 ). I have
 client and server quorum turned on. I rebooted one of the 3 bricks. When it
 came back up, the client started throwing error messages that one of the
 files went into split brain.
 
 This is a good example of how split brain can happen even with all kinds of
 quorum enabled.  Let's look at those xattrs.  BTW, thank you for a very
 nicely detailed bug report which includes those.
 
 BRICK1
 
 [root@ip-172-31-38-189 ~]# getfattr -d -m . -e hex
 /data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
 getfattr: Removing leading '/' from absolute path names
 # file:
 data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
 trusted.afr.PL2-client-0=0x
 trusted.afr.PL2-client-1=0x0001
 trusted.afr.PL2-client-2=0x0001
 trusted.gfid=0xea950263977e46bf89a0ef631ca139c2
 
 BRICK 2
 ===
 [root@ip-172-31-16-220 ~]# getfattr -d -m . -e hex
 /data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
 getfattr: Removing leading '/' from absolute path names
 # file:
 data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
 trusted.afr.PL2-client-0=0x0d46
 trusted.afr.PL2-client-1=0x
 trusted.afr.PL2-client-2=0x
 trusted.gfid=0xea950263977e46bf89a0ef631ca139c2
 
 BRICK 3
 =
 [root@ip-172-31-12-218 ~]# getfattr -d -m . -e hex
 /data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
 getfattr: Removing leading '/' from absolute path names
 # file:
 data/vol2/gluster-data/apache_cp_mm1/logs/access_log.2014-09-05-17_00_00
 trusted.afr.PL2-client-0=0x0d46
 trusted.afr.PL2-client-1=0x
 trusted.afr.PL2-client-2=0x
 trusted.gfid=0xea950263977e46bf89a0ef631ca139c2
 
 Here, we see that brick 1 shows a single pending operation for the other
 two, while they show 0xd46 (3398) pending operations for brick 1.
 Here's how this can happen.
 
 (1) There is exactly one pending operation.
 
 (2) Brick1 completes the write first, and says so.
 
 (3) Client sends messages to all three, saying to decrement brick1's
 count.
 
 (4) All three bricks receive and process that message.
 
 (5) Brick1 fails.
 
 (6) Brick2 and brick3 complete the write, and say so.
 
 (7) Client tells all bricks to decrement remaining counts.
 
 (8) Brick2 and brick3 receive and process that message.
 
 (9) Brick1 is dead, so its counts for brick2/3 stay at one.
 
 (10) Brick2 and brick3 have quorum, with all-zero pending counters.
 
 (11) Client sends 0xd46 more writes to brick2 and brick3.
 
 Note that at no point did we lose quorum. Note also the tight timing
 required.  If brick1 had failed an instant earlier, it would not have
 decremented its own counter.  If it had failed an instant later, it
 would have decremented brick2's and brick3's as well.  If brick1 had not
 finished first, we'd be in yet another scenario.  If delayed changelog
 had been operative, the messages at (3) and (7) would have been combined
 to leave us in yet another scenario.  As far as I can tell, we would
 have been able to resolve the conflict in all those cases.
 
 *** Key point: quorum enforcement does not totally eliminate split
 brain.  It only makes the frequency a few orders of magnitude lower. ***
 
 So, is there any way to prevent this completely?  Some AFR enhancements,
 such as the oft-promised outcast feature[1], might have helped.
 NSR[2] is immune to this particular problem.  Policy based split brain
 resolution[3] might have resolved it automatically instead of merely
 flagging it.  Unfortunately, those are all in the future.  For now, I'd
 say the best approach is to resolve the conflict manually and try to
 move on.  Unless there's more going on than meets the eye, recurrence
 should be very unlikely.
 
 [1] http://www.gluster.org/community/documentation/index.php/Features/outcast
 
 [2] 
 http://www.gluster.org/community/documentation/index.php/Features/new-style-replication
 
 [3] http://www.gluster.org/community/documentation/index.php/Features/pbspbr
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain on directory?

[david.zhang700]

looks weird,but I don't think it's a strictly defined split brain.can you 
paste the output of getattr command on the directory you mentioned?


-原始邮件- 
From: Alan Orth

Sent: Monday, July 28, 2014 6:09 PM
To: gluster-users
Subject: [Gluster-users] Split brain on directory?

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users 


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain on directory?

[Alan Orth]

Hi,

Yeah, it's some sort of split brain in that it exists on one brick and
not the other :P

Here's the getfattr for the directory at the end of the hierarchy in the
brick:

# getfattr -n trusted.gfid --absolute-names -e hex
/mnt/gfs/wingu1/sdb1/homes/mnt/gfs/wingu1/sdb1/homes/
# file: /mnt/gfs/wingu1/sdb1/homes/mnt/gfs/wingu1/sdb1/homes/
trusted.gfid=0x0001

Thanks,

On 07/28/2014 02:19 PM, david.zhang...@gmail.com wrote:
 looks weird,but I don't think it's a strictly defined split brain.can
 you paste the output of getattr command on the directory you mentioned?
 
 -原始邮件- From: Alan Orth
 Sent: Monday, July 28, 2014 6:09 PM
 To: gluster-users
 Subject: [Gluster-users] Split brain on directory?
 
 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 http://supercolony.gluster.org/mailman/listinfo/gluster-users


-- 
Alan Orth
alan.o...@gmail.com
http://alaninkenya.org
http://mjanja.co.ke
I have always wished for my computer to be as easy to use as my
telephone; my wish has come true because I can no longer figure out how
to use my telephone. -Bjarne Stroustrup, inventor of C++
GPG public key ID: 0x8cb0d0acb5cd81ec209c6cdfbd1a0e09c2f836c0



signature.asc
Description: OpenPGP digital signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain on directory?

[david.zhang700]

please run:
getfattr -de hex -m .  the_directory

-原始邮件- 
From: Alan Orth

Sent: Monday, July 28, 2014 7:42 PM
To: david.zhang...@gmail.com ; gluster-users
Subject: Re: [Gluster-users] Split brain on directory?

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain on directory?

[Joe Julian]

Yes, it's split brain when there are conflicting definitions for a filesystem 
entry. One brick has a directory and the other a file. That definitely meets 
the criteria.

Yes, it looks like you should be able to delete the errant directory tree.

On July 28, 2014 4:19:27 AM PDT, david.zhang...@gmail.com wrote:
looks weird,but I don't think it's a strictly defined split brain.can
you 
paste the output of getattr command on the directory you mentioned?

-原始邮件- 
From: Alan Orth
Sent: Monday, July 28, 2014 6:09 PM
To: gluster-users
Subject: [Gluster-users] Split brain on directory?

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users 

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split-brain

[Joe Julian]

On 02/20/2014 02:43 PM, William Kwan wrote:

Hi all,

Running glusterfs-3.4.2-1.el6.x86_6 on centos6.5

Due to some smart people screw up my network connection on the nodes 
for don't know how long. I found that I have my GlusterFS volume in 
split-brain.  I googled and found different way to clean this.  I need 
some extra help on this.


# gluster volume heal kvm1 info split-brain
Gathering Heal info on volume kvm1 has been successful

Brick mgmt1:/gluster/brick1
Number of entries: 21
atpath on brick
---
2014-02-20 22:33:41 
/d058a735-0fca-430a-a3d7-cf0a77097e5d/images/714c56a8-db1d-42d5-bf76-869bd6c87eef/0ea0a280-4c2c-48ab-ad95-8cb48e6cf02b
2014-02-20 22:33:41 
/d058a735-0fca-430a-a3d7-cf0a77097e5d/images/20b728b6-dd39-4d2e-a5c0-2dee22df6e95/a6a9b083-b04c-4ac8-86cb-ed4eb697c2c3

2014-02-20 22:33:41 /d058a735-0fca-430a-a3d7-cf0a77097e5d/dom_md/ids
... truncated

Brick mgmt2:/gluster/brick1
Number of entries: 28
atpath on brick
---
2014-02-20 22:37:38 /d058a735-0fca-430a-a3d7-cf0a77097e5d/dom_md/ids
2014-02-20 22:37:38 
/d058a735-0fca-430a-a3d7-cf0a77097e5d/images/714c56a8-db1d-42d5-bf76-869bd6c87eef/0ea0a280-4c2c-48ab-ad95-8cb48e6cf02b
2014-02-20 22:37:38 
/d058a735-0fca-430a-a3d7-cf0a77097e5d/images/20b728b6-dd39-4d2e-a5c0-2dee22df6e95/a6a9b083-b04c-4ac8-86cb-ed4eb697c2c3

2014-02-20 22:27:38 /d058a735-0fca-430a-a3d7-cf0a77097e5d/dom_md/ids
2014-02-20 22:27:38 /d058a735-0fca-430a-a3d7-cf0
... truncated


1. what's the best way?
Here's the write-up I did about split-brain: 
http://joejulian.name/blog/fixing-split-brain-with-glusterfs-33/


2.gluster volume heal doesnt' really save this, right?
No, the nature of split-brain is such that there is no automated way to 
recover from it.


3. kind of shooting from the dark as I can't see the data content. The 
volume is holding VM images.  Picking the latest copies should be good?
That does seem a reasonably safe assumption, especially if your vm's are 
cattle instead of kittens 
http://etherealmind.com/cattle-vs-kittens-on-cloud-platforms-no-one-hears-the-kittens-dying/.


___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split-brain

[William Kwan]

Hi all,

really need some guidance here.  Is this a good direction?
http://www.gluster.org/2012/07/fixing-split-brain-with-glusterfs-3-3/





On Thursday, February 20, 2014 5:43 PM, William Kwan pota...@yahoo.com wrote:
 
Hi all,

Running glusterfs-3.4.2-1.el6.x86_6 on centos6.5

Due to some smart people screw up my network connection on the nodes for don't 
know how long. I found that I have my GlusterFS volume in split-brain.  I 
googled and found different way to clean this.  I need some extra help on this.

# gluster volume heal kvm1 info split-brain
Gathering Heal info on volume kvm1 has been successful

Brick mgmt1:/gluster/brick1
Number of entries: 21
at    path on brick
---
2014-02-20 22:33:41
 
/d058a735-0fca-430a-a3d7-cf0a77097e5d/images/714c56a8-db1d-42d5-bf76-869bd6c87eef/0ea0a280-4c2c-48ab-ad95-8cb48e6cf02b
2014-02-20 22:33:41 
/d058a735-0fca-430a-a3d7-cf0a77097e5d/images/20b728b6-dd39-4d2e-a5c0-2dee22df6e95/a6a9b083-b04c-4ac8-86cb-ed4eb697c2c3
2014-02-20 22:33:41 /d058a735-0fca-430a-a3d7-cf0a77097e5d/dom_md/ids
... truncated

Brick mgmt2:/gluster/brick1
Number of entries: 28
at    path on brick
---
2014-02-20 22:37:38 /d058a735-0fca-430a-a3d7-cf0a77097e5d/dom_md/ids
2014-02-20 22:37:38 
/d058a735-0fca-430a-a3d7-cf0a77097e5d/images/714c56a8-db1d-42d5-bf76-869bd6c87eef/0ea0a280-4c2c-48ab-ad95-8cb48e6cf02b
2014-02-20 22:37:38 
/d058a735-0fca-430a-a3d7-cf0a77097e5d/images/20b728b6-dd39-4d2e-a5c0-2dee22df6e95/a6a9b083-b04c-4ac8-86cb-ed4eb697c2c3
2014-02-20
 22:27:38 /d058a735-0fca-430a-a3d7-cf0a77097e5d/dom_md/ids
2014-02-20 22:27:38 /d058a735-0fca-430a-a3d7-cf0
... truncated


1. what's the best way?  

2.gluster volume heal doesnt' really save this, right?

3. kind of shooting from the dark as I can't see the data content. The volume 
is holding VM images.  Picking the latest copies should be good?

Thanks
Will___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split-brain

[William Kwan]

Hi all,

really need some guidance here.  Is this a good direction?
http://www.gluster.org/2012/07/fixing-split-brain-with-glusterfs-3-3/





On Thursday, February 20, 2014 5:43 PM, William Kwan pota...@yahoo.com wrote:
 
Hi all,

Running glusterfs-3.4.2-1.el6.x86_6 on centos6.5

Due to some smart people screw up my network connection on the nodes for don't 
know how long. I found that I have my GlusterFS volume in split-brain.  I 
googled and found different way to clean this.  I need some extra help on this.

# gluster volume heal kvm1 info split-brain
Gathering Heal info on volume kvm1 has been successful

Brick mgmt1:/gluster/brick1
Number of entries: 21
at    path on brick
---
2014-02-20 22:33:41
 
/d058a735-0fca-430a-a3d7-cf0a77097e5d/images/714c56a8-db1d-42d5-bf76-869bd6c87eef/0ea0a280-4c2c-48ab-ad95-8cb48e6cf02b
2014-02-20 22:33:41 
/d058a735-0fca-430a-a3d7-cf0a77097e5d/images/20b728b6-dd39-4d2e-a5c0-2dee22df6e95/a6a9b083-b04c-4ac8-86cb-ed4eb697c2c3
2014-02-20 22:33:41 /d058a735-0fca-430a-a3d7-cf0a77097e5d/dom_md/ids
... truncated

Brick mgmt2:/gluster/brick1
Number of entries: 28
at    path on brick
---
2014-02-20 22:37:38 /d058a735-0fca-430a-a3d7-cf0a77097e5d/dom_md/ids
2014-02-20 22:37:38 
/d058a735-0fca-430a-a3d7-cf0a77097e5d/images/714c56a8-db1d-42d5-bf76-869bd6c87eef/0ea0a280-4c2c-48ab-ad95-8cb48e6cf02b
2014-02-20 22:37:38 
/d058a735-0fca-430a-a3d7-cf0a77097e5d/images/20b728b6-dd39-4d2e-a5c0-2dee22df6e95/a6a9b083-b04c-4ac8-86cb-ed4eb697c2c3
2014-02-20
 22:27:38 /d058a735-0fca-430a-a3d7-cf0a77097e5d/dom_md/ids
2014-02-20 22:27:38 /d058a735-0fca-430a-a3d7-cf0
... truncated


1. what's the best way?  

2.gluster volume heal doesnt' really save this, right?

3. kind of shooting from the dark as I can't see the data content. The volume 
is holding VM images.  Picking the latest copies should be good?

Thanks
Will___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain files show no filename/path with gfid

[raghav]

This could happen if under a split brain, the files have different gfids 
on different replicated bricks. Volume info heal would then display only 
the file names.


Regards
Raghav
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split brain recovery 3.3.0

[samuel]

No one around can help with this situation?

The file finally got corrupted and it's impossible to read from any of
the current mounted native gluster clients.

Reading the attributes, all flags are set as 0:

# file: dd29900f215b175939816f94c907a31b
trusted.afr.storage-client-6=0x
trusted.afr.storage-client-7=0x
trusted.gfid=0x59311906edf04464be5f00f505b3aebb
trusted.storage-stripe-1.stripe-count=0x3200
trusted.storage-stripe-1.stripe-index=0x3100
trusted.storage-stripe-1.stripe-size=0x31333130373200

so there should not be read as split brain from clients but we got the
following logs:
split brain detected during lookup of

thanks in advance,
Samuel.


On 4 March 2013 14:37, samuel sam...@gmail.com wrote:

 Hi folks,

 We have detected a split-brain on 3.3.0 stripped replicated 8 nodes
 cluster.
 We've followed the instructions from:
 http://www.gluster.org/2012/07/fixing-split-brain-with-glusterfs-3-3/
 and we've manually recovered the information.

 The problem is that we've got 4 gluster native clients and from 2 of them
 we got
 0-storage-replicate-3: failed to open as split brain seen, returning EIO
 but from 2 of them we can access the recovered file.

 We're this information locally storage on clients so we can update all
 of them and recover the information in a coherent manner?

 Thanks a lot in advance,
 Samuel.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split brain recovery 3.3.0

[Patric Uebele]

Hi Samuel,

can you try to drop caches on the clients:

“echo 3  /proc/sys/vm/drop_caches”

/Patric

On Fri, 2013-03-08 at 10:13 +0100, samuel wrote:
 No one around can help with this situation?
 
 The file finally got corrupted and it's impossible to read from any
 of the current mounted native gluster clients.
 
 Reading the attributes, all flags are set as 0:
 
 # file: dd29900f215b175939816f94c907a31b
 trusted.afr.storage-client-6=0x
 trusted.afr.storage-client-7=0x
 trusted.gfid=0x59311906edf04464be5f00f505b3aebb
 trusted.storage-stripe-1.stripe-count=0x3200
 trusted.storage-stripe-1.stripe-index=0x3100
 trusted.storage-stripe-1.stripe-size=0x31333130373200
 
 so there should not be read as split brain from clients but we got the
 following logs:
 split brain detected during lookup of
 
 thanks in advance,
 Samuel.
 
 
 On 4 March 2013 14:37, samuel sam...@gmail.com wrote:
 Hi folks,
 
 We have detected a split-brain on 3.3.0 stripped replicated 8
 nodes cluster.
 We've followed the instructions from:
 http://www.gluster.org/2012/07/fixing-split-brain-with-glusterfs-3-3/
 and we've manually recovered the information.
 
 The problem is that we've got 4 gluster native clients and
 from 2 of them we got 
 0-storage-replicate-3: failed to open as split brain seen,
 returning EIO
 but from 2 of them we can access the recovered file.
 
 We're this information locally storage on clients so we can
 update all of them and recover the information in a
 coherent manner?
 
 Thanks a lot in advance,
 Samuel.
 
 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 http://supercolony.gluster.org/mailman/listinfo/gluster-users

-- 
Patric Uebele 
Solution Architect Storage

Red Hat GmbH 
Technopark II, Haus C 
Werner-von-Siemens-Ring 14 
85630 Grasbrunn 
Germany 

Office:+49 89 205071-162 
Cell:  +49 172 669 14 99 
mailto:patric.ueb...@redhat.com 

gpg keyid: 48E64CC1
gpg fingerprint: C63E 6320 A03B 4410 D208  4EE7 12FC D0E6 48E6 4CC1

 
Reg. Adresse: Red Hat GmbH, Werner-von-Siemens-Ring 14, 85630 Grasbrunn 
Handelsregister: Amtsgericht Muenchen HRB 153243 
Geschaeftsfuehrer: Mark Hegarty, Charlie Peters, Michael Cunningham,
Charles Cachera


smime.p7s
Description: S/MIME cryptographic signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] split brain recovery 3.3.0

[samuel]

Dear Patric,

That seemed to make the trick because we can now see the file and access it.

Is there any place where it could be documented?

Thanks a lot,
Samuel

On 8 March 2013 10:40, Patric Uebele pueb...@redhat.com wrote:

 Hi Samuel,

 can you try to drop caches on the clients:

 “echo 3  /proc/sys/vm/drop_caches”

 /Patric

 On Fri, 2013-03-08 at 10:13 +0100, samuel wrote:
  No one around can help with this situation?
 
  The file finally got corrupted and it's impossible to read from any
  of the current mounted native gluster clients.
 
  Reading the attributes, all flags are set as 0:
 
  # file: dd29900f215b175939816f94c907a31b
  trusted.afr.storage-client-6=0x
  trusted.afr.storage-client-7=0x
  trusted.gfid=0x59311906edf04464be5f00f505b3aebb
  trusted.storage-stripe-1.stripe-count=0x3200
  trusted.storage-stripe-1.stripe-index=0x3100
  trusted.storage-stripe-1.stripe-size=0x31333130373200
 
  so there should not be read as split brain from clients but we got the
  following logs:
  split brain detected during lookup of
 
  thanks in advance,
  Samuel.
 
 
  On 4 March 2013 14:37, samuel sam...@gmail.com wrote:
  Hi folks,
 
  We have detected a split-brain on 3.3.0 stripped replicated 8
  nodes cluster.
  We've followed the instructions from:
 
 http://www.gluster.org/2012/07/fixing-split-brain-with-glusterfs-3-3/
  and we've manually recovered the information.
 
  The problem is that we've got 4 gluster native clients and
  from 2 of them we got
  0-storage-replicate-3: failed to open as split brain seen,
  returning EIO
  but from 2 of them we can access the recovered file.
 
  We're this information locally storage on clients so we can
  update all of them and recover the information in a
  coherent manner?
 
  Thanks a lot in advance,
  Samuel.
 
  ___
  Gluster-users mailing list
  Gluster-users@gluster.org
  http://supercolony.gluster.org/mailman/listinfo/gluster-users

 --
 Patric Uebele
 Solution Architect Storage

 Red Hat GmbH
 Technopark II, Haus C
 Werner-von-Siemens-Ring 14
 85630 Grasbrunn
 Germany

 Office:+49 89 205071-162
 Cell:  +49 172 669 14 99
 mailto:patric.ueb...@redhat.com

 gpg keyid: 48E64CC1
 gpg fingerprint: C63E 6320 A03B 4410 D208  4EE7 12FC D0E6 48E6 4CC1

 
 Reg. Adresse: Red Hat GmbH, Werner-von-Siemens-Ring 14, 85630 Grasbrunn
 Handelsregister: Amtsgericht Muenchen HRB 153243
 Geschaeftsfuehrer: Mark Hegarty, Charlie Peters, Michael Cunningham,
 Charles Cachera

___
Gluster-users mailing list
Gluster-users@gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain; which file to choose for repair?

[Anand Avati]

Occurance of a split brain situation is only under a specific sequence of
events and modifications and the filesystem cannot decide which of the two
copies of the file is updated. It might so happen that the two changes were
actually the same change and hence the two copies of your file might match
md5sum (in which case you can delete one arbitrarily). If not, you need to
know how your application works and which of the file (inspecting the
content) is more appropriate to be deleted.

Avati

On Wed, May 4, 2011 at 5:54 PM, Martin Schenker 
martin.schen...@profitbricks.com wrote:

 Hi all!

 Is there anybody who can give some pointers regarding which file to choose
 in a split brain condition?

 What tests do I need to run?

 What does the hex AFR code actually show? Is there a way to pinpoint the
 better/worse file for deletion?

 On pserver12:

 # file: mnt/gluster/brick0/storage/pserver3-19
 trusted.afr.storage0-client-5=0x3f01

 On pserver13:

 # file: mnt/gluster/brick0/storage/pserver3-19
 trusted.afr.storage0-client-4=0xd701

 These are test files, but I'd like to know what to do in a LIFE situation
 which will be just around the corner.

 The Timestamps show the same values, so I'm a bit puzzled HOW to choose a
 file.

 pserver12:

 0 root@de-dc1-c1-pserver12:~ # ls -al
 /mnt/gluster/brick0/storage/pserver3-19
 -rw-r--r-- 1 vcb root 3456106496 Apr 29 17:40
 /mnt/gluster/brick0/storage/pserver3-19

 0 root@de-dc1-c1-pserver12:~ # ls -alu
 /mnt/gluster/brick0/storage/pserver3-19
 -rw-r--r-- 1 vcb root 3456106496 Apr 28 16:18
 /mnt/gluster/brick0/storage/pserver3-19

 pserver13:

 0 root@de-dc1-c1-pserver13:~ # ls -al
 /mnt/gluster/brick0/storage/pserver3-19
 -rw-r--r-- 1 vcb root 3456106496 Apr 29 17:40
 /mnt/gluster/brick0/storage/pserver3-19

 0 root@de-dc1-c1-pserver13:~ # ls -alu
 /mnt/gluster/brick0/storage/pserver3-19
 -rw-r--r-- 1 vcb root 3456106496 Apr 28 16:18
 /mnt/gluster/brick0/storage/pserver3-19

 Best, Martin

 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain; which file to choose for repair?

[Martin Schenker]

Thanks!

 

Unfortunately the md5sums don't match...  but the sizes  timestamps do. 

These binary files from VMs are quite difficult to work with. That's why I
was after some ideas WHICH file should be preferred.

 

I'm actually quite concerned how on earth do we trigger split brain
conditions on a regular basis. 

 

Are there any good DON'Ts for server updates/maintenance? The current way
is shutting down one server , upgrading/installing it, bringing up the
Gluster daemons (server  client)/mounting the file system, checking
everything is running. Then starting on the next server. 

So a pure sequential approach. Thought this would minimise the risk of
getting the files in a twist?!?

 

Best, Martin

 

From: Anand Avati [mailto:anand.av...@gmail.com] 
Sent: Wednesday, May 04, 2011 2:55 PM
To: Martin Schenker
Cc: gluster-users@gluster.org
Subject: Re: [Gluster-users] Split brain; which file to choose for repair?

 

Occurance of a split brain situation is only under a specific sequence of
events and modifications and the filesystem cannot decide which of the two
copies of the file is updated. It might so happen that the two changes were
actually the same change and hence the two copies of your file might match
md5sum (in which case you can delete one arbitrarily). If not, you need to
know how your application works and which of the file (inspecting the
content) is more appropriate to be deleted.

 

Avati

On Wed, May 4, 2011 at 5:54 PM, Martin Schenker
martin.schen...@profitbricks.com wrote:

Hi all!

Is there anybody who can give some pointers regarding which file to choose
in a split brain condition?

What tests do I need to run?

What does the hex AFR code actually show? Is there a way to pinpoint the
better/worse file for deletion?

On pserver12:


# file: mnt/gluster/brick0/storage/pserver3-19
trusted.afr.storage0-client-5=0x3f01

On pserver13:


# file: mnt/gluster/brick0/storage/pserver3-19
trusted.afr.storage0-client-4=0xd701

These are test files, but I'd like to know what to do in a LIFE situation
which will be just around the corner.

The Timestamps show the same values, so I'm a bit puzzled HOW to choose a
file.

pserver12:

0 root@de-dc1-c1-pserver12:~ # ls -al
/mnt/gluster/brick0/storage/pserver3-19
-rw-r--r-- 1 vcb root 3456106496 Apr 29 17:40
/mnt/gluster/brick0/storage/pserver3-19

0 root@de-dc1-c1-pserver12:~ # ls -alu
/mnt/gluster/brick0/storage/pserver3-19
-rw-r--r-- 1 vcb root 3456106496 Apr 28 16:18
/mnt/gluster/brick0/storage/pserver3-19

pserver13:

0 root@de-dc1-c1-pserver13:~ # ls -al
/mnt/gluster/brick0/storage/pserver3-19
-rw-r--r-- 1 vcb root 3456106496 Apr 29 17:40
/mnt/gluster/brick0/storage/pserver3-19

0 root@de-dc1-c1-pserver13:~ # ls -alu
/mnt/gluster/brick0/storage/pserver3-19
-rw-r--r-- 1 vcb root 3456106496 Apr 28 16:18
/mnt/gluster/brick0/storage/pserver3-19

Best, Martin


___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

 

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain; which file to choose for repair?

[Joe Landman]

On 05/04/2011 08:24 AM, Martin Schenker wrote:

Hi all!

Is there anybody who can give some pointers regarding which file to choose
in a split brain condition?

What tests do I need to run?


MD5sums.  Did the logs indicate a split brain?  Or are the signatures 
simply different?




What does the hex AFR code actually show? Is there a way to pinpoint the
better/worse file for deletion?

On pserver12:

# file: mnt/gluster/brick0/storage/pserver3-19
trusted.afr.storage0-client-5=0x3f01

On pserver13:

# file: mnt/gluster/brick0/storage/pserver3-19
trusted.afr.storage0-client-4=0xd701

These are test files, but I'd like to know what to do in a LIFE situation
which will be just around the corner.

The Timestamps show the same values, so I'm a bit puzzled HOW to choose a
file.


File sizes and time stamps the same?

Hmmm ... this sounds like an underlying caching issue (probably not 
flushed completely/properly on one or more of the units before reboot) 
with the base machine.  Check the battery backup  on the RAID and make 
sure it is functional.


Also, run an file system check on the underlying backend storage.

--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: land...@scalableinformatics.com
web  : http://scalableinformatics.com
   http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain; which file to choose for repair?

[Anand Babu Periasamy]

On Wed, May 4, 2011 at 7:29 PM, Joe Landman land...@scalableinformatics.com
 wrote:

 On 05/04/2011 08:24 AM, Martin Schenker wrote:

 Hi all!

 Is there anybody who can give some pointers regarding which file to choose
 in a split brain condition?

 What tests do I need to run?


 MD5sums.  Did the logs indicate a split brain?  Or are the signatures
 simply different?



 What does the hex AFR code actually show? Is there a way to pinpoint the
 better/worse file for deletion?

 On pserver12:

 # file: mnt/gluster/brick0/storage/pserver3-19
 trusted.afr.storage0-client-5=0x3f01

 On pserver13:

 # file: mnt/gluster/brick0/storage/pserver3-19
 trusted.afr.storage0-client-4=0xd701

 These are test files, but I'd like to know what to do in a LIFE situation
 which will be just around the corner.

 The Timestamps show the same values, so I'm a bit puzzled HOW to choose a
 file.


 File sizes and time stamps the same?

 Hmmm ... this sounds like an underlying caching issue (probably not flushed
 completely/properly on one or more of the units before reboot) with the base
 machine.  Check the battery backup  on the RAID and make sure it is
 functional.

 Also, run an file system check on the underlying backend storage.

 --
 Joseph Landman, Ph.D
 Founder and CEO
 Scalable Informatics, Inc.
 email: land...@scalableinformatics.com
 web  : http://scalableinformatics.com
   http://scalableinformatics.com/sicluster
 phone: +1 734 786 8423 x121
 fax  : +1 866 888 3112
 cell : +1 734 612 4615

 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 http://gluster.org/cgi-bin/mailman/listinfo/gluster-users



Yes I agree with Joe. I will check the underlying disk filesystem first as
first step. Look at kernel logs and dmesg for file system errors. Even if
you did not find any, try running a forceful fsck on them. Another possible
cause is silent data corruption. If everything is fine, then it can likely
be a GlusterFS bug

-- 
Anand Babu Periasamy
Blog [http://www.unlocksmith.org]

Imagination is more important than knowledge --Albert Einstein
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain errors

[Mohit Anchlia]

Can someone from dev please help reply? Should I open a bug?

On Thu, Apr 28, 2011 at 2:17 PM, Mohit Anchlia mohitanch...@gmail.com wrote:
 I got some help and fixed these by setting xattr. For eg changed I
 to A using setfattr.

 But now my next question is why did this happen at first place and
 what measure needs to be taken so that this doesn't happen? It keeps
 happening even if I start clean, stop vol, delete vol, delete
 contents, re-create vols. Also, some of the bricks don't have
 stress-volume attr.



 getfattr -dm - /data1/gluster
 getfattr: Removing leading '/' from absolute path names
 # file: data1/gluster
 trusted.afr.stress-volume-client-8=0sAAIA
 trusted.afr.stress-volume-client-9=0s
 trusted.gfid=0sAQ==
 trusted.glusterfs.dht=0sAQAqPg==
 trusted.glusterfs.test=working\000


 On Thu, Apr 28, 2011 at 9:24 AM, Mohit Anchlia mohitanch...@gmail.com wrote:
 I create 30K directories in the client mountpoint. But I've done this
 test with mkfs -I 256 and with default 128 byte (Red hat 5.6). Only
 when I create mkfs -I 256 I see these errors. Looks like the reason
 for the failure because otherwise everything else is same. Same no of
 bricks, servers, user (root) etc.

 I run the stress test and client mount logs are full with these errors
 for every subvolume. Looks like it's happening for every file that's
 being writen

 On Thu, Apr 28, 2011 at 9:20 AM, Amar Tumballi a...@gluster.com wrote:
 I am seeing the directory size to be different here. Let me confirm if we
 are checking extra for size to be same also (for directories it will not be
 needed). In that case, this log makes sense, but surely that is a false
 positive.
 -Amar

 On Thu, Apr 28, 2011 at 9:44 PM, Mohit Anchlia mohitanch...@gmail.com
 wrote:

 Yes they are the same. It looks like this problem appears only when I
 use -I 256 when creating mkfs. Why would that be?

 [root@dsdb1 ~]# ls -ltr /data/
 total 5128
 drwx--     2 root root   16384 Apr 27 16:57 lost+found
 drwxr-xr-x 30003 root root 4562944 Apr 27 17:15 mnt-stress
 drwxr-xr-x 30003 root root  598016 Apr 27 17:15 gluster
 [root@dsdb1 ~]# ls -ltr /data1/
 total 572
 drwx--     2 root root  16384 Apr 27 16:59 lost+found
 drwxr-xr-x 30003 root root 561152 Apr 27 17:15 gluster

 [root@dsdb2 ~]# ls -ltr /data
 total 588
 drwx--     2 root root  16384 Apr 27 16:52 lost+found
 drwxr-xr-x     2 root root   4096 Apr 27 17:09 mnt-stress
 drwxr-xr-x 30003 root root 573440 Apr 27 17:15 gluster
 [root@dsdb2 ~]# ls -ltr /data1
 total 592
 drwx--     2 root root  16384 Apr 27 16:54 lost+found
 drwxr-xr-x 30003 root root 581632 Apr 27 17:15 gluster


 On Wed, Apr 27, 2011 at 11:18 PM, Amar Tumballi a...@gluster.com wrote:
 
  [2011-04-27 17:11:29.13142] E
  [afr-self-heal-metadata.c:524:afr_sh_metadata_fix]
  0-stress-volume-replicate-0: Unable to self-heal permissions/ownership
  of '/' (possible split-brain). Please fix the file on all backend
  volumes
 
  Can someone please help me reason for this problem?
 
   gluster volume info all
 
  Volume Name: stress-volume
  Type: Distributed-Replicate
  Status: Started
  Number of Bricks: 8 x 2 = 16
  Transport-type: tcp
  Bricks:
  Brick1: dsdb1:/data/gluster
  Brick2: dsdb2:/data/gluster
 
  Did you check the permission/ownership of these exports? Please make
  sure
  that they are same.
  Regards,
  Amar
 
 
  Brick3: dsdb3:/data/gluster
  Brick4: dsdb4:/data/gluster
  Brick5: dsdb5:/data/gluster
  Brick6: dsdb6:/data/gluster
  Brick7: dslg1:/data/gluster
  Brick8: dslg2:/data/gluster
  Brick9: dsdb1:/data1/gluster
  Brick10: dsdb2:/data1/gluster
  Brick11: dsdb3:/data1/gluster
  Brick12: dsdb4:/data1/gluster
  Brick13: dsdb5:/data1/gluster
  Brick14: dsdb6:/data1/gluster
  Brick15: dslg1:/data1/gluster
  Brick16: dslg2:/data1/gluster
 
 
 




___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain errors

[Pranith Kumar. Karampuri]

hi Mohit,
 Do you know what exact steps are leading to this problem?.

Pranith.

- Original Message -
From: Mohit Anchlia mohitanch...@gmail.com
To: Amar Tumballi a...@gluster.com, gluster-users@gluster.org
Sent: Friday, April 29, 2011 9:49:33 PM
Subject: Re: [Gluster-users] Split brain errors

Can someone from dev please help reply? Should I open a bug?

On Thu, Apr 28, 2011 at 2:17 PM, Mohit Anchlia mohitanch...@gmail.com wrote:
 I got some help and fixed these by setting xattr. For eg changed I
 to A using setfattr.

 But now my next question is why did this happen at first place and
 what measure needs to be taken so that this doesn't happen? It keeps
 happening even if I start clean, stop vol, delete vol, delete
 contents, re-create vols. Also, some of the bricks don't have
 stress-volume attr.



 getfattr -dm - /data1/gluster
 getfattr: Removing leading '/' from absolute path names
 # file: data1/gluster
 trusted.afr.stress-volume-client-8=0sAAIA
 trusted.afr.stress-volume-client-9=0s
 trusted.gfid=0sAQ==
 trusted.glusterfs.dht=0sAQAqPg==
 trusted.glusterfs.test=working\000


 On Thu, Apr 28, 2011 at 9:24 AM, Mohit Anchlia mohitanch...@gmail.com wrote:
 I create 30K directories in the client mountpoint. But I've done this
 test with mkfs -I 256 and with default 128 byte (Red hat 5.6). Only
 when I create mkfs -I 256 I see these errors. Looks like the reason
 for the failure because otherwise everything else is same. Same no of
 bricks, servers, user (root) etc.

 I run the stress test and client mount logs are full with these errors
 for every subvolume. Looks like it's happening for every file that's
 being writen

 On Thu, Apr 28, 2011 at 9:20 AM, Amar Tumballi a...@gluster.com wrote:
 I am seeing the directory size to be different here. Let me confirm if we
 are checking extra for size to be same also (for directories it will not be
 needed). In that case, this log makes sense, but surely that is a false
 positive.
 -Amar

 On Thu, Apr 28, 2011 at 9:44 PM, Mohit Anchlia mohitanch...@gmail.com
 wrote:

 Yes they are the same. It looks like this problem appears only when I
 use -I 256 when creating mkfs. Why would that be?

 [root@dsdb1 ~]# ls -ltr /data/
 total 5128
 drwx--     2 root root   16384 Apr 27 16:57 lost+found
 drwxr-xr-x 30003 root root 4562944 Apr 27 17:15 mnt-stress
 drwxr-xr-x 30003 root root  598016 Apr 27 17:15 gluster
 [root@dsdb1 ~]# ls -ltr /data1/
 total 572
 drwx--     2 root root  16384 Apr 27 16:59 lost+found
 drwxr-xr-x 30003 root root 561152 Apr 27 17:15 gluster

 [root@dsdb2 ~]# ls -ltr /data
 total 588
 drwx--     2 root root  16384 Apr 27 16:52 lost+found
 drwxr-xr-x     2 root root   4096 Apr 27 17:09 mnt-stress
 drwxr-xr-x 30003 root root 573440 Apr 27 17:15 gluster
 [root@dsdb2 ~]# ls -ltr /data1
 total 592
 drwx--     2 root root  16384 Apr 27 16:54 lost+found
 drwxr-xr-x 30003 root root 581632 Apr 27 17:15 gluster


 On Wed, Apr 27, 2011 at 11:18 PM, Amar Tumballi a...@gluster.com wrote:
 
  [2011-04-27 17:11:29.13142] E
  [afr-self-heal-metadata.c:524:afr_sh_metadata_fix]
  0-stress-volume-replicate-0: Unable to self-heal permissions/ownership
  of '/' (possible split-brain). Please fix the file on all backend
  volumes
 
  Can someone please help me reason for this problem?
 
   gluster volume info all
 
  Volume Name: stress-volume
  Type: Distributed-Replicate
  Status: Started
  Number of Bricks: 8 x 2 = 16
  Transport-type: tcp
  Bricks:
  Brick1: dsdb1:/data/gluster
  Brick2: dsdb2:/data/gluster
 
  Did you check the permission/ownership of these exports? Please make
  sure
  that they are same.
  Regards,
  Amar
 
 
  Brick3: dsdb3:/data/gluster
  Brick4: dsdb4:/data/gluster
  Brick5: dsdb5:/data/gluster
  Brick6: dsdb6:/data/gluster
  Brick7: dslg1:/data/gluster
  Brick8: dslg2:/data/gluster
  Brick9: dsdb1:/data1/gluster
  Brick10: dsdb2:/data1/gluster
  Brick11: dsdb3:/data1/gluster
  Brick12: dsdb4:/data1/gluster
  Brick13: dsdb5:/data1/gluster
  Brick14: dsdb6:/data1/gluster
  Brick15: dslg1:/data1/gluster
  Brick16: dslg2:/data1/gluster
 
 
 




___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain errors

[Mohit Anchlia]

All I do is
stop volume,
delete volume,
remove mount dir,
create mount dir
create volume and this happens.

I have also tried stopping glusterd after deleting volume and then
start before creating volume again. But this consistently happens.

Also, when I add multiple bricks on the same server some of the
servers don't have all the xattr.
On Fri, Apr 29, 2011 at 10:25 AM, Pranith Kumar. Karampuri
prani...@gluster.com wrote:
 hi Mohit,
         Do you know what exact steps are leading to this problem?.

 Pranith.

 - Original Message -
 From: Mohit Anchlia mohitanch...@gmail.com
 To: Amar Tumballi a...@gluster.com, gluster-users@gluster.org
 Sent: Friday, April 29, 2011 9:49:33 PM
 Subject: Re: [Gluster-users] Split brain errors

 Can someone from dev please help reply? Should I open a bug?

 On Thu, Apr 28, 2011 at 2:17 PM, Mohit Anchlia mohitanch...@gmail.com wrote:
 I got some help and fixed these by setting xattr. For eg changed I
 to A using setfattr.

 But now my next question is why did this happen at first place and
 what measure needs to be taken so that this doesn't happen? It keeps
 happening even if I start clean, stop vol, delete vol, delete
 contents, re-create vols. Also, some of the bricks don't have
 stress-volume attr.



 getfattr -dm - /data1/gluster
 getfattr: Removing leading '/' from absolute path names
 # file: data1/gluster
 trusted.afr.stress-volume-client-8=0sAAIA
 trusted.afr.stress-volume-client-9=0s
 trusted.gfid=0sAQ==
 trusted.glusterfs.dht=0sAQAqPg==
 trusted.glusterfs.test=working\000


 On Thu, Apr 28, 2011 at 9:24 AM, Mohit Anchlia mohitanch...@gmail.com 
 wrote:
 I create 30K directories in the client mountpoint. But I've done this
 test with mkfs -I 256 and with default 128 byte (Red hat 5.6). Only
 when I create mkfs -I 256 I see these errors. Looks like the reason
 for the failure because otherwise everything else is same. Same no of
 bricks, servers, user (root) etc.

 I run the stress test and client mount logs are full with these errors
 for every subvolume. Looks like it's happening for every file that's
 being writen

 On Thu, Apr 28, 2011 at 9:20 AM, Amar Tumballi a...@gluster.com wrote:
 I am seeing the directory size to be different here. Let me confirm if we
 are checking extra for size to be same also (for directories it will not be
 needed). In that case, this log makes sense, but surely that is a false
 positive.
 -Amar

 On Thu, Apr 28, 2011 at 9:44 PM, Mohit Anchlia mohitanch...@gmail.com
 wrote:

 Yes they are the same. It looks like this problem appears only when I
 use -I 256 when creating mkfs. Why would that be?

 [root@dsdb1 ~]# ls -ltr /data/
 total 5128
 drwx--     2 root root   16384 Apr 27 16:57 lost+found
 drwxr-xr-x 30003 root root 4562944 Apr 27 17:15 mnt-stress
 drwxr-xr-x 30003 root root  598016 Apr 27 17:15 gluster
 [root@dsdb1 ~]# ls -ltr /data1/
 total 572
 drwx--     2 root root  16384 Apr 27 16:59 lost+found
 drwxr-xr-x 30003 root root 561152 Apr 27 17:15 gluster

 [root@dsdb2 ~]# ls -ltr /data
 total 588
 drwx--     2 root root  16384 Apr 27 16:52 lost+found
 drwxr-xr-x     2 root root   4096 Apr 27 17:09 mnt-stress
 drwxr-xr-x 30003 root root 573440 Apr 27 17:15 gluster
 [root@dsdb2 ~]# ls -ltr /data1
 total 592
 drwx--     2 root root  16384 Apr 27 16:54 lost+found
 drwxr-xr-x 30003 root root 581632 Apr 27 17:15 gluster


 On Wed, Apr 27, 2011 at 11:18 PM, Amar Tumballi a...@gluster.com wrote:
 
  [2011-04-27 17:11:29.13142] E
  [afr-self-heal-metadata.c:524:afr_sh_metadata_fix]
  0-stress-volume-replicate-0: Unable to self-heal permissions/ownership
  of '/' (possible split-brain). Please fix the file on all backend
  volumes
 
  Can someone please help me reason for this problem?
 
   gluster volume info all
 
  Volume Name: stress-volume
  Type: Distributed-Replicate
  Status: Started
  Number of Bricks: 8 x 2 = 16
  Transport-type: tcp
  Bricks:
  Brick1: dsdb1:/data/gluster
  Brick2: dsdb2:/data/gluster
 
  Did you check the permission/ownership of these exports? Please make
  sure
  that they are same.
  Regards,
  Amar
 
 
  Brick3: dsdb3:/data/gluster
  Brick4: dsdb4:/data/gluster
  Brick5: dsdb5:/data/gluster
  Brick6: dsdb6:/data/gluster
  Brick7: dslg1:/data/gluster
  Brick8: dslg2:/data/gluster
  Brick9: dsdb1:/data1/gluster
  Brick10: dsdb2:/data1/gluster
  Brick11: dsdb3:/data1/gluster
  Brick12: dsdb4:/data1/gluster
  Brick13: dsdb5:/data1/gluster
  Brick14: dsdb6:/data1/gluster
  Brick15: dslg1:/data1/gluster
  Brick16: dslg2:/data1/gluster
 
 
 




 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain errors

[Pranith Kumar. Karampuri]

Please check if outputs of getfattr -d -m trusted* on all the brick 
directories differ.

Pranith.
- Original Message -
From: Mohit Anchlia mohitanch...@gmail.com
To: Pranith Kumar. Karampuri prani...@gluster.com
Cc: Amar Tumballi a...@gluster.com, gluster-users@gluster.org
Sent: Friday, April 29, 2011 11:37:01 PM
Subject: Re: [Gluster-users] Split brain errors

All I do is
stop volume,
delete volume,
remove mount dir,
create mount dir
create volume and this happens.

I have also tried stopping glusterd after deleting volume and then
start before creating volume again. But this consistently happens.

Also, when I add multiple bricks on the same server some of the
servers don't have all the xattr.
On Fri, Apr 29, 2011 at 10:25 AM, Pranith Kumar. Karampuri
prani...@gluster.com wrote:
 hi Mohit,
         Do you know what exact steps are leading to this problem?.

 Pranith.

 - Original Message -
 From: Mohit Anchlia mohitanch...@gmail.com
 To: Amar Tumballi a...@gluster.com, gluster-users@gluster.org
 Sent: Friday, April 29, 2011 9:49:33 PM
 Subject: Re: [Gluster-users] Split brain errors

 Can someone from dev please help reply? Should I open a bug?

 On Thu, Apr 28, 2011 at 2:17 PM, Mohit Anchlia mohitanch...@gmail.com wrote:
 I got some help and fixed these by setting xattr. For eg changed I
 to A using setfattr.

 But now my next question is why did this happen at first place and
 what measure needs to be taken so that this doesn't happen? It keeps
 happening even if I start clean, stop vol, delete vol, delete
 contents, re-create vols. Also, some of the bricks don't have
 stress-volume attr.



 getfattr -dm - /data1/gluster
 getfattr: Removing leading '/' from absolute path names
 # file: data1/gluster
 trusted.afr.stress-volume-client-8=0sAAIA
 trusted.afr.stress-volume-client-9=0s
 trusted.gfid=0sAQ==
 trusted.glusterfs.dht=0sAQAqPg==
 trusted.glusterfs.test=working\000


 On Thu, Apr 28, 2011 at 9:24 AM, Mohit Anchlia mohitanch...@gmail.com 
 wrote:
 I create 30K directories in the client mountpoint. But I've done this
 test with mkfs -I 256 and with default 128 byte (Red hat 5.6). Only
 when I create mkfs -I 256 I see these errors. Looks like the reason
 for the failure because otherwise everything else is same. Same no of
 bricks, servers, user (root) etc.

 I run the stress test and client mount logs are full with these errors
 for every subvolume. Looks like it's happening for every file that's
 being writen

 On Thu, Apr 28, 2011 at 9:20 AM, Amar Tumballi a...@gluster.com wrote:
 I am seeing the directory size to be different here. Let me confirm if we
 are checking extra for size to be same also (for directories it will not be
 needed). In that case, this log makes sense, but surely that is a false
 positive.
 -Amar

 On Thu, Apr 28, 2011 at 9:44 PM, Mohit Anchlia mohitanch...@gmail.com
 wrote:

 Yes they are the same. It looks like this problem appears only when I
 use -I 256 when creating mkfs. Why would that be?

 [root@dsdb1 ~]# ls -ltr /data/
 total 5128
 drwx--     2 root root   16384 Apr 27 16:57 lost+found
 drwxr-xr-x 30003 root root 4562944 Apr 27 17:15 mnt-stress
 drwxr-xr-x 30003 root root  598016 Apr 27 17:15 gluster
 [root@dsdb1 ~]# ls -ltr /data1/
 total 572
 drwx--     2 root root  16384 Apr 27 16:59 lost+found
 drwxr-xr-x 30003 root root 561152 Apr 27 17:15 gluster

 [root@dsdb2 ~]# ls -ltr /data
 total 588
 drwx--     2 root root  16384 Apr 27 16:52 lost+found
 drwxr-xr-x     2 root root   4096 Apr 27 17:09 mnt-stress
 drwxr-xr-x 30003 root root 573440 Apr 27 17:15 gluster
 [root@dsdb2 ~]# ls -ltr /data1
 total 592
 drwx--     2 root root  16384 Apr 27 16:54 lost+found
 drwxr-xr-x 30003 root root 581632 Apr 27 17:15 gluster


 On Wed, Apr 27, 2011 at 11:18 PM, Amar Tumballi a...@gluster.com wrote:
 
  [2011-04-27 17:11:29.13142] E
  [afr-self-heal-metadata.c:524:afr_sh_metadata_fix]
  0-stress-volume-replicate-0: Unable to self-heal permissions/ownership
  of '/' (possible split-brain). Please fix the file on all backend
  volumes
 
  Can someone please help me reason for this problem?
 
   gluster volume info all
 
  Volume Name: stress-volume
  Type: Distributed-Replicate
  Status: Started
  Number of Bricks: 8 x 2 = 16
  Transport-type: tcp
  Bricks:
  Brick1: dsdb1:/data/gluster
  Brick2: dsdb2:/data/gluster
 
  Did you check the permission/ownership of these exports? Please make
  sure
  that they are same.
  Regards,
  Amar
 
 
  Brick3: dsdb3:/data/gluster
  Brick4: dsdb4:/data/gluster
  Brick5: dsdb5:/data/gluster
  Brick6: dsdb6:/data/gluster
  Brick7: dslg1:/data/gluster
  Brick8: dslg2:/data/gluster
  Brick9: dsdb1:/data1/gluster
  Brick10: dsdb2:/data1/gluster
  Brick11: dsdb3:/data1/gluster
  Brick12: dsdb4:/data1/gluster
  Brick13: dsdb5:/data1/gluster
  Brick14: dsdb6:/data1/gluster
  Brick15: dslg1:/data1/gluster
  Brick16: dslg2:/data1/gluster

Re: [Gluster-users] Split brain errors

[Mohit Anchlia]

Yes I did check and that's how I fixed it by making it all hex A
using setfattr. That's one problem.

Also, I mentioned before there are some bricks missing xattr info. for eg:

[root@dsdb1 gluster]# getfattr -dm - /data2/gluster
getfattr: Removing leading '/' from absolute path names
# file: data2/gluster
trusted.afr.stress-volume-client-16=0s
trusted.afr.stress-volume-client-17=0s
trusted.gfid=0sAQ==
trusted.glusterfs.dht=0sAQB+lVVVUg==
trusted.glusterfs.test=working\000

[root@dsdb3 ~]# ls /data2/gluster/12657/372657
/data2/gluster/12657/372657
[root@dsdb3 ~]# getfattr -dm - /data2/gluster
getfattr: Removing leading '/' from absolute path names
# file: data2/gluster
trusted.gfid=0sAQ==
trusted.glusterfs.dht=0sAQCTpw==
trusted.glusterfs.test=working\000


[root@dsdb4 ~]# ls /data2/gluster/12657/372657
/data2/gluster/12657/372657
[root@dsdb4 ~]# getfattr -dm - /data2/gluster
getfattr: Removing leading '/' from absolute path names
# file: data2/gluster
trusted.gfid=0sAQ==
trusted.glusterfs.dht=0sAQCTpw==
trusted.glusterfs.test=working\000

On Fri, Apr 29, 2011 at 11:34 AM, Pranith Kumar. Karampuri
prani...@gluster.com wrote:
 Please check if outputs of getfattr -d -m trusted* on all the brick 
 directories differ.

 Pranith.
 - Original Message -
 From: Mohit Anchlia mohitanch...@gmail.com
 To: Pranith Kumar. Karampuri prani...@gluster.com
 Cc: Amar Tumballi a...@gluster.com, gluster-users@gluster.org
 Sent: Friday, April 29, 2011 11:37:01 PM
 Subject: Re: [Gluster-users] Split brain errors

 All I do is
 stop volume,
 delete volume,
 remove mount dir,
 create mount dir
 create volume and this happens.

 I have also tried stopping glusterd after deleting volume and then
 start before creating volume again. But this consistently happens.

 Also, when I add multiple bricks on the same server some of the
 servers don't have all the xattr.
 On Fri, Apr 29, 2011 at 10:25 AM, Pranith Kumar. Karampuri
 prani...@gluster.com wrote:
 hi Mohit,
         Do you know what exact steps are leading to this problem?.

 Pranith.

 - Original Message -
 From: Mohit Anchlia mohitanch...@gmail.com
 To: Amar Tumballi a...@gluster.com, gluster-users@gluster.org
 Sent: Friday, April 29, 2011 9:49:33 PM
 Subject: Re: [Gluster-users] Split brain errors

 Can someone from dev please help reply? Should I open a bug?

 On Thu, Apr 28, 2011 at 2:17 PM, Mohit Anchlia mohitanch...@gmail.com 
 wrote:
 I got some help and fixed these by setting xattr. For eg changed I
 to A using setfattr.

 But now my next question is why did this happen at first place and
 what measure needs to be taken so that this doesn't happen? It keeps
 happening even if I start clean, stop vol, delete vol, delete
 contents, re-create vols. Also, some of the bricks don't have
 stress-volume attr.



 getfattr -dm - /data1/gluster
 getfattr: Removing leading '/' from absolute path names
 # file: data1/gluster
 trusted.afr.stress-volume-client-8=0sAAIA
 trusted.afr.stress-volume-client-9=0s
 trusted.gfid=0sAQ==
 trusted.glusterfs.dht=0sAQAqPg==
 trusted.glusterfs.test=working\000


 On Thu, Apr 28, 2011 at 9:24 AM, Mohit Anchlia mohitanch...@gmail.com 
 wrote:
 I create 30K directories in the client mountpoint. But I've done this
 test with mkfs -I 256 and with default 128 byte (Red hat 5.6). Only
 when I create mkfs -I 256 I see these errors. Looks like the reason
 for the failure because otherwise everything else is same. Same no of
 bricks, servers, user (root) etc.

 I run the stress test and client mount logs are full with these errors
 for every subvolume. Looks like it's happening for every file that's
 being writen

 On Thu, Apr 28, 2011 at 9:20 AM, Amar Tumballi a...@gluster.com wrote:
 I am seeing the directory size to be different here. Let me confirm if we
 are checking extra for size to be same also (for directories it will not 
 be
 needed). In that case, this log makes sense, but surely that is a false
 positive.
 -Amar

 On Thu, Apr 28, 2011 at 9:44 PM, Mohit Anchlia mohitanch...@gmail.com
 wrote:

 Yes they are the same. It looks like this problem appears only when I
 use -I 256 when creating mkfs. Why would that be?

 [root@dsdb1 ~]# ls -ltr /data/
 total 5128
 drwx--     2 root root   16384 Apr 27 16:57 lost+found
 drwxr-xr-x 30003 root root 4562944 Apr 27 17:15 mnt-stress
 drwxr-xr-x 30003 root root  598016 Apr 27 17:15 gluster
 [root@dsdb1 ~]# ls -ltr /data1/
 total 572
 drwx--     2 root root  16384 Apr 27 16:59 lost+found
 drwxr-xr-x 30003 root root 561152 Apr 27 17:15 gluster

 [root@dsdb2 ~]# ls -ltr /data
 total 588
 drwx--     2 root root  16384 Apr 27 16:52 lost+found
 drwxr-xr-x     2 root root   4096 Apr 27 17:09 mnt-stress
 drwxr-xr-x 30003 root root 573440 Apr 27 17:15 gluster
 [root@dsdb2 ~]# ls -ltr /data1

Re: [Gluster-users] Split brain errors

[Amar Tumballi]

 [2011-04-27 17:11:29.13142] E
 [afr-self-heal-metadata.c:524:afr_sh_metadata_fix]
 0-stress-volume-replicate-0: Unable to self-heal permissions/ownership
 of '/' (possible split-brain). Please fix the file on all backend
 volumes

 Can someone please help me reason for this problem?

  gluster volume info all

 Volume Name: stress-volume
 Type: Distributed-Replicate
 Status: Started
 Number of Bricks: 8 x 2 = 16
 Transport-type: tcp
 Bricks:
 Brick1: dsdb1:/data/gluster
 Brick2: dsdb2:/data/gluster


Did you check the permission/ownership of these exports? Please make sure
that they are same.

Regards,
Amar


 Brick3: dsdb3:/data/gluster
 Brick4: dsdb4:/data/gluster
 Brick5: dsdb5:/data/gluster
 Brick6: dsdb6:/data/gluster
 Brick7: dslg1:/data/gluster
 Brick8: dslg2:/data/gluster
 Brick9: dsdb1:/data1/gluster
 Brick10: dsdb2:/data1/gluster
 Brick11: dsdb3:/data1/gluster
 Brick12: dsdb4:/data1/gluster
 Brick13: dsdb5:/data1/gluster
 Brick14: dsdb6:/data1/gluster
 Brick15: dslg1:/data1/gluster
 Brick16: dslg2:/data1/gluster

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain errors

[Mohit Anchlia]

Yes they are the same. It looks like this problem appears only when I
use -I 256 when creating mkfs. Why would that be?

[root@dsdb1 ~]# ls -ltr /data/
total 5128
drwx-- 2 root root   16384 Apr 27 16:57 lost+found
drwxr-xr-x 30003 root root 4562944 Apr 27 17:15 mnt-stress
drwxr-xr-x 30003 root root  598016 Apr 27 17:15 gluster
[root@dsdb1 ~]# ls -ltr /data1/
total 572
drwx-- 2 root root  16384 Apr 27 16:59 lost+found
drwxr-xr-x 30003 root root 561152 Apr 27 17:15 gluster

[root@dsdb2 ~]# ls -ltr /data
total 588
drwx-- 2 root root  16384 Apr 27 16:52 lost+found
drwxr-xr-x 2 root root   4096 Apr 27 17:09 mnt-stress
drwxr-xr-x 30003 root root 573440 Apr 27 17:15 gluster
[root@dsdb2 ~]# ls -ltr /data1
total 592
drwx-- 2 root root  16384 Apr 27 16:54 lost+found
drwxr-xr-x 30003 root root 581632 Apr 27 17:15 gluster


On Wed, Apr 27, 2011 at 11:18 PM, Amar Tumballi a...@gluster.com wrote:

 [2011-04-27 17:11:29.13142] E
 [afr-self-heal-metadata.c:524:afr_sh_metadata_fix]
 0-stress-volume-replicate-0: Unable to self-heal permissions/ownership
 of '/' (possible split-brain). Please fix the file on all backend
 volumes

 Can someone please help me reason for this problem?

  gluster volume info all

 Volume Name: stress-volume
 Type: Distributed-Replicate
 Status: Started
 Number of Bricks: 8 x 2 = 16
 Transport-type: tcp
 Bricks:
 Brick1: dsdb1:/data/gluster
 Brick2: dsdb2:/data/gluster

 Did you check the permission/ownership of these exports? Please make sure
 that they are same.
 Regards,
 Amar


 Brick3: dsdb3:/data/gluster
 Brick4: dsdb4:/data/gluster
 Brick5: dsdb5:/data/gluster
 Brick6: dsdb6:/data/gluster
 Brick7: dslg1:/data/gluster
 Brick8: dslg2:/data/gluster
 Brick9: dsdb1:/data1/gluster
 Brick10: dsdb2:/data1/gluster
 Brick11: dsdb3:/data1/gluster
 Brick12: dsdb4:/data1/gluster
 Brick13: dsdb5:/data1/gluster
 Brick14: dsdb6:/data1/gluster
 Brick15: dslg1:/data1/gluster
 Brick16: dslg2:/data1/gluster



___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

Re: [Gluster-users] Split brain errors

[Amar Tumballi]

I am seeing the directory size to be different here. Let me confirm if we
are checking extra for size to be same also (for directories it will not be
needed). In that case, this log makes sense, but surely that is a false
positive.

-Amar

On Thu, Apr 28, 2011 at 9:44 PM, Mohit Anchlia mohitanch...@gmail.comwrote:

 Yes they are the same. It looks like this problem appears only when I
 use -I 256 when creating mkfs. Why would that be?

 [root@dsdb1 ~]# ls -ltr /data/
 total 5128
 drwx-- 2 root root   16384 Apr 27 16:57 lost+found
 drwxr-xr-x 30003 root root 4562944 Apr 27 17:15 mnt-stress
 drwxr-xr-x 30003 root root  598016 Apr 27 17:15 gluster
 [root@dsdb1 ~]# ls -ltr /data1/
 total 572
 drwx-- 2 root root  16384 Apr 27 16:59 lost+found
 drwxr-xr-x 30003 root root 561152 Apr 27 17:15 gluster

 [root@dsdb2 ~]# ls -ltr /data
 total 588
 drwx-- 2 root root  16384 Apr 27 16:52 lost+found
 drwxr-xr-x 2 root root   4096 Apr 27 17:09 mnt-stress
 drwxr-xr-x 30003 root root 573440 Apr 27 17:15 gluster
 [root@dsdb2 ~]# ls -ltr /data1
 total 592
 drwx-- 2 root root  16384 Apr 27 16:54 lost+found
 drwxr-xr-x 30003 root root 581632 Apr 27 17:15 gluster


 On Wed, Apr 27, 2011 at 11:18 PM, Amar Tumballi a...@gluster.com wrote:
 
  [2011-04-27 17:11:29.13142] E
  [afr-self-heal-metadata.c:524:afr_sh_metadata_fix]
  0-stress-volume-replicate-0: Unable to self-heal permissions/ownership
  of '/' (possible split-brain). Please fix the file on all backend
  volumes
 
  Can someone please help me reason for this problem?
 
   gluster volume info all
 
  Volume Name: stress-volume
  Type: Distributed-Replicate
  Status: Started
  Number of Bricks: 8 x 2 = 16
  Transport-type: tcp
  Bricks:
  Brick1: dsdb1:/data/gluster
  Brick2: dsdb2:/data/gluster
 
  Did you check the permission/ownership of these exports? Please make sure
  that they are same.
  Regards,
  Amar
 
 
  Brick3: dsdb3:/data/gluster
  Brick4: dsdb4:/data/gluster
  Brick5: dsdb5:/data/gluster
  Brick6: dsdb6:/data/gluster
  Brick7: dslg1:/data/gluster
  Brick8: dslg2:/data/gluster
  Brick9: dsdb1:/data1/gluster
  Brick10: dsdb2:/data1/gluster
  Brick11: dsdb3:/data1/gluster
  Brick12: dsdb4:/data1/gluster
  Brick13: dsdb5:/data1/gluster
  Brick14: dsdb6:/data1/gluster
  Brick15: dslg1:/data1/gluster
  Brick16: dslg2:/data1/gluster
 
 
 

___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users