Re: [Gluster-devel] query about a split-brain problem found in glusterfs3.12.3

2018-02-10 Thread Zhou, Cynthia (NSB - CN/Hangzhou) 
Hi,
I check the link you provided. It does not mention the the "dirty" attribute, 
if I try to fix this split-brain by manually setfattr command, should I only 
set the "trusted.afr.export-client-0" command?
By the way, I feel it is quite strange that the output of "gluster volume heal 
export info" command there is two entries with the same name, how does this 
happen?
gluster volume heal export info
Brick sn-0.local:/mnt/bricks/export/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick
/testdir - Is in split-brain

/testdir - Possibly undergoing heal

Status: Connected
Number of entries: 2

I also do some other test, when sn-0 side file/dir does not has "dirty" and 
"trusted.afr.export-client-*" attribute and sn-1 side file/dir has both "dirty" 
and "trusted.afr.export-client-*" non-zero. The gluster could self heal such 
scenario. But in this case the it could never self heal.

From: Ravishankar N [mailto:ravishan...@redhat.com]
Sent: Thursday, February 08, 2018 11:56 AM
To: Zhou, Cynthia (NSB - CN/Hangzhou) ; 
Gluster-devel@gluster.org
Subject: Re: query about a split-brain problem found in glusterfs3.12.3




On 02/08/2018 07:16 AM, Zhou, Cynthia (NSB - CN/Hangzhou) wrote:
Hi,
Thanks for responding?
If split-brain happen in such kind of test is reasonable, how to fix this 
split-brain situation?

If you are using replica 2, then there is no prevention. Once they occur, you 
can resolve them using 
http://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/

If you want to prevent split-brain, you would need to use replica 3 or arbiter 
volume.

Regards,
Ravi

From: Ravishankar N [mailto:ravishan...@redhat.com]
Sent: Thursday, February 08, 2018 12:12 AM
To: Zhou, Cynthia (NSB - CN/Hangzhou) 
; 
Gluster-devel@gluster.org
Subject: Re: query about a split-brain problem found in glusterfs3.12.3




On 02/07/2018 10:39 AM, Zhou, Cynthia (NSB - CN/Hangzhou) wrote:

Hi glusterfs expert:
   Good day.
   Lately, we meet a glusterfs split brain problem in our env in 
/mnt/export/testdir. We start 3 ior process (IOR tool) from non-sn nodes, which 
is creating/removing files repeatedly in testdir. then we reboot sn nodes(sn0 
and sn1) by sequence. Then we meet following problem.
Do you have some comments on how this could happen? And how to fix it in 
this situation? Thanks!

Is the problem that split-brain is happening? Is this a replica 2 volume? If 
yes, then it looks like it is expected behavior?
Regards
Ravi




gluster volume heal export info
Brick sn-0.local:/mnt/bricks/export/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick
/testdir - Is in split-brain

/testdir - Possibly undergoing heal

Status: Connected
Number of entries: 2

wait for a while .

gluster volume heal export info
Brick sn-0.local:/mnt/bricks/export/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick
/testdir - Possibly undergoing heal

/testdir - Possibly undergoing heal

and finally:

[root@sn-0:/root]
# gluster v heal export info
Brick sn-0.local:/mnt/bricks/export/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick
/testdir - Is in split-brain

Status: Connected
Number of entries: 1



[root@sn-0:/root]

# getfattr -m .* -d -e  hex /mnt/bricks/export/brick/testdir

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/export/brick/testdir

trusted.gfid=0x5622cff893b3484dbdb6a20a0edb0e77

trusted.glusterfs.dht=0x0001



[root@sn-1:/root]

# getfattr -m .* -d -e  hex /mnt/bricks/export/brick/testdir

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/export/brick/testdir

trusted.afr.dirty=0x0001

trusted.afr.export-client-0=0x0038

trusted.gfid=0x5622cff893b3484dbdb6a20a0edb0e77

trusted.glusterfs.dht=0x0001




___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] query about a split-brain problem found in glusterfs3.12.3

2018-02-10 Thread Zhou, Cynthia (NSB - CN/Hangzhou) 
Hi,
Thanks for responding?
If split-brain happen in such kind of test is reasonable, how to fix this 
split-brain situation?

From: Ravishankar N [mailto:ravishan...@redhat.com]
Sent: Thursday, February 08, 2018 12:12 AM
To: Zhou, Cynthia (NSB - CN/Hangzhou) ; 
Gluster-devel@gluster.org
Subject: Re: query about a split-brain problem found in glusterfs3.12.3




On 02/07/2018 10:39 AM, Zhou, Cynthia (NSB - CN/Hangzhou) wrote:

Hi glusterfs expert:
   Good day.
   Lately, we meet a glusterfs split brain problem in our env in 
/mnt/export/testdir. We start 3 ior process (IOR tool) from non-sn nodes, which 
is creating/removing files repeatedly in testdir. then we reboot sn nodes(sn0 
and sn1) by sequence. Then we meet following problem.
Do you have some comments on how this could happen? And how to fix it in 
this situation? Thanks!

Is the problem that split-brain is happening? Is this a replica 2 volume? If 
yes, then it looks like it is expected behavior?
Regards
Ravi



gluster volume heal export info
Brick sn-0.local:/mnt/bricks/export/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick
/testdir - Is in split-brain

/testdir - Possibly undergoing heal

Status: Connected
Number of entries: 2

wait for a while .

gluster volume heal export info
Brick sn-0.local:/mnt/bricks/export/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick
/testdir - Possibly undergoing heal

/testdir - Possibly undergoing heal

and finally:

[root@sn-0:/root]
# gluster v heal export info
Brick sn-0.local:/mnt/bricks/export/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick
/testdir - Is in split-brain

Status: Connected
Number of entries: 1



[root@sn-0:/root]

# getfattr -m .* -d -e  hex /mnt/bricks/export/brick/testdir

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/export/brick/testdir

trusted.gfid=0x5622cff893b3484dbdb6a20a0edb0e77

trusted.glusterfs.dht=0x0001



[root@sn-1:/root]

# getfattr -m .* -d -e  hex /mnt/bricks/export/brick/testdir

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/export/brick/testdir

trusted.afr.dirty=0x0001

trusted.afr.export-client-0=0x0038

trusted.gfid=0x5622cff893b3484dbdb6a20a0edb0e77

trusted.glusterfs.dht=0x0001



___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] query about a split-brain problem found in glusterfs3.12.3

2018-02-08 Thread Ravishankar N 



On 02/08/2018 01:08 PM, Zhou, Cynthia (NSB - CN/Hangzhou) wrote:


Hi,

I check the link you provided. It does not mention the the “dirty” 
attribute, if I try to fix this split-brain by manually setfattr 
command, should I only set the “trusted.afr.export-client-0” command?


Manually resetting xattrs is not recommended. Use the gluster CLI to 
resolve it.


By the way, I feel it is quite strange that the output of “gluster 
volume heal export info” command there is two entries with the same 
name, how does this happen?


Maybe the same entry is listed in different subfolders of 
.glusterfs/indices?


gluster volume heal export info
Brick sn-0.local:/mnt/bricks/export/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick
/testdir - Is in split-brain

/testdir - Possibly undergoing heal

Status: Connected
Number of entries: 2

I also do some other test, when sn-0 side file/dir does not has 
“dirty” and “trusted.afr.export-client-*” attribute and sn-1 side 
file/dir has both “dirty” and “trusted.afr.export-client-*” non-zero. 
The gluster could self heal such scenario. But in this case the it 
could never self heal.


*From:*Ravishankar N [mailto:ravishan...@redhat.com]
*Sent:* Thursday, February 08, 2018 11:56 AM
*To:* Zhou, Cynthia (NSB - CN/Hangzhou) 
; Gluster-devel@gluster.org

*Subject:* Re: query about a split-brain problem found in glusterfs3.12.3

On 02/08/2018 07:16 AM, Zhou, Cynthia (NSB - CN/Hangzhou) wrote:

Hi,

Thanks for responding?

If split-brain happen in such kind of test is reasonable, how to
fix this split-brain situation?

If you are using replica 2, then there is no prevention. Once they 
occur, you can resolve them using 
http://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/


If you want to prevent split-brain, you would need to use replica 3 or 
arbiter volume.


Regards,
Ravi

*From:*Ravishankar N [mailto:ravishan...@redhat.com]
*Sent:* Thursday, February 08, 2018 12:12 AM
*To:* Zhou, Cynthia (NSB - CN/Hangzhou)

; Gluster-devel@gluster.org

*Subject:* Re: query about a split-brain problem found in
glusterfs3.12.3

On 02/07/2018 10:39 AM, Zhou, Cynthia (NSB - CN/Hangzhou) wrote:

Hi glusterfs expert:

Good day.

Lately, we meet a glusterfs split brain problem in our env in
/mnt/export/testdir. We start 3 ior process (IOR tool) from
non-sn nodes, which is creating/removing files repeatedly in
testdir. then we reboot sn nodes(sn0 and sn1) by sequence.
Then we meet following problem.

Do you have some comments on how this could happen? And how to
fix it in this situation? Thanks!


Is the problem that split-brain is happening? Is this a replica 2
volume? If yes, then it looks like it is expected behavior?
Regards
Ravi


gluster volume heal export info
Brick sn-0.local:/mnt/bricks/export/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick
/testdir - Is in split-brain

/testdir - Possibly undergoing heal

Status: Connected
Number of entries: 2

wait for a while …..

gluster volume heal export info
Brick sn-0.local:/mnt/bricks/export/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick
/testdir - Possibly undergoing heal

/testdir - Possibly undergoing heal

and finally:

[root@sn-0:/root ]
# gluster v heal export info
Brick sn-0.local:/mnt/bricks/export/brick

Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick

/testdir - Is in split-brain

Status: Connected
Number of entries: 1

[root@sn-0:/root ]

# getfattr -m .* -d -e  hex /mnt/bricks/export/brick/testdir

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/export/brick/testdir

trusted.gfid=0x5622cff893b3484dbdb6a20a0edb0e77

trusted.glusterfs.dht=0x0001

[root@sn-1:/root ]

# getfattr -m .* -d -e  hex /mnt/bricks/export/brick/testdir

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/export/brick/testdir

trusted.afr.dirty=0x0001

trusted.afr.export-client-0=0x0038

trusted.gfid=0x5622cff893b3484dbdb6a20a0edb0e77

trusted.glusterfs.dht=0x0001



___

Re: [Gluster-devel] query about a split-brain problem found in glusterfs3.12.3

2018-02-07 Thread Ravishankar N 



On 02/08/2018 07:16 AM, Zhou, Cynthia (NSB - CN/Hangzhou) wrote:


Hi,

Thanks for responding?

If split-brain happen in such kind of test is reasonable, how to fix 
this split-brain situation?


If you are using replica 2, then there is no prevention. Once they 
occur, you can resolve them using 
http://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/


If you want to prevent split-brain, you would need to use replica 3 or 
arbiter volume.


Regards,
Ravi


*From:*Ravishankar N [mailto:ravishan...@redhat.com]
*Sent:* Thursday, February 08, 2018 12:12 AM
*To:* Zhou, Cynthia (NSB - CN/Hangzhou) 
; Gluster-devel@gluster.org

*Subject:* Re: query about a split-brain problem found in glusterfs3.12.3

On 02/07/2018 10:39 AM, Zhou, Cynthia (NSB - CN/Hangzhou) wrote:

Hi glusterfs expert:

Good day.

Lately, we meet a glusterfs split brain problem in our env in
/mnt/export/testdir. We start 3 ior process (IOR tool) from non-sn
nodes, which is creating/removing files repeatedly in testdir.
then we reboot sn nodes(sn0 and sn1) by sequence. Then we meet
following problem.

Do you have some comments on how this could happen? And how to fix
it in this situation? Thanks!


Is the problem that split-brain is happening? Is this a replica 2 
volume? If yes, then it looks like it is expected behavior?

Regards
Ravi

gluster volume heal export info
Brick sn-0.local:/mnt/bricks/export/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick
/testdir - Is in split-brain

/testdir - Possibly undergoing heal

Status: Connected
Number of entries: 2

wait for a while …..

gluster volume heal export info
Brick sn-0.local:/mnt/bricks/export/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick
/testdir - Possibly undergoing heal

/testdir - Possibly undergoing heal

and finally:

[root@sn-0:/root ]
# gluster v heal export info
Brick sn-0.local:/mnt/bricks/export/brick

Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick

/testdir - Is in split-brain

Status: Connected
Number of entries: 1

[root@sn-0:/root ]

# getfattr -m .* -d -e  hex /mnt/bricks/export/brick/testdir

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/export/brick/testdir

trusted.gfid=0x5622cff893b3484dbdb6a20a0edb0e77

trusted.glusterfs.dht=0x0001

[root@sn-1:/root ]

# getfattr -m .* -d -e  hex /mnt/bricks/export/brick/testdir

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/export/brick/testdir

trusted.afr.dirty=0x0001

trusted.afr.export-client-0=0x0038

trusted.gfid=0x5622cff893b3484dbdb6a20a0edb0e77

trusted.glusterfs.dht=0x0001



___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] query about a split-brain problem found in glusterfs3.12.3

2018-02-07 Thread Ravishankar N 



On 02/07/2018 10:39 AM, Zhou, Cynthia (NSB - CN/Hangzhou) wrote:


Hi glusterfs expert:

Good day.

Lately, we meet a glusterfs split brain problem in our env in 
/mnt/export/testdir. We start 3 ior process (IOR tool) from non-sn 
nodes, which is creating/removing files repeatedly in testdir. then we 
reboot sn nodes(sn0 and sn1) by sequence. Then we meet following problem.


Do you have some comments on how this could happen? And how to fix it 
in this situation? Thanks!




Is the problem that split-brain is happening? Is this a replica 2 
volume? If yes, then it looks like it is expected behavior?

Regards
Ravi


gluster volume heal export info
Brick sn-0.local:/mnt/bricks/export/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick
/testdir - Is in split-brain

/testdir - Possibly undergoing heal

Status: Connected
Number of entries: 2

wait for a while …..

gluster volume heal export info
Brick sn-0.local:/mnt/bricks/export/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick
/testdir - Possibly undergoing heal

/testdir - Possibly undergoing heal

and finally:

[root@sn-0:/root ]
# gluster v heal export info
Brick sn-0.local:/mnt/bricks/export/brick 


Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick 


/testdir - Is in split-brain

Status: Connected
Number of entries: 1

[root@sn-0:/root ]

# getfattr -m .* -d -e  hex /mnt/bricks/export/brick/testdir

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/export/brick/testdir

trusted.gfid=0x5622cff893b3484dbdb6a20a0edb0e77

trusted.glusterfs.dht=0x0001

[root@sn-1:/root ]

# getfattr -m .* -d -e  hex /mnt/bricks/export/brick/testdir

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/export/brick/testdir

trusted.afr.dirty=0x0001

trusted.afr.export-client-0=0x0038

trusted.gfid=0x5622cff893b3484dbdb6a20a0edb0e77

trusted.glusterfs.dht=0x0001



___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] query about a split-brain problem found in glusterfs3.12.3

2018-02-07 Thread Zhou, Cynthia (NSB - CN/Hangzhou) 

Hi glusterfs expert:
   Good day.
   Lately, we meet a glusterfs split brain problem in our env in 
/mnt/export/testdir. We start 3 ior process (IOR tool) from non-sn nodes, which 
is creating/removing files repeatedly in testdir. then we reboot sn nodes(sn0 
and sn1) by sequence. Then we meet following problem.
Do you have some comments on how this could happen? And how to fix it in 
this situation? Thanks!


gluster volume heal export info
Brick sn-0.local:/mnt/bricks/export/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick
/testdir - Is in split-brain

/testdir - Possibly undergoing heal

Status: Connected
Number of entries: 2

wait for a while .

gluster volume heal export info
Brick sn-0.local:/mnt/bricks/export/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick
/testdir - Possibly undergoing heal

/testdir - Possibly undergoing heal

and finally:

[root@sn-0:/root]
# gluster v heal export info
Brick sn-0.local:/mnt/bricks/export/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick
/testdir - Is in split-brain

Status: Connected
Number of entries: 1



[root@sn-0:/root]

# getfattr -m .* -d -e  hex /mnt/bricks/export/brick/testdir

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/export/brick/testdir

trusted.gfid=0x5622cff893b3484dbdb6a20a0edb0e77

trusted.glusterfs.dht=0x0001



[root@sn-1:/root]

# getfattr -m .* -d -e  hex /mnt/bricks/export/brick/testdir

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/export/brick/testdir

trusted.afr.dirty=0x0001

trusted.afr.export-client-0=0x0038

trusted.gfid=0x5622cff893b3484dbdb6a20a0edb0e77

trusted.glusterfs.dht=0x0001


___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel

[Gluster-devel] query about a split-brain problem found in glusterfs3.12.3

2018-02-07 Thread Zhou, Cynthia (NSB - CN/Hangzhou) 
Hi glusterfs expert:
   Good day.
   Lately, we meet a glusterfs split brain problem in our env in 
/mnt/export/testdir. We start 3 ior process (IOR tool) from non-sn nodes, which 
is creating/removing files repeatedly in testdir. then we reboot sn nodes(sn0 
and sn1) by sequence. Then we meet following problem.
Do you have some comments on how this could happen? And how to fix it in 
this situation? Thanks!


gluster volume heal export info
Brick sn-0.local:/mnt/bricks/export/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick
/testdir - Is in split-brain

/testdir - Possibly undergoing heal

Status: Connected
Number of entries: 2

wait for a while …..

gluster volume heal export info
Brick sn-0.local:/mnt/bricks/export/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick
/testdir - Possibly undergoing heal

/testdir - Possibly undergoing heal
and finally:

[root@sn-0:/root]
# gluster v heal export info
Brick sn-0.local:/mnt/bricks/export/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/export/brick
/testdir - Is in split-brain

Status: Connected
Number of entries: 1



[root@sn-0:/root]

# getfattr -m .* -d -e  hex /mnt/bricks/export/brick/testdir

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/export/brick/testdir

trusted.gfid=0x5622cff893b3484dbdb6a20a0edb0e77

trusted.glusterfs.dht=0x0001



[root@sn-1:/root]

# getfattr -m .* -d -e  hex /mnt/bricks/export/brick/testdir

getfattr: Removing leading '/' from absolute path names

# file: mnt/bricks/export/brick/testdir

trusted.afr.dirty=0x0001

trusted.afr.export-client-0=0x0038

trusted.gfid=0x5622cff893b3484dbdb6a20a0edb0e77

trusted.glusterfs.dht=0x0001

___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel