Re: [Gluster-users] Files not healing & missing their extended attributes - Help!

[Gambit15]

On 3 July 2018 at 23:37, Vlad Kopylov  wrote:

> might be too late but sort of simple always working solution for such
> cases is rebuilding .glusterfs
>
> kill it and query attr for all files again, it will recreate .glusterfs on
> all bricks
>
> something like mentioned here
> https://lists.gluster.org/pipermail/gluster-users/2018-January/033352.html
>

Is my problem with .glusterfs though? I'd be super cautious removing the
entire directory unless I'm sure that's the solution...

Cheers,


> On Tue, Jul 3, 2018 at 4:27 PM, Gambit15  wrote:
>
>> On 1 July 2018 at 22:37, Ashish Pandey  wrote:
>>
>>>
>>> The only problem at the moment is that arbiter brick offline. You should
>>> only bother about completion of maintenance of arbiter brick ASAP.
>>> Bring this brick UP, start FULL heal or index heal and the volume will
>>> be in healthy state.
>>>
>>
>> Doesn't the arbiter only resolve split-brain situations? None of the
>> files that have been marked for healing are marked as in split-brain.
>>
>> The arbiter has now been brought back up, however the problem continues.
>>
>> I've found the following information in the client log:
>>
>> [2018-07-03 19:09:29.245089] W [MSGID: 108008]
>> [afr-self-heal-name.c:354:afr_selfheal_name_gfid_mismatch_check]
>> 0-engine-replicate-0: GFID mismatch for > dcd437ba7be2>/hosted-engine.metadata 5e95ba8c-2f12-49bf-be2d-b4baf210d366
>> on engine-client-1 and b9cd7613-3b96-415d-a549-1dc788a4f94d on
>> engine-client-0
>> [2018-07-03 19:09:29.245585] W [fuse-bridge.c:471:fuse_entry_cbk]
>> 0-glusterfs-fuse: 10430040: LOOKUP() /98495dbc-a29c-4893-b6a0-0aa70
>> 860d0c9/ha_agent/hosted-engine.metadata => -1 (Input/output error)
>> [2018-07-03 19:09:30.619000] W [MSGID: 108008]
>> [afr-self-heal-name.c:354:afr_selfheal_name_gfid_mismatch_check]
>> 0-engine-replicate-0: GFID mismatch for > dcd437ba7be2>/hosted-engine.lockspace 8e86902a-c31c-4990-b0c5-0318807edb8f
>> on engine-client-1 and e5899a4c-dc5d-487e-84b0-9bbc73133c25 on
>> engine-client-0
>> [2018-07-03 19:09:30.619360] W [fuse-bridge.c:471:fuse_entry_cbk]
>> 0-glusterfs-fuse: 10430656: LOOKUP() /98495dbc-a29c-4893-b6a0-0aa70
>> 860d0c9/ha_agent/hosted-engine.lockspace => -1 (Input/output error)
>>
>> As you can see from the logs I posted previously, neither of those two
>> files, on either of the two servers, have any of gluster's extended
>> attributes set.
>>
>> The arbiter doesn't have any record of the files in question, as they
>> were created after it went offline.
>>
>> How do I fix this? Is it possible to locate the correct gfids somewhere &
>> redefine them on the files manually?
>>
>> Cheers,
>>  Doug
>>
>> --
>>> *From: *"Gambit15" 
>>> *To: *"Ashish Pandey" 
>>> *Cc: *"gluster-users" 
>>> *Sent: *Monday, July 2, 2018 1:45:01 AM
>>> *Subject: *Re: [Gluster-users] Files not healing & missing their
>>> extended attributes - Help!
>>>
>>>
>>> Hi Ashish,
>>>
>>> The output is below. It's a rep 2+1 volume. The arbiter is offline for
>>> maintenance at the moment, however quorum is met & no files are reported as
>>> in split-brain (it hosts VMs, so files aren't accessed concurrently).
>>>
>>> ==
>>> [root@v0 glusterfs]# gluster volume info engine
>>>
>>> Volume Name: engine
>>> Type: Replicate
>>> Volume ID: 279737d3-3e5a-4ee9-8d4a-97edcca42427
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 1 x (2 + 1) = 3
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: s0:/gluster/engine/brick
>>> Brick2: s1:/gluster/engine/brick
>>> Brick3: s2:/gluster/engine/arbiter (arbiter)
>>> Options Reconfigured:
>>> nfs.disable: on
>>> performance.readdir-ahead: on
>>> transport.address-family: inet
>>> performance.quick-read: off
>>> performance.read-ahead: off
>>> performance.io-cache: off
>>> performance.stat-prefetch: off
>>> cluster.eager-lock: enable
>>> network.remote-dio: enable
>>> cluster.quorum-type: auto
>>> cluster.server-quorum-type: server
>>> storage.owner-uid: 36
>>> storage.owner-gid: 36
>>> performance.low-prio-threads: 32
>>>
>>> ==
>>>
>>> [root@v0 glusterfs]# gluster volume heal engine info
>>> Brick s0:/gluster/engine/brick
>>> /__DIRECT_IO_TEST__
>>> /98495dbc-a29c-4893-b6a0-0aa70860d0c9/ha_agent
>>> /98495dbc-a29c-4893-b6a0-0aa70860d0c9
>>> 
>>> Status: Connected
>>> Number of entries: 34
>>>
>>> Brick s1:/gluster/engine/brick
>>> 
>>> Status: Connected
>>> Number of entries: 34
>>>
>>> Brick s2:/gluster/engine/arbiter
>>> Status: Ponto final de transporte não está conectado
>>> Number of entries: -
>>>
>>> ==
>>> === PEER V0 ===
>>>
>>> [root@v0 glusterfs]# getfattr -m . -d -e hex
>>> /gluster/engine/brick/98495dbc-a29c-4893-b6a0-0aa70860d0c9/ha_agent
>>> getfattr: Removing leading '/' from absolute path names
>>> # file: gluster/engine/brick/98495dbc-a29c-4893-b6a0-0aa70860d0c9/ha
>>> _agent
>>> security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6
>>> c6162656c65645f743a733000
>>> 

Re: [Gluster-users] delettion of files in gluster directories

[Vlad Kopylov]

If you delete those from the bricks it will start healing them - restoring
from other bricks
I have similar issue with email storage which uses maildir format with
millions of small files

doing delete on the server takes days

sometimes worth recreating volumes wiping .glusterfs on bricks, deleting
files on bricks, creating volumes again and repopulating .glusterfs by
querying attr
https://lists.gluster.org/pipermail/gluster-users/2018-July/034310.html

On Wed, Jul 4, 2018 at 9:57 AM, hsafe  wrote:

> Hi all,
>
> I have a rather simplistic question, there are dirs that contain a lot of
> small files in a 2x replica set accessed natively on the clients. Due to
> the directory file number; it fails to show the dir contents from clients.
>
> In case of move or deletion of the dirs natively and from the server's
> view of the dirs , how does glusterfs converge or "heal" if you can call it
> the dirs as emptied or as if moved?
>
> I am running on Glusterfs-server and Glusterfs-client version: 3.10.12.
>
> To add more details,it is that we learned it the hard way that our app is
> shipping too small files into dirs with daily accumulaiton, accesed for
> serving by an nginx.
>
> Here is a little more info:
>
> # gluster volume info
>
> Volume Name: gv1
> Type: Replicate
> Volume ID: f1c955a1-7a92-4b1b-acb5-8b72b41aaace
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: IMG-01:/images/storage/brick1
> Brick2: IMG-02:/images/storage/brick1
> Options Reconfigured:
> nfs.disable: true
> diagnostics.count-fop-hits: on
> diagnostics.latency-measurement: on
> server.statedump-path: /tmp
> performance.readdir-ahead: on
> # gluster volume status
> Status of volume: gv1
> Gluster process TCP Port  RDMA Port Online
> Pid
> 
> --
> Brick IMG-01:/images/storage/brick1 49152 0 Y   3577
> Brick IMG-02:/images/storage/brick1 49152 0 Y   21699
> Self-heal Daemon on localhost   N/A   N/A Y   24813
> Self-heal Daemon on IMG-01  N/A   N/A Y   3560
>
> Task Status of Volume gv1
> 
> --
> There are no active volume tasks
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Files not healing & missing their extended attributes - Help!

[Gambit15]

Hi Karthik,
 Many thanks for the response!

On 4 July 2018 at 05:26, Karthik Subrahmanya  wrote:

> Hi,
>
> From the logs you have pasted it looks like those files are in GFID
> split-brain.
> They should have the GFIDs assigned on both the data bricks but they will
> be different.
>
> Can you please paste the getfattr output of those files and their parent
> from all the bricks again?
>

The files don't have any attributes set, however I did manage to find their
corresponding entries in .glusterfs

==
[root@v0 .glusterfs]# getfattr -m . -d -e hex
/gluster/engine/brick/98495dbc-a29c-4893-b6a0-0aa70860d0c9/ha_agent
getfattr: Removing leading '/' from absolute path names
# file: gluster/engine/brick/98495dbc-a29c-4893-b6a0-0aa70860d0c9/ha_agent
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x
trusted.afr.engine-client-2=0x24ea
trusted.gfid=0xdb9afb92d2bc49ed8e34dcd437ba7be2
trusted.glusterfs.dht=0x0001

[root@v0 .glusterfs]# getfattr -m . -d -e hex
/gluster/engine/brick/98495dbc-a29c-4893-b6a0-0aa70860d0c9/ha_agent/*
getfattr: Removing leading '/' from absolute path names
# file:
gluster/engine/brick/98495dbc-a29c-4893-b6a0-0aa70860d0c9/ha_agent/hosted-engine.lockspace
security.selinux=0x73797374656d5f753a6f626a6563745f723a6675736566735f743a733000

# file:
gluster/engine/brick/98495dbc-a29c-4893-b6a0-0aa70860d0c9/ha_agent/hosted-engine.metadata
security.selinux=0x73797374656d5f753a6f626a6563745f723a6675736566735f743a733000

[root@v0 .glusterfs]# ls -l
/gluster/engine/brick/.glusterfs/db/9a/db9afb92-d2bc-49ed-8e34-dcd437ba7be2/
total 0
lrwxrwxrwx. 2 vdsm kvm 132 Jun 30 14:55 hosted-engine.lockspace ->
/var/run/vdsm/storage/98495dbc-a29c-4893-b6a0-0aa70860d0c9/2502aff4-6c67-4643-b681-99f2c87e793d/03919182-6be2-4cbc-aea2-b9d68422a800
lrwxrwxrwx. 2 vdsm kvm 132 Jun 30 14:55 hosted-engine.metadata ->
/var/run/vdsm/storage/98495dbc-a29c-4893-b6a0-0aa70860d0c9/99510501-6bdc-485a-98e8-c2f82ff8d519/71fa7e6c-cdfb-4da8-9164-2404b518d0ee

==

Again, here are the relevant client log entries:

[2018-07-03 19:09:29.245089] W [MSGID: 108008]
[afr-self-heal-name.c:354:afr_selfheal_name_gfid_mismatch_check]
0-engine-replicate-0: GFID mismatch for
/hosted-engine.metadata
5e95ba8c-2f12-49bf-be2d-b4baf210d366 on engine-client-1 and
b9cd7613-3b96-415d-a549-1dc788a4f94d on engine-client-0
[2018-07-03 19:09:29.245585] W [fuse-bridge.c:471:fuse_entry_cbk]
0-glusterfs-fuse: 10430040: LOOKUP()
/98495dbc-a29c-4893-b6a0-0aa70860d0c9/ha_agent/hosted-engine.metadata => -1
(Input/output error)
[2018-07-03 19:09:30.619000] W [MSGID: 108008]
[afr-self-heal-name.c:354:afr_selfheal_name_gfid_mismatch_check]
0-engine-replicate-0: GFID mismatch for
/hosted-engine.lockspace
8e86902a-c31c-4990-b0c5-0318807edb8f on engine-client-1 and
e5899a4c-dc5d-487e-84b0-9bbc73133c25 on engine-client-0
[2018-07-03 19:09:30.619360] W [fuse-bridge.c:471:fuse_entry_cbk]
0-glusterfs-fuse: 10430656: LOOKUP()
/98495dbc-a29c-4893-b6a0-0aa70860d0c9/ha_agent/hosted-engine.lockspace =>
-1 (Input/output error)

[root@v0 .glusterfs]# find . -type f | grep -E
"5e95ba8c-2f12-49bf-be2d-b4baf210d366|8e86902a-c31c-4990-b0c5-0318807edb8f|b9cd7613-3b96-415d-a549-1dc788a4f94d|e5899a4c-dc5d-487e-84b0-9bbc73133c25"
[root@v0 .glusterfs]#

==


> Which version of gluster you are using?
>

3.8.5
An upgrade is on the books, however I had to go back on my last attempt as
3.12 didn't work with 3.8 & I was unable to do a live rolling upgrade. Once
I've got this GFID mess sorted out, I'll give a full upgrade a go as I've
already had to failover this cluster's services to another cluster.

If you are using a version higher than or equal to 3.12 gfid split brains
> can be resolved using the methods (except method 4)
> explained in the "Resolution of split-brain using gluster CLI" section in
> [1].
> Also note that for gfid split-brain resolution using CLI you have to pass
> the name of the file as argument and not the GFID.
>
> If it is lower than 3.12 (Please consider upgrading them since they are
> EOL) you have to resolve it manually as explained in [2]
>
> [1] https://docs.gluster.org/en/latest/Troubleshooting/
> resolving-splitbrain/
> [2] https://docs.gluster.org/en/latest/Troubleshooting/
> resolving-splitbrain/#dir-split-brain
>

"The user needs to remove either file '1' on brick-a or the file '1' on
brick-b to resolve the split-brain. In addition, the corresponding
gfid-link file also needs to be removed."

Okay, so as you can see above, the files don't have a trusted.gfid
attribute, & on the brick I didn't find any files in .glusterfs with the
same name as the GFID's reported in the client log. I did however find the
symlinked files in a .glusterfs directory under the parent directory's GFID.

[root@v0 .glusterfs]# ls -l

Re: [Gluster-users] Files not healing & missing their extended attributes - Help!

[Vlad Kopylov]

you'll need to query attr of those files for them to be updated in .
glusterfs

regarding wiping .glusterfs - I've done it half a dozen times on live data:
it is a simple drill which fixes almost everything.
often you don't have time to ask around etc. you just need it working ASAP
so you delete gluster, wipe all configs, versions, volumes and .glusterfs
on all bricks
reinstall everything, create volumes anew pointing to bricks with existing
data
then run attr query to each file for it to populate .glusterfs (
https://lists.gluster.org/pipermail/gluster-users/2018-January/033352.html)

In real life situations this works much faster then anything else
unless you suspect network issues or something else non gluster related

-v


On Wed, Jul 4, 2018 at 8:39 PM, Gambit15  wrote:

> Hi Karthik,
>  Many thanks for the response!
>
> On 4 July 2018 at 05:26, Karthik Subrahmanya  wrote:
>
>> Hi,
>>
>> From the logs you have pasted it looks like those files are in GFID
>> split-brain.
>> They should have the GFIDs assigned on both the data bricks but they will
>> be different.
>>
>> Can you please paste the getfattr output of those files and their parent
>> from all the bricks again?
>>
>
> The files don't have any attributes set, however I did manage to find
> their corresponding entries in .glusterfs
>
> ==
> [root@v0 .glusterfs]# getfattr -m . -d -e hex /gluster/engine/brick/
> 98495dbc-a29c-4893-b6a0-0aa70860d0c9/ha_agent
> getfattr: Removing leading '/' from absolute path names
> # file: gluster/engine/brick/98495dbc-a29c-4893-b6a0-0aa70860d0c9/ha_agent
> security.selinux=0x73797374656d5f753a6f626a6563
> 745f723a756e6c6162656c65645f743a733000
> trusted.afr.dirty=0x
> trusted.afr.engine-client-2=0x24ea
> trusted.gfid=0xdb9afb92d2bc49ed8e34dcd437ba7be2
> trusted.glusterfs.dht=0x0001
>
> [root@v0 .glusterfs]# getfattr -m . -d -e hex /gluster/engine/brick/
> 98495dbc-a29c-4893-b6a0-0aa70860d0c9/ha_agent/*
> getfattr: Removing leading '/' from absolute path names
> # file: gluster/engine/brick/98495dbc-a29c-4893-b6a0-0aa70860d0c9/
> ha_agent/hosted-engine.lockspace
> security.selinux=0x73797374656d5f753a6f626a6563
> 745f723a6675736566735f743a733000
>
> # file: gluster/engine/brick/98495dbc-a29c-4893-b6a0-0aa70860d0c9/
> ha_agent/hosted-engine.metadata
> security.selinux=0x73797374656d5f753a6f626a6563
> 745f723a6675736566735f743a733000
>
> [root@v0 .glusterfs]# ls -l /gluster/engine/brick/.
> glusterfs/db/9a/db9afb92-d2bc-49ed-8e34-dcd437ba7be2/
> total 0
> lrwxrwxrwx. 2 vdsm kvm 132 Jun 30 14:55 hosted-engine.lockspace ->
> /var/run/vdsm/storage/98495dbc-a29c-4893-b6a0-0aa70860d0c9/2502aff4-6c67-
> 4643-b681-99f2c87e793d/03919182-6be2-4cbc-aea2-b9d68422a800
> lrwxrwxrwx. 2 vdsm kvm 132 Jun 30 14:55 hosted-engine.metadata ->
> /var/run/vdsm/storage/98495dbc-a29c-4893-b6a0-0aa70860d0c9/99510501-6bdc-
> 485a-98e8-c2f82ff8d519/71fa7e6c-cdfb-4da8-9164-2404b518d0ee
>
> ==
>
> Again, here are the relevant client log entries:
>
> [2018-07-03 19:09:29.245089] W [MSGID: 108008]
> [afr-self-heal-name.c:354:afr_selfheal_name_gfid_mismatch_check]
> 0-engine-replicate-0: GFID mismatch for  dcd437ba7be2>/hosted-engine.metadata 5e95ba8c-2f12-49bf-be2d-b4baf210d366
> on engine-client-1 and b9cd7613-3b96-415d-a549-1dc788a4f94d on
> engine-client-0
> [2018-07-03 19:09:29.245585] W [fuse-bridge.c:471:fuse_entry_cbk]
> 0-glusterfs-fuse: 10430040: LOOKUP() /98495dbc-a29c-4893-b6a0-
> 0aa70860d0c9/ha_agent/hosted-engine.metadata => -1 (Input/output error)
> [2018-07-03 19:09:30.619000] W [MSGID: 108008]
> [afr-self-heal-name.c:354:afr_selfheal_name_gfid_mismatch_check]
> 0-engine-replicate-0: GFID mismatch for  dcd437ba7be2>/hosted-engine.lockspace 8e86902a-c31c-4990-b0c5-0318807edb8f
> on engine-client-1 and e5899a4c-dc5d-487e-84b0-9bbc73133c25 on
> engine-client-0
> [2018-07-03 19:09:30.619360] W [fuse-bridge.c:471:fuse_entry_cbk]
> 0-glusterfs-fuse: 10430656: LOOKUP() /98495dbc-a29c-4893-b6a0-
> 0aa70860d0c9/ha_agent/hosted-engine.lockspace => -1 (Input/output error)
>
> [root@v0 .glusterfs]# find . -type f | grep -E "5e95ba8c-2f12-49bf-be2d-
> b4baf210d366|8e86902a-c31c-4990-b0c5-0318807edb8f|b9cd7613-3b96-415d-a549-
> 1dc788a4f94d|e5899a4c-dc5d-487e-84b0-9bbc73133c25"
> [root@v0 .glusterfs]#
>
> ==
>
>
>> Which version of gluster you are using?
>>
>
> 3.8.5
> An upgrade is on the books, however I had to go back on my last attempt as
> 3.12 didn't work with 3.8 & I was unable to do a live rolling upgrade. Once
> I've got this GFID mess sorted out, I'll give a full upgrade a go as I've
> already had to failover this cluster's services to another cluster.
>
> If you are using a version higher than or equal to 3.12 gfid split brains
>> can be resolved using the methods (except method 4)
>> explained in the "Resolution of split-brain using gluster CLI" section in
>> [1].

[Gluster-users] Gluster 4.1 servers - KVM Centos 7.5 clients - replica 2 - continuous healing taking place.

[Claus Jeppesen]

We have a replica 2 setup using glusterfs-4.1.1-1 which we use to store VM
files. The Gluster
clients are Centos 7.5 using gluster "url" setup for the VM disks - e.g.
like:

   
 
 
   
 
 ...


We have the problem that gluster on the 2 brick server keeps on running
healing operations - it's
a bit hard to understand why as we cannot see any drops in network traffic
between the 2 bricks
(either errors or bandwidth issue).

   - Is there a fundamental versioning conflict ?
 (libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.5.x86_64 vs.
   gluster 4.1)

   - Is there a limit to how many clients a gluster server can handle ?

   - Some setting we need to adjust ?

I hope someone has an idea,

Thanx,

Claus.

-- 
*Claus Jeppesen*
Manager, Network Services
Datto, Inc.
p +45 6170 5901 | Copenhagen Office
www.datto.com


___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Gluster Outreachy

[Bhumika Goyal]

Hi all,

Gnome has been working on an initiative known as Outreachy[1] since 2010.
Outreachy is a three months remote internship program. It aims to increase
the participation of women and members from under-represented groups in
open source. This program is held twice in a year. During the internship
period, interns contribute to a project under the guidance of one or more
mentors.

For the next round(Dec 2018- March 2019) we are planning to apply projects
from Gluster. We would like you to propose projects ideas or/and come
forward as mentors/volunteers.
Please feel free to add project ideas in this doc[2]. The doc[2] will be
open for editing till July end.

[1]: https://www.outreachy.org/
[2]: https://docs.google.com/document/d/16yKKDD2Dd6Ag0tssrdoFPojKsF16Q
I5-j7cUHcR5Pq4/edit?usp=sharing

Outreachy timeline:
Pre-Application Period - Late August to early September
Application Period - Early September to mid-October
Internship Period -  December to March

Thanks,
Bhumika
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Announcing Glusterfs release 3.12.10 (Long Term Maintenance)

[Niels de Vos]

On Tue, Jul 03, 2018 at 05:20:44PM -0500, Darrell Budic wrote:
> I’ve now tested 3.12.11 on my centos 7.5 ovirt dev cluster, and all
> appears good. Should be safe to move from -test to release for
> centos-gluster312

Thanks for the report! Someone else already informed me earlier as well.
The packages are not available in the release yet, the CentOS team was
busy with CentOS 6.10 and that caused some delays. I expect the packages
to show up on the mirrors later this week.

Niels


> 
> Thanks!
> 
>   -Darrell
> 
> > From: Jiffin Tony Thottan 
> > Subject: [Gluster-users] Announcing Glusterfs release 3.12.10 (Long Term 
> > Maintenance)
> > Date: June 15, 2018 at 12:53:03 PM CDT
> > To: gluster-users@gluster.org, Gluster Devel; annou...@gluster.org
> > 
> > The Gluster community is pleased to announce the release of Gluster 3.12.10 
> > (packages available at [1,2,3]).
> > 
> > Release notes for the release can be found at [4].
> > 
> > Thanks,
> > Gluster community
> > 
> > 
> > [1] https://download.gluster.org/pub/gluster/glusterfs/3.12/3.12.10/
> > [2] https://launchpad.net/~gluster/+archive/ubuntu/glusterfs-3.12 
> > 
> > [3] https://build.opensuse.org/project/subprojects/home:glusterfs
> > [4] Release notes: 
> > https://gluster.readthedocs.io/en/latest/release-notes/3.12.10/
> > 
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://lists.gluster.org/mailman/listinfo/gluster-users
> 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] delettion of files in gluster directories

[hsafe]

Hi all,

I have a rather simplistic question, there are dirs that contain a lot 
of small files in a 2x replica set accessed natively on the clients. Due 
to the directory file number; it fails to show the dir contents from 
clients.


In case of move or deletion of the dirs natively and from the server's 
view of the dirs , how does glusterfs converge or "heal" if you can call 
it the dirs as emptied or as if moved?


I am running on Glusterfs-server and Glusterfs-client version: 3.10.12.

To add more details,it is that we learned it the hard way that our app 
is shipping too small files into dirs with daily accumulaiton, accesed 
for serving by an nginx.


Here is a little more info:

# gluster volume info

Volume Name: gv1
Type: Replicate
Volume ID: f1c955a1-7a92-4b1b-acb5-8b72b41aaace
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: IMG-01:/images/storage/brick1
Brick2: IMG-02:/images/storage/brick1
Options Reconfigured:
nfs.disable: true
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
server.statedump-path: /tmp
performance.readdir-ahead: on
# gluster volume status
Status of volume: gv1
Gluster process TCP Port  RDMA Port Online  Pid
--
Brick IMG-01:/images/storage/brick1 49152 0 Y   3577
Brick IMG-02:/images/storage/brick1 49152 0 Y   21699
Self-heal Daemon on localhost   N/A   N/A Y   24813
Self-heal Daemon on IMG-01  N/A   N/A Y   3560

Task Status of Volume gv1
--
There are no active volume tasks


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Files not healing & missing their extended attributes - Help!

[Karthik Subrahmanya]

Hi,

>From the logs you have pasted it looks like those files are in GFID
split-brain.
They should have the GFIDs assigned on both the data bricks but they will
be different.

Can you please paste the getfattr output of those files and their parent
from all the bricks again?
Which version of gluster you are using?

If you are using a version higher than or equal to 3.12 gfid split brains
can be resolved using the methods (except method 4)
explained in the "Resolution of split-brain using gluster CLI" section in
[1].
Also note that for gfid split-brain resolution using CLI you have to pass
the name of the file as argument and not the GFID.

If it is lower than 3.12 (Please consider upgrading them since they are
EOL) you have to resolve it manually as explained in [2]

[1] https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/
[2]
https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/#dir-split-brain

Thanks & Regards,
Karthik

On Wed, Jul 4, 2018 at 1:59 AM Gambit15  wrote:

> On 1 July 2018 at 22:37, Ashish Pandey  wrote:
>
>>
>> The only problem at the moment is that arbiter brick offline. You should
>> only bother about completion of maintenance of arbiter brick ASAP.
>> Bring this brick UP, start FULL heal or index heal and the volume will be
>> in healthy state.
>>
>
> Doesn't the arbiter only resolve split-brain situations? None of the files
> that have been marked for healing are marked as in split-brain.
>
> The arbiter has now been brought back up, however the problem continues.
>
> I've found the following information in the client log:
>
> [2018-07-03 19:09:29.245089] W [MSGID: 108008]
> [afr-self-heal-name.c:354:afr_selfheal_name_gfid_mismatch_check]
> 0-engine-replicate-0: GFID mismatch for
> /hosted-engine.metadata
> 5e95ba8c-2f12-49bf-be2d-b4baf210d366 on engine-client-1 and
> b9cd7613-3b96-415d-a549-1dc788a4f94d on engine-client-0
> [2018-07-03 19:09:29.245585] W [fuse-bridge.c:471:fuse_entry_cbk]
> 0-glusterfs-fuse: 10430040: LOOKUP()
> /98495dbc-a29c-4893-b6a0-0aa70860d0c9/ha_agent/hosted-engine.metadata => -1
> (Input/output error)
> [2018-07-03 19:09:30.619000] W [MSGID: 108008]
> [afr-self-heal-name.c:354:afr_selfheal_name_gfid_mismatch_check]
> 0-engine-replicate-0: GFID mismatch for
> /hosted-engine.lockspace
> 8e86902a-c31c-4990-b0c5-0318807edb8f on engine-client-1 and
> e5899a4c-dc5d-487e-84b0-9bbc73133c25 on engine-client-0
> [2018-07-03 19:09:30.619360] W [fuse-bridge.c:471:fuse_entry_cbk]
> 0-glusterfs-fuse: 10430656: LOOKUP()
> /98495dbc-a29c-4893-b6a0-0aa70860d0c9/ha_agent/hosted-engine.lockspace =>
> -1 (Input/output error)
>
> As you can see from the logs I posted previously, neither of those two
> files, on either of the two servers, have any of gluster's extended
> attributes set.
>
> The arbiter doesn't have any record of the files in question, as they were
> created after it went offline.
>
> How do I fix this? Is it possible to locate the correct gfids somewhere &
> redefine them on the files manually?
>
> Cheers,
>  Doug
>
> --
>> *From: *"Gambit15" 
>> *To: *"Ashish Pandey" 
>> *Cc: *"gluster-users" 
>> *Sent: *Monday, July 2, 2018 1:45:01 AM
>> *Subject: *Re: [Gluster-users] Files not healing & missing their
>> extended attributes - Help!
>>
>>
>> Hi Ashish,
>>
>> The output is below. It's a rep 2+1 volume. The arbiter is offline for
>> maintenance at the moment, however quorum is met & no files are reported as
>> in split-brain (it hosts VMs, so files aren't accessed concurrently).
>>
>> ==
>> [root@v0 glusterfs]# gluster volume info engine
>>
>> Volume Name: engine
>> Type: Replicate
>> Volume ID: 279737d3-3e5a-4ee9-8d4a-97edcca42427
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 1 x (2 + 1) = 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: s0:/gluster/engine/brick
>> Brick2: s1:/gluster/engine/brick
>> Brick3: s2:/gluster/engine/arbiter (arbiter)
>> Options Reconfigured:
>> nfs.disable: on
>> performance.readdir-ahead: on
>> transport.address-family: inet
>> performance.quick-read: off
>> performance.read-ahead: off
>> performance.io-cache: off
>> performance.stat-prefetch: off
>> cluster.eager-lock: enable
>> network.remote-dio: enable
>> cluster.quorum-type: auto
>> cluster.server-quorum-type: server
>> storage.owner-uid: 36
>> storage.owner-gid: 36
>> performance.low-prio-threads: 32
>>
>> ==
>>
>> [root@v0 glusterfs]# gluster volume heal engine info
>> Brick s0:/gluster/engine/brick
>> /__DIRECT_IO_TEST__
>> /98495dbc-a29c-4893-b6a0-0aa70860d0c9/ha_agent
>> /98495dbc-a29c-4893-b6a0-0aa70860d0c9
>> 
>> Status: Connected
>> Number of entries: 34
>>
>> Brick s1:/gluster/engine/brick
>> 
>> Status: Connected
>> Number of entries: 34
>>
>> Brick s2:/gluster/engine/arbiter
>> Status: Ponto final de transporte não está conectado
>> Number of entries: -
>>
>> ==
>> === PEER V0 ===
>>
>> [root@v0 glusterfs]# getfattr -m . -d -e hex
>> 

Re: [Gluster-users] New 3.12.7 possible split-brain on replica 3

[Ravishankar N]

Hi mabi, there are a couple of AFR patches  from master that I'm 
currently back porting to the 3.12 branch:


afr: heal gfids when file is not present on all bricks
afr: don't update readables if inode refresh failed on all children
afr: fix bug-1363721.t failure
afr: add quorum checks in pre-op
afr: don't treat all cases all bricks being blamed as split-brain
afr: capture the correct errno in post-op quorum check
afr: add quorum checks in post-op

Many of these help make the transaction code more robust by fixing 
various corner cases. It would be great if you can wait for the next 
3.12 minor release (3.12.12 ?) and upgrade to that build and see if the 
issues go away.


Note: CC'ing Karthik and Jiffin for their help in reviewing and merging 
the backports for the above patches.


Thanks,
Ravi

On 07/04/2018 06:51 PM, mabi wrote:

Hello,

I just wanted to let you know that last week I have upgraded my two replica 
nodes from Debian 8 to Debian 9 so now all my 3 nodes (including aribter) are 
running Debian 9 with a Linux 4 kernel.

Unfortunately I still have the exact same issue. Another detail I might have 
not mentioned yet is that I have quotas enabled on this volume, I don't really 
know if that is relevant but who knows...

As a reminder here is what happens on the client side which has the volume 
mounted via FUSE (take earlier today from the 
/var/log/glusterfs/mnt-myvol-private.log logfile). Note here that in this 
specific case it's only one single file who had this issue.

[2018-07-04 08:23:49.314252] E [MSGID: 109089] 
[dht-helper.c:1481:dht_migration_complete_check_task] 0-myvol-private-dht: 
failed to open the fd (0x7fccb00a5120, flags=010) on file 
/dir1/data/dir2/files_encryption/keys/files/dir3/dir4/dir5/dir6/dir7/OC_DEFAULT_MODULE/file.shareKey
 @ myvol-replicate-0 [Input/output error]
[2018-07-04 08:23:49.328712] W [MSGID: 108027] 
[afr-common.c:2821:afr_discover_done] 0-myvol-private-replicate-0: no read 
subvols for 
/dir1/data/dir2/files_encryption/keys/files/dir3/dir4/dir5/dir6/dir7/OC_DEFAULT_MODULE/file.shareKey
[2018-07-04 08:23:49.330749] W [fuse-bridge.c:779:fuse_truncate_cbk] 
0-glusterfs-fuse: 55916791: TRUNCATE() 
/dir1/data/dir2/files_encryption/keys/files/dir3/dir4/dir5/dir6/dir7/OC_DEFAULT_MODULE/file.shareKey
 => -1 (Input/output error)

Best regards,
M.

‐‐‐ Original Message ‐‐‐

On June 22, 2018 4:44 PM, mabi  wrote:


​​

Hi,

Now that this issue has happened a few times I noticed a few things which might 
be helpful for debugging:

-   This problem happens when files are uploaded via a cloud app called 
Nextcloud where the files are encrypted by the app itself on the server side 
(PHP code) but only rarely and randomly.
-   It does not seem to happen with Nextcloud installation which does not have 
server side encryption enabled.
-   When this happens both first and second node of the replica have 120k of 
context switches and 25k interrupts, the arbiter node 30k context switches/20k 
interrupts. No nodes are overloaded, there is no io/wait and no network issues 
or disconnections.
-   All of the problematic files to heal have spaces in one of their 
sub-directories (might be totally irrelevant).
 
 If that's of any use my two replica nodes are Debian 8 physical servers with ZFS as file system for the bricks and the arbiter is a Debian 9 virtual machine with XFS as file system for the brick. To mount the volume I use a glusterfs fuse mount on the web server which has Nextcloud running.
 
 Regards,
 
 M.
 
 ‐‐‐ Original Message ‐‐‐
 
 On May 25, 2018 5:55 PM, mabi m...@protonmail.ch wrote:
 


Thanks Ravi. Let me know when you have time to have a look. It sort of happens 
around once or twice per week but today it was 24 files in one go which are 
unsynched and where I need to manually reset the xattrs on the arbiter node.

By the way on this volume I use quotas which I set on specifc directories, I 
don't know if this is relevant or not but thought I would just mention.

‐‐‐ Original Message ‐‐‐

On May 23, 2018 9:25 AM, Ravishankar N ravishan...@redhat.com wrote:


On 05/23/2018 12:47 PM, mabi wrote:


Hello,

I just wanted to ask if you had time to look into this bug I am encountering 
and if there is anything else I can do?

For now in order to get rid of these 3 unsynched files shall I do the same 
method that was suggested to me in this thread?

Sorry Mabi,  I haven't had a chance to dig deeper into this. The

workaround of resetting xattrs should be fine though.

Thanks,

Ravi


Thanks,

Mabi

‐‐‐ Original Message ‐‐‐

On May 17, 2018 11:07 PM, mabi m...@protonmail.ch wrote:


Hi Ravi,

Please fine below the answers to your questions

1.  I have never touched the cluster.quorum-type option. Currently it is set as 
following for this volume:
 
 Option Value
 


cluster.quorum-type none

2.  The .shareKey files are not supposed to be empty. They should be 512 bytes 
big 

Re: [Gluster-users] Failed to mount nfs due to split-brain and Input/Output Error

[Anh Vo]

If I run "sudo gluster volume heal gv0 split-brain latest-mtime /" I get
the following:

Lookup failed on /:Invalid argument.
Volume heal failed.

node2 was not connected at that time, because if we connect it to the
system after a few minutes gluster will become almost unusable and we have
many jobs failing. This morning I reconnected it and ran heal info and we
have about 3 entries to heal (15K from gfs-vm000 and 15k from
gfs-vm001, 80% are all gfid, 20% have file names). It's not feasible for us
to check the individual gfid so we kinda rely on gluster self heal to
handle those gfid. The "/" is a concern because it prevents us from
mounting nfs. We do need to mount nfs for some of our management because
gluster fuse mount is much slower compared to nfs when it comes to
recursive operations like 'du'

Do you have any suggestion for healing the metadata on '/' ?

Thanks
Anh

On Tue, Jul 3, 2018 at 8:02 PM, Ravishankar N 
wrote:

> Hi,
>
> What version of gluster are you using?
>
> 1. The afr xattrs on '/' indicate a meta-data split-brain. You can resolve
> it using one of the policies listed in https://docs.gluster.org/en/
> latest/Troubleshooting/resolving-splitbrain/
>
> For example, "gluster volume heal gv0 split-brain latest-mtime / "
> 2. Is the file corresponding to the other gfid 
> (81289110-867b-42ff-ba3b-1373a187032b)
> present in all bricks? What do the getfattr outputs for this file indicate?
>
> 3. As for the discrepancy in output of heal info, is node2 connected to
> the other nodes? Does heal info still print the details of all 3 bricks
> when you run it on node2 ?
> -Ravi
>
>
> On 07/04/2018 01:47 AM, Anh Vo wrote:
>
> Actually we just discovered that the heal info command was returning
> different things when executed on the different nodes of our 3-replica
> setup.
> When we execute it on node2 we did not see the split brain reported "/"
> but if I execute it on node0 and node1 I am seeing:
>
> x@gfs-vm001:~$ sudo gluster volume heal gv0 info | tee heal-info
> Brick gfs-vm000:/gluster/brick/brick0
> 
> / - Is in split-brain
>
> Status: Connected
> Number of entries: 2
>
> Brick gfs-vm001:/gluster/brick/brick0
> / - Is in split-brain
>
> 
> Status: Connected
> Number of entries: 2
>
> Brick gfs-vm002:/gluster/brick/brick0
> / - Is in split-brain
>
> Status: Connected
> Number of entries: 1
>
>
> I ran getfattr -d -m . -e hex /gluster/brick/brick0 on all three nodes and
> I am seeing node2 has slightly different attr:
> node0:
> sudo getfattr -d -m . -e hex /gluster/brick/brick0
> getfattr: Removing leading '/' from absolute path names
> # file: gluster/brick/brick0
> trusted.afr.gv0-client-2=0x0001
> trusted.gfid=0x0001
> trusted.glusterfs.dht=0x0001
> trusted.glusterfs.volume-id=0x7fa3aac372d543f987ed0c66b77f02e2
>
> node1:
> sudo getfattr -d -m . -e hex /gluster/brick/brick0
> getfattr: Removing leading '/' from absolute path names
> # file: gluster/brick/brick0
> trusted.afr.gv0-client-2=0x0001
> trusted.gfid=0x0001
> trusted.glusterfs.dht=0x0001
> trusted.glusterfs.volume-id=0x7fa3aac372d543f987ed0c66b77f02e2
>
> node2:
> sudo getfattr -d -m . -e hex /gluster/brick/brick0
> getfattr: Removing leading '/' from absolute path names
> # file: gluster/brick/brick0
> trusted.afr.dirty=0x
> trusted.afr.gv0-client-0=0x0002
> trusted.afr.gv0-client-1=0x0002
> trusted.afr.gv0-client-2=0x
> trusted.gfid=0x0001
> trusted.glusterfs.dht=0x0001
> trusted.glusterfs.volume-id=0x7fa3aac372d543f987ed0c66b77f02e2
>
> Where do I go from here? Thanks
>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Failed to mount nfs due to split-brain and Input/Output Error

[Anh Vo]

I forgot to mention we're using 3.12.10

On Wed, Jul 4, 2018 at 8:45 AM, Anh Vo  wrote:

> If I run "sudo gluster volume heal gv0 split-brain latest-mtime /" I get
> the following:
>
> Lookup failed on /:Invalid argument.
> Volume heal failed.
>
> node2 was not connected at that time, because if we connect it to the
> system after a few minutes gluster will become almost unusable and we have
> many jobs failing. This morning I reconnected it and ran heal info and we
> have about 3 entries to heal (15K from gfs-vm000 and 15k from
> gfs-vm001, 80% are all gfid, 20% have file names). It's not feasible for us
> to check the individual gfid so we kinda rely on gluster self heal to
> handle those gfid. The "/" is a concern because it prevents us from
> mounting nfs. We do need to mount nfs for some of our management because
> gluster fuse mount is much slower compared to nfs when it comes to
> recursive operations like 'du'
>
> Do you have any suggestion for healing the metadata on '/' ?
>
> Thanks
> Anh
>
> On Tue, Jul 3, 2018 at 8:02 PM, Ravishankar N 
> wrote:
>
>> Hi,
>>
>> What version of gluster are you using?
>>
>> 1. The afr xattrs on '/' indicate a meta-data split-brain. You can
>> resolve it using one of the policies listed in
>> https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/
>>
>> For example, "gluster volume heal gv0 split-brain latest-mtime / "
>> 2. Is the file corresponding to the other gfid
>> (81289110-867b-42ff-ba3b-1373a187032b) present in all bricks? What do
>> the getfattr outputs for this file indicate?
>>
>> 3. As for the discrepancy in output of heal info, is node2 connected to
>> the other nodes? Does heal info still print the details of all 3 bricks
>> when you run it on node2 ?
>> -Ravi
>>
>>
>> On 07/04/2018 01:47 AM, Anh Vo wrote:
>>
>> Actually we just discovered that the heal info command was returning
>> different things when executed on the different nodes of our 3-replica
>> setup.
>> When we execute it on node2 we did not see the split brain reported "/"
>> but if I execute it on node0 and node1 I am seeing:
>>
>> x@gfs-vm001:~$ sudo gluster volume heal gv0 info | tee heal-info
>> Brick gfs-vm000:/gluster/brick/brick0
>> 
>> / - Is in split-brain
>>
>> Status: Connected
>> Number of entries: 2
>>
>> Brick gfs-vm001:/gluster/brick/brick0
>> / - Is in split-brain
>>
>> 
>> Status: Connected
>> Number of entries: 2
>>
>> Brick gfs-vm002:/gluster/brick/brick0
>> / - Is in split-brain
>>
>> Status: Connected
>> Number of entries: 1
>>
>>
>> I ran getfattr -d -m . -e hex /gluster/brick/brick0 on all three nodes
>> and I am seeing node2 has slightly different attr:
>> node0:
>> sudo getfattr -d -m . -e hex /gluster/brick/brick0
>> getfattr: Removing leading '/' from absolute path names
>> # file: gluster/brick/brick0
>> trusted.afr.gv0-client-2=0x0001
>> trusted.gfid=0x0001
>> trusted.glusterfs.dht=0x0001
>> trusted.glusterfs.volume-id=0x7fa3aac372d543f987ed0c66b77f02e2
>>
>> node1:
>> sudo getfattr -d -m . -e hex /gluster/brick/brick0
>> getfattr: Removing leading '/' from absolute path names
>> # file: gluster/brick/brick0
>> trusted.afr.gv0-client-2=0x0001
>> trusted.gfid=0x0001
>> trusted.glusterfs.dht=0x0001
>> trusted.glusterfs.volume-id=0x7fa3aac372d543f987ed0c66b77f02e2
>>
>> node2:
>> sudo getfattr -d -m . -e hex /gluster/brick/brick0
>> getfattr: Removing leading '/' from absolute path names
>> # file: gluster/brick/brick0
>> trusted.afr.dirty=0x
>> trusted.afr.gv0-client-0=0x0002
>> trusted.afr.gv0-client-1=0x0002
>> trusted.afr.gv0-client-2=0x
>> trusted.gfid=0x0001
>> trusted.glusterfs.dht=0x0001
>> trusted.glusterfs.volume-id=0x7fa3aac372d543f987ed0c66b77f02e2
>>
>> Where do I go from here? Thanks
>>
>>
>>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Failed to mount nfs due to split-brain and Input/Output Error

[Ravishankar N]

On 07/04/2018 09:20 PM, Anh Vo wrote:

I forgot to mention we're using 3.12.10

On Wed, Jul 4, 2018 at 8:45 AM, Anh Vo > wrote:


If I run "sudo gluster volume heal gv0 split-brain latest-mtime /"
I get the following:

Lookup failed on /:Invalid argument.
Volume heal failed.



Can you share the glfsheal-.log on the node where you ran this 
failed command?



node2 was not connected at that time, because if we connect it to
the system after a few minutes gluster will become almost unusable
and we have many jobs failing. This morning I reconnected it and
ran heal info and we have about 3 entries to heal (15K from
gfs-vm000 and 15k from gfs-vm001, 80% are all gfid, 20% have file
names). It's not feasible for us to check the individual gfid so
we kinda rely on gluster self heal to handle those gfid. The "/"
is a concern because it prevents us from mounting nfs. We do need
to mount nfs for some of our management because gluster fuse mount
is much slower compared to nfs when it comes to recursive
operations like 'du'

Do you have any suggestion for healing the metadata on '/' ?


You can manually delete the afr xattrs on node 3 as a workaround:
setfattr -x trusted.afr.gv0-client-0 gluster/brick/brick0
setfattr -x trusted.afr.gv0-client-1 gluster/brick/brick0

This should remove the split-brain on root.

HTH,
Ravi



Thanks
Anh

On Tue, Jul 3, 2018 at 8:02 PM, Ravishankar N
mailto:ravishan...@redhat.com>> wrote:

Hi,

What version of gluster are you using?

1. The afr xattrs on '/' indicate a meta-data split-brain. You
can resolve it using one of the policies listed in
https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/



For example, "|gluster volume heal gv0 split-brain
latest-mtime / "
|

2. Is the file corresponding to the other gfid
(81289110-867b-42ff-ba3b-1373a187032b) present in all bricks?
What do the getfattr outputs for this file indicate?

3. As for the discrepancy in output of heal info, is node2
connected to the other nodes? Does heal info still print the
details of all 3 bricks when you run it on node2 ?
-Ravi


On 07/04/2018 01:47 AM, Anh Vo wrote:

Actually we just discovered that the heal info command was
returning different things when executed on the different
nodes of our 3-replica setup.
When we execute it on node2 we did not see the split brain
reported "/" but if I execute it on node0 and node1 I am seeing:

x@gfs-vm001:~$ sudo gluster volume heal gv0 info | tee heal-info
Brick gfs-vm000:/gluster/brick/brick0

/ - Is in split-brain

Status: Connected
Number of entries: 2

Brick gfs-vm001:/gluster/brick/brick0
/ - Is in split-brain


Status: Connected
Number of entries: 2

Brick gfs-vm002:/gluster/brick/brick0
/ - Is in split-brain

Status: Connected
Number of entries: 1


I ran getfattr -d -m . -e hex /gluster/brick/brick0 on all
three nodes and I am seeing node2 has slightly different attr:
node0:
sudo getfattr -d -m . -e hex /gluster/brick/brick0
getfattr: Removing leading '/' from absolute path names
# file: gluster/brick/brick0
trusted.afr.gv0-client-2=0x0001
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x7fa3aac372d543f987ed0c66b77f02e2

node1:
sudo getfattr -d -m . -e hex /gluster/brick/brick0
getfattr: Removing leading '/' from absolute path names
# file: gluster/brick/brick0
trusted.afr.gv0-client-2=0x0001
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x7fa3aac372d543f987ed0c66b77f02e2

node2:
sudo getfattr -d -m . -e hex /gluster/brick/brick0
getfattr: Removing leading '/' from absolute path names
# file: gluster/brick/brick0
trusted.afr.dirty=0x
trusted.afr.gv0-client-0=0x0002
trusted.afr.gv0-client-1=0x0002
trusted.afr.gv0-client-2=0x
trusted.gfid=0x0001
trusted.glusterfs.dht=0x0001
trusted.glusterfs.volume-id=0x7fa3aac372d543f987ed0c66b77f02e2

Where do I go from here? Thanks






___
Gluster-users mailing list
Gluster-users@gluster.org

Re: [Gluster-users] New 3.12.7 possible split-brain on replica 3

[mabi]

Hello,

I just wanted to let you know that last week I have upgraded my two replica 
nodes from Debian 8 to Debian 9 so now all my 3 nodes (including aribter) are 
running Debian 9 with a Linux 4 kernel.

Unfortunately I still have the exact same issue. Another detail I might have 
not mentioned yet is that I have quotas enabled on this volume, I don't really 
know if that is relevant but who knows...

As a reminder here is what happens on the client side which has the volume 
mounted via FUSE (take earlier today from the 
/var/log/glusterfs/mnt-myvol-private.log logfile). Note here that in this 
specific case it's only one single file who had this issue.

[2018-07-04 08:23:49.314252] E [MSGID: 109089] 
[dht-helper.c:1481:dht_migration_complete_check_task] 0-myvol-private-dht: 
failed to open the fd (0x7fccb00a5120, flags=010) on file 
/dir1/data/dir2/files_encryption/keys/files/dir3/dir4/dir5/dir6/dir7/OC_DEFAULT_MODULE/file.shareKey
 @ myvol-replicate-0 [Input/output error]
[2018-07-04 08:23:49.328712] W [MSGID: 108027] 
[afr-common.c:2821:afr_discover_done] 0-myvol-private-replicate-0: no read 
subvols for 
/dir1/data/dir2/files_encryption/keys/files/dir3/dir4/dir5/dir6/dir7/OC_DEFAULT_MODULE/file.shareKey
[2018-07-04 08:23:49.330749] W [fuse-bridge.c:779:fuse_truncate_cbk] 
0-glusterfs-fuse: 55916791: TRUNCATE() 
/dir1/data/dir2/files_encryption/keys/files/dir3/dir4/dir5/dir6/dir7/OC_DEFAULT_MODULE/file.shareKey
 => -1 (Input/output error) 

Best regards,
M.

‐‐‐ Original Message ‐‐‐

On June 22, 2018 4:44 PM, mabi  wrote:

> ​​
> 
> Hi,
> 
> Now that this issue has happened a few times I noticed a few things which 
> might be helpful for debugging:
> 
> -   This problem happens when files are uploaded via a cloud app called 
> Nextcloud where the files are encrypted by the app itself on the server side 
> (PHP code) but only rarely and randomly.
> -   It does not seem to happen with Nextcloud installation which does not 
> have server side encryption enabled.
> -   When this happens both first and second node of the replica have 120k of 
> context switches and 25k interrupts, the arbiter node 30k context 
> switches/20k interrupts. No nodes are overloaded, there is no io/wait and no 
> network issues or disconnections.
> -   All of the problematic files to heal have spaces in one of their 
> sub-directories (might be totally irrelevant).
> 
> If that's of any use my two replica nodes are Debian 8 physical servers 
> with ZFS as file system for the bricks and the arbiter is a Debian 9 virtual 
> machine with XFS as file system for the brick. To mount the volume I use a 
> glusterfs fuse mount on the web server which has Nextcloud running.
> 
> Regards,
> 
> M.
> 
> ‐‐‐ Original Message ‐‐‐
> 
> On May 25, 2018 5:55 PM, mabi m...@protonmail.ch wrote:
> 
> 
> > Thanks Ravi. Let me know when you have time to have a look. It sort of 
> > happens around once or twice per week but today it was 24 files in one go 
> > which are unsynched and where I need to manually reset the xattrs on the 
> > arbiter node.
> > 
> > By the way on this volume I use quotas which I set on specifc directories, 
> > I don't know if this is relevant or not but thought I would just mention.
> > 
> > ‐‐‐ Original Message ‐‐‐
> > 
> > On May 23, 2018 9:25 AM, Ravishankar N ravishan...@redhat.com wrote:
> > 
> > > On 05/23/2018 12:47 PM, mabi wrote:
> > > 
> > > > Hello,
> > > > 
> > > > I just wanted to ask if you had time to look into this bug I am 
> > > > encountering and if there is anything else I can do?
> > > > 
> > > > For now in order to get rid of these 3 unsynched files shall I do the 
> > > > same method that was suggested to me in this thread?
> > > 
> > > Sorry Mabi,  I haven't had a chance to dig deeper into this. The
> > > 
> > > workaround of resetting xattrs should be fine though.
> > > 
> > > Thanks,
> > > 
> > > Ravi
> > > 
> > > > Thanks,
> > > > 
> > > > Mabi
> > > > 
> > > > ‐‐‐ Original Message ‐‐‐
> > > > 
> > > > On May 17, 2018 11:07 PM, mabi m...@protonmail.ch wrote:
> > > > 
> > > > > Hi Ravi,
> > > > > 
> > > > > Please fine below the answers to your questions
> > > > > 
> > > > > 1.  I have never touched the cluster.quorum-type option. Currently it 
> > > > > is set as following for this volume:
> > > > > 
> > > > > Option Value
> > > > > 
> > > > > 
> > > > > cluster.quorum-type none
> > > > > 
> > > > > 2.  The .shareKey files are not supposed to be empty. They should be 
> > > > > 512 bytes big and contain binary data (PGP Secret Sub-key). I am not 
> > > > > in a position to say why it is in this specific case only 0 bytes and 
> > > > > if it is the fault of the software (Nextcloud) or GlusterFS. I can 
> > > > > just say here that I have another file server which is a simple NFS 
> > > > > server with another Nextcloud installation and there I never saw any 
> > > > > 0 bytes .shareKey files being created.

Re: [Gluster-users] Failed to mount nfs due to split-brain and Input/Output Error

[Anh Vo]

Output of glfsheal-gv0.log:

[2018-07-04 16:11:05.435680] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-gv0-client-1: Server lk version = 1
[2018-07-04 16:11:05.436847] I [rpc-clnt.c:1986:rpc_clnt_reconfig]
0-gv0-client-2: changing port to 49153 (from 0)
[2018-07-04 16:11:05.437722] W [MSGID: 114007]
[client-handshake.c:1190:client_setvolume_cbk]
0-gv0-client-0: failed to find key 'child_up' in the options
[2018-07-04 16:11:05.437744] I [MSGID: 114046]
[client-handshake.c:1231:client_setvolume_cbk]
0-gv0-client-0: Connected to gv0-client-0, attached to remote volume
'/gluster/brick/brick0'.
[2018-07-04 16:11:05.437755] I [MSGID: 114047]
[client-handshake.c:1242:client_setvolume_cbk]
0-gv0-client-0: Server and Client lk-version numbers are not same,
reopening the fds
[2018-07-04 16:11:05.531514] I [MSGID: 108002]
[afr-common.c:5312:afr_notify] 0-gv0-replicate-0: Client-quorum is met
[2018-07-04 16:11:05.531550] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-gv0-client-0: Server lk version = 1
[2018-07-04 16:11:05.532115] I [MSGID: 114057] [client-handshake.c:1478:
select_server_supported_programs] 0-gv0-client-2: Using Program GlusterFS
3.3, Num (1298437), Version (330)
[2018-07-04 16:11:05.537528] I [MSGID: 114046]
[client-handshake.c:1231:client_setvolume_cbk]
0-gv0-client-2: Connected to gv0-client-2, attached to remote volume
'/gluster/brick/brick0'.
[2018-07-04 16:11:05.537569] I [MSGID: 114047]
[client-handshake.c:1242:client_setvolume_cbk]
0-gv0-client-2: Server and Client lk-version numbers are not same,
reopening the fds
[2018-07-04 16:11:05.544248] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-gv0-client-2: Server lk version = 1
[2018-07-04 16:11:05.547665] I [MSGID: 108031]
[afr-common.c:2458:afr_local_discovery_cbk]
0-gv0-replicate-0: selecting local read_child gv0-client-1
[2018-07-04 16:11:05.556948] W [MSGID: 108027]
[afr-common.c:2821:afr_discover_done]
0-gv0-replicate-0: no read subvols for /
[2018-07-04 16:11:05.577751] W [MSGID: 108027]
[afr-common.c:2821:afr_discover_done]
0-gv0-replicate-0: no read subvols for /
[2018-07-04 16:11:05.577839] I [MSGID: 104041]
[glfs-resolve.c:971:__glfs_active_subvol]
0-gv0: switched to graph 6766732d-766d-3030-312d-37373932362d (0)
[2018-07-04 16:11:05.578355] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-gv0-client-1: remote operation failed. Path: /
(----)
[Invalid argument]
[2018-07-04 16:11:05.579562] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-gv0-client-0: remote operation failed. Path: /
(----)
[Invalid argument]
[2018-07-04 16:11:05.579776] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-gv0-client-2: remote operation failed. Path: /
(----)
[Invalid argument]

Removing the afr xattrs on node 3 did solve the split brain issue on root.
Thank you!


On Wed, Jul 4, 2018 at 9:01 AM, Ravishankar N 
wrote:

>
>
> On 07/04/2018 09:20 PM, Anh Vo wrote:
>
> I forgot to mention we're using 3.12.10
>
> On Wed, Jul 4, 2018 at 8:45 AM, Anh Vo  wrote:
>
>> If I run "sudo gluster volume heal gv0 split-brain latest-mtime /" I get
>> the following:
>>
>> Lookup failed on /:Invalid argument.
>> Volume heal failed.
>>
>
> Can you share the glfsheal-.log on the node where you ran this
> failed command?
>
>
>> node2 was not connected at that time, because if we connect it to the
>> system after a few minutes gluster will become almost unusable and we have
>> many jobs failing. This morning I reconnected it and ran heal info and we
>> have about 3 entries to heal (15K from gfs-vm000 and 15k from
>> gfs-vm001, 80% are all gfid, 20% have file names). It's not feasible for us
>> to check the individual gfid so we kinda rely on gluster self heal to
>> handle those gfid. The "/" is a concern because it prevents us from
>> mounting nfs. We do need to mount nfs for some of our management because
>> gluster fuse mount is much slower compared to nfs when it comes to
>> recursive operations like 'du'
>>
>> Do you have any suggestion for healing the metadata on '/' ?
>>
> You can manually delete the afr xattrs on node 3 as a workaround:
> setfattr -x trusted.afr.gv0-client-0 gluster/brick/brick0
> setfattr -x trusted.afr.gv0-client-1 gluster/brick/brick0
>
> This should remove the split-brain on root.
>
> HTH,
> Ravi
>
>
>> Thanks
>> Anh
>>
>> On Tue, Jul 3, 2018 at 8:02 PM, Ravishankar N 
>> wrote:
>>
>>> Hi,
>>>
>>> What version of gluster are you using?
>>>
>>> 1. The afr xattrs on '/' indicate a meta-data split-brain. You can
>>> resolve it using one of the policies listed in
>>> https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/
>>>
>>> For example, "gluster volume heal gv0 split-brain latest-mtime / "
>>> 2. Is the file corresponding to the other gfid
>>> (81289110-867b-42ff-ba3b-1373a187032b) present in all bricks? What