Re: [Gluster-users] Indexing/Importing existing files on disk

[Olaf Buitelaar]

Hi Strahil,

interesting, would you happen to know how you could let oVirt mount it with
this flag?

Thanks Olaf

Op ma 4 apr. 2022 om 22:05 schreef Strahil Nikolov :

> * it instructs the kernel to skip 'getfattr'
>
> On Mon, Apr 4, 2022 at 23:04, Strahil Nikolov
>  wrote:
> I agree. Also it has a typo.Obviously I needed more coffee or a bigger
> break.
>
> It should be treated as 'relatime/noatime reduce the ammount of I/O
> requests to your brick physical device , which increases performance by
> reducing unnecessary IOPS'
>
> Same is valid for SELINUX's mount option:
> context="system_u:object_r:glusterd_brick_t:s0" . It instructs the kernel
> to 'getfattr' each file in the brick as it has a specific selinux context
> (this case Gluster brick files), thus reducing unnecessary IOPS.
>
>
>
> Best Regards,
> Strahil Nikolov
>
> On Mon, Apr 4, 2022 at 9:41, Diego Zuccato
>  wrote:
> Il 03/04/2022 16:36, Strahil Nikolov ha scritto:
>
> > Using relative/noatime mount option reduces the I/O to the brick
> device.IMVHO this sentence could cause misunderstandings. :)
> It could be read like "noatime slows down your brick" while, IIUC, it
> really means it *improves* the brick's performance by reducing the
> number of "housekeeping" IOs.
>
> --
> Diego Zuccato
> DIFA - Dip. di Fisica e Astronomia
> Servizi Informatici
> Alma Mater Studiorum - Università di Bologna
> V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
> tel.: +39 051 20 95786
>
> 
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
> 
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Changing harddrive

[Olaf Buitelaar]

Hi Stefan,

the size doesn't matter, as long you have enough.

Best regards,

Olaf

Op do 31 mrt. 2022 om 08:45 schreef Stefan Kania :

> That's the way I also wuld prefere, but what about thr size? If the size
> of the new SSD differs from the size of the HDD?
>
>
> Am 31.03.22 um 00:53 schrieb Strahil Nikolov:
> > With replica volumes I prefer to avoid 'reset-brick' in favor of
> > 'remove-brick replica '  and once you replace the ssds
> > 'add-brick replica ' + gluster volume heal  full
> >
> > Best Regards,
> > Strahil Nikolov
> >
> > On Wed, Mar 30, 2022 at 20:41, Olaf Buitelaar
> >  wrote:
> > 
> >
> >
> >
> > Community Meeting Calendar:
> >
> > Schedule -
> > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> > Bridge: https://meet.google.com/cpu-eiue-hvk
> > <https://meet.google.com/cpu-eiue-hvk>
> > Gluster-users mailing list
> > Gluster-users@gluster.org <mailto:Gluster-users@gluster.org>
> > https://lists.gluster.org/mailman/listinfo/gluster-users
> > <https://lists.gluster.org/mailman/listinfo/gluster-users>
> >
>
> --
> Stefan Kania
> Landweg 13
> 25693 St. Michaelisdonn
>
>
> Signieren jeder E-Mail hilft Spam zu reduzieren und schützt Ihre
> Privatsphäre. Ein kostenfreies Zertifikat erhalten Sie unter
> https://www.dgn.de/dgncert/index.html
> Download der root-Zertifikate: https://www.dgn.de/dgncert/downloads.html
>
>
>




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Changing harddrive

[Olaf Buitelaar]

Hi Stefan,

I think there are 3 routes;
1. you can use the reset-brick command;
https://docs.gluster.org/en/latest/release-notes/3.9.0/#introducing-reset-brick-command
2. copy the files, make sure the gluster processes are stopped before you
start the copy, and make sure to include all included attributes. you could
use rsync like; rsync -aAHXz. Do one host per replacement, and wait for the
heal to complete, before moving to the next.
3. make a file system dump of the hdd's and restore it on your ssd's.

Best regards,

Olaf

Op di 29 mrt. 2022 om 18:38 schreef Stefan Kania :

> Hi to all,
>
> I'm having a replicated Cluster with three nodes. At the moment Gluster
> 6.x is running. I would like to upgrade to 10.x. After upgrading I would
> like to change the disks from HDD to SSD. Can I just remove one of the
> bricks replace the HDD with SSD, then add the brick to the volume again.
> Do the step wit all three nodes. One thing, the size of the SSD is not
> the same as the size of the HDD.
> Or is there a better way to change disks?
>
> Stefan
> 
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] ​Can't mount particular brick even though the brick port is reachable, error message "Transport endpoint is not connected"

[Olaf Buitelaar]

Hi Peter,

I think Staril means, running the command; hosted-engine --set-maintenance
--mode=local, this is also possible from the ovirt ui, via the ribbon on
the hosts section;
[image: image.png]

>From the log's it seems gluster has difficulty find the shared's, e.g.;
.shard/e5f699e2-de11-41be-bd24-e29876928f0f.1279
(be318638-e8a0-4c6d-977d-7a937aa84806/e5f699e2-de11-41be-bd24-e29876928f0f.1279
Do these files exist within your brick's directories?
Did you try to repair the filesystem using xfs_repair?

Best regards,

Olaf

Op ma 28 mrt. 2022 om 10:01 schreef Peter Schmidt <
peterschmidt18...@yandex.com>:

>
> Hi Olaf,
>
> I tried running "gluster volume start hdd force" but sadly it did not
> change anything.
>
> the raid rebuild has finished now and everything seems to be fine:
> md6 : active raid6 sdu1[2] sdx1[5] sds1[0] sdt1[1] sdz1[7] sdv1[3] sdw1[4]
> sdaa1[8] sdy1[6]
>   68364119040 blocks super 1.2 level 6, 512k chunk, algorithm 2 [9/9]
> [U]
>   bitmap: 0/73 pages [0KB], 65536KB chunk
>
> Best regards
> Peter
>
> 25.03.2022, 12:36, "Olaf Buitelaar" :
>
> Hi Peter,
>
> I see your raid array is rebuilding, could it be your xfs needs a repair,
> using xfs_repair?
> did you try running gluster v hdd start force?
>
> Kind regards,
>
> Olaf
>
>
> Op do 24 mrt. 2022 om 15:54 schreef Peter Schmidt <
> peterschmidt18...@yandex.com>:
>
> Hello everyone,
>
> I'm running an oVirt cluster on top of a distributed-replicate gluster
> volume and one of the bricks cannot be mounted anymore from my oVirt hosts.
> This morning I also noticed a stack trace and a spike in TCP connections on
> one of the three gluster nodes (storage2), which I have attached at the end
> of this mail. Only this particular brick on storage2 seems to be causing
> trouble:
> *Brick storage2:/data/glusterfs/hdd/brick3/brick*
> *Status: Transport endpoint is not connected*
>
> I don't know what's causing this or how to resolve this issue. I would
> appreciate it if someone could take a look at my logs and point me in the
> right direction. If any additional logs are required, please let me know.
> Thank you in advance!
>
> Operating system on all hosts: Centos 7.9.2009
> oVirt version: 4.3.10.4-1
> Gluster versions:
> - storage1: 6.10-1
> - storage2: 6.7-1
> - storage3: 6.7-1
>
> 
> # brick is not connected/mounted on the oVirt hosts
>
> *[xlator.protocol.client.hdd-client-7.priv]*
> *fd.0.remote_fd = -1*
> *-- = --*
> *granted-posix-lock[0] = owner = 9d673ffe323e25cd, cmd = F_SETLK fl_type =
> F_RDLCK, fl_start = 100, fl_end = 100, user_flock: l_type = F_RDLCK,
> l_start = 100, l_len = 1*
> *granted-posix-lock[1] = owner = 9d673ffe323e25cd, cmd = F_SETLK fl_type =
> F_RDLCK, fl_start = 101, fl_end = 101, user_flock: l_type = F_RDLCK,
> l_start = 101, l_len = 1*
> *-- = --*
> *connected = 0*
> *total_bytes_read = 11383136800*
> *ping_timeout = 10*
> *total_bytes_written = 16699851552*
> *ping_msgs_sent = 1*
> *msgs_sent = 2*
>
> 
> # mount log from one of the oVirt hosts
> # the IP 172.22.102.142 corresponds to my gluster node "storage2"
> # the port 49154 corresponds to the brick
> storage2:/data/glusterfs/hdd/brick3/brick
>
> *[2022-03-24 10:59:28.138178] W [rpc-clnt-ping.c:210:rpc_clnt_ping_cbk]
> 0-hdd-client-7: socket disconnected*
> *[2022-03-24 10:59:38.142698] I [rpc-clnt.c:2028:rpc_clnt_reconfig]
> 0-hdd-client-7: changing port to 49154 (from 0)*
> *The message "I [MSGID: 114018] [client.c:2331:client_rpc_notify]
> 0-hdd-client-7: disconnected from hdd-client-7. Client process will keep
> trying to connect to glusterd until brick's port is available" repeated 4
> times between [2022-03-24 10:58:04.114741] and [2022-03-24 10:59:28.137380]*
> *The message "W [MSGID: 114032]
> [client-handshake.c:1546:client_dump_version_cbk] 0-hdd-client-7: received
> RPC status error [Transport endpoint is not connected]" repeated 4 times
> between [2022-03-24 10:58:04.115169] and [2022-03-24 10:59:28.138052]*
> *[2022-03-24 10:59:49.143217] C
> [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 0-hdd-client-7: server
> 172.22.102.142:49154 <http://172.22.102.142:49154/> has not responded in
> the last 10 seconds, disconnecting.*
> *[2022-03-24 10:59:49.143838] I [MSGID: 114018]
> [client.c:2331:client_rpc_notify] 0-hdd-client-7: disconnected from
> hdd-client-7. Client process will keep trying to connect to glusterd until
> brick's port is available*
> *[2022-03-24 10:59:49.144540] E [rpc-clnt.c:346:saved_frames_unwind] (-->
> /l

Re: [Gluster-users] ​Can't mount particular brick even though the brick port is reachable, error message "Transport endpoint is not connected"

[Olaf Buitelaar]

Hi Peter,

I see your raid array is rebuilding, could it be your xfs needs a repair,
using xfs_repair?
did you try running gluster v hdd start force?

Kind regards,

Olaf


Op do 24 mrt. 2022 om 15:54 schreef Peter Schmidt <
peterschmidt18...@yandex.com>:

> Hello everyone,
>
> I'm running an oVirt cluster on top of a distributed-replicate gluster
> volume and one of the bricks cannot be mounted anymore from my oVirt hosts.
> This morning I also noticed a stack trace and a spike in TCP connections on
> one of the three gluster nodes (storage2), which I have attached at the end
> of this mail. Only this particular brick on storage2 seems to be causing
> trouble:
> *Brick storage2:/data/glusterfs/hdd/brick3/brick*
> *Status: Transport endpoint is not connected*
>
> I don't know what's causing this or how to resolve this issue. I would
> appreciate it if someone could take a look at my logs and point me in the
> right direction. If any additional logs are required, please let me know.
> Thank you in advance!
>
> Operating system on all hosts: Centos 7.9.2009
> oVirt version: 4.3.10.4-1
> Gluster versions:
> - storage1: 6.10-1
> - storage2: 6.7-1
> - storage3: 6.7-1
>
> 
> # brick is not connected/mounted on the oVirt hosts
>
> *[xlator.protocol.client.hdd-client-7.priv]*
> *fd.0.remote_fd = -1*
> *-- = --*
> *granted-posix-lock[0] = owner = 9d673ffe323e25cd, cmd = F_SETLK fl_type =
> F_RDLCK, fl_start = 100, fl_end = 100, user_flock: l_type = F_RDLCK,
> l_start = 100, l_len = 1*
> *granted-posix-lock[1] = owner = 9d673ffe323e25cd, cmd = F_SETLK fl_type =
> F_RDLCK, fl_start = 101, fl_end = 101, user_flock: l_type = F_RDLCK,
> l_start = 101, l_len = 1*
> *-- = --*
> *connected = 0*
> *total_bytes_read = 11383136800*
> *ping_timeout = 10*
> *total_bytes_written = 16699851552*
> *ping_msgs_sent = 1*
> *msgs_sent = 2*
>
> 
> # mount log from one of the oVirt hosts
> # the IP 172.22.102.142 corresponds to my gluster node "storage2"
> # the port 49154 corresponds to the brick
> storage2:/data/glusterfs/hdd/brick3/brick
>
> *[2022-03-24 10:59:28.138178] W [rpc-clnt-ping.c:210:rpc_clnt_ping_cbk]
> 0-hdd-client-7: socket disconnected*
> *[2022-03-24 10:59:38.142698] I [rpc-clnt.c:2028:rpc_clnt_reconfig]
> 0-hdd-client-7: changing port to 49154 (from 0)*
> *The message "I [MSGID: 114018] [client.c:2331:client_rpc_notify]
> 0-hdd-client-7: disconnected from hdd-client-7. Client process will keep
> trying to connect to glusterd until brick's port is available" repeated 4
> times between [2022-03-24 10:58:04.114741] and [2022-03-24 10:59:28.137380]*
> *The message "W [MSGID: 114032]
> [client-handshake.c:1546:client_dump_version_cbk] 0-hdd-client-7: received
> RPC status error [Transport endpoint is not connected]" repeated 4 times
> between [2022-03-24 10:58:04.115169] and [2022-03-24 10:59:28.138052]*
> *[2022-03-24 10:59:49.143217] C
> [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 0-hdd-client-7: server
> 172.22.102.142:49154  has not responded in the
> last 10 seconds, disconnecting.*
> *[2022-03-24 10:59:49.143838] I [MSGID: 114018]
> [client.c:2331:client_rpc_notify] 0-hdd-client-7: disconnected from
> hdd-client-7. Client process will keep trying to connect to glusterd until
> brick's port is available*
> *[2022-03-24 10:59:49.144540] E [rpc-clnt.c:346:saved_frames_unwind] (-->
> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f6724643adb] (-->
> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f67243ea7e4] (-->
> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f67243ea8fe] (-->
> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f67243eb987] (-->
> /lib64/libgfrpc.so.0(+0xf518)[0x7f67243ec518] ) 0-hdd-client-7: forced
> unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2022-03-24
> 10:59:38.145208 (xid=0x861)*
> *[2022-03-24 10:59:49.144557] W [MSGID: 114032]
> [client-handshake.c:1546:client_dump_version_cbk] 0-hdd-client-7: received
> RPC status error [Transport endpoint is not connected]*
> *[2022-03-24 10:59:49.144653] E [rpc-clnt.c:346:saved_frames_unwind] (-->
> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f6724643adb] (-->
> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f67243ea7e4] (-->
> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f67243ea8fe] (-->
> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f67243eb987] (-->
> /lib64/libgfrpc.so.0(+0xf518)[0x7f67243ec518] ) 0-hdd-client-7: forced
> unwinding frame type(GF-DUMP) op(NULL(2)) called at 2022-03-24
> 10:59:38.145218 (xid=0x862)*
> *[2022-03-24 10:59:49.144665] W [rpc-clnt-ping.c:210:rpc_clnt_ping_cbk]
> 0-hdd-client-7: socket disconnected*
>
> 
> # netcat/telnet to the brick's port of storage2 are working
>
> *[root@storage1  ~]#  netcat -z -v 172.22.102.142 49154*
> *Connection to 172.22.102.142 49154 port [tcp/*] succeeded!*
>
> *[root@storage3  ~]# netcat -z -v 172.22.102.142 49154*
> *Connection to 172.22.102

Re: [Gluster-users] [EXT] Renaming a brick.

[Olaf Buitelaar]

Hi Tom,

I think you could use the replace-brick sub-command; like: "gluster volume
replace-brick  :/bricks/0/gv01
 :/bricks/0/abc-gv01  commit force"
see;
https://docs.gluster.org/en/latest/Administrator-Guide/Managing-Volumes/#replace-brick

Kind regards,

Olaf

Op di 4 jan. 2022 om 12:50 schreef Stefan Solbrig :

> Hi,
>
> I use the sequence below to move a brick to a different peer. I assume it
> also works for renaming a brick (just put peer01 = peer02 =
> whateveryourpeeris).  I used this on a distributed-only volume (no
> replication) running glusterfs 9.3.
> Please note that I didn't to extensive tests and I didn't find any
> documentation if this is officially supported.
>
> best wishes,
> -Stefan
>
> #--
>
> # stop glusterfsd for brick:
>
> gluster v reset-brick gvol peer01:/old/path start
>
> # in your case,  you probably want to do something here like
>
> mv  /old/path /new/path
>
> #  add the moved path as new brick:
>
> gluster v add-brick gvol peer02:/new/path force
>
> # Order is important!
> # if brick is removed before other brick is added,
> # will lead to duplicate files.
> #  remove old brick (which does not exist any longer)
>
> gluster v remove-brick gvol peer01:/old/path force
>
> # probably not necessary in your case:
>
> gluster v rebalance gvol fix-layout  start
>
> #--
>
>
> > Hey All,
> >
> > Is there a way to rename a brick within an existing gluster volume?
> >
> > /bricks/0/gv01  -> /bricks/0/abc-gv01
> >
> >
> >
> > --
> > Thx,
> > TK.
> >
>
> 
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] glusterfs health-check failed, (brick) going down

[Olaf Buitelaar]

Hi Jiri,

Unfortunately i don't know a solution to fix it, other than what i already
mentioned, which doesn't seem to be applicable to your specific setup.
I don't think it's ovirt related (running ovirt my-self as well, but being
stuck at 4.3 atm, since centos 7 is not supported for 4.4).
If memory serves me well, i believe i start seeing this issue after
upgrading from glusterfs 3.12 to 4.x (I believe this whent together with
upgrade from ovirt 4.1 to 4.2), then since is version i've observed this
issue. currently running 7.9.
It would be nice to get to the bottom of this. I'm still not 100% sure it
might even be a glusterfs issue, or something might be wrong with XFS or
somewhere else in the IO stack. But I don't know what the next debugging
steps could be.
Just as a side note i've also observed this issue on systems without LVM
cache.

Cheers Olaf


Op do 8 jul. 2021 om 16:53 schreef Jiří Sléžka :

> Hi Olaf,
>
> thanks for reply.
>
> On 7/8/21 3:29 PM, Olaf Buitelaar wrote:
> > Hi Jiri,
> >
> > your probleem looks pretty similar to mine, see;
> >
> https://lists.gluster.org/pipermail/gluster-users/2021-February/039134.html
> > <
> https://lists.gluster.org/pipermail/gluster-users/2021-February/039134.html
> >
> > Any chance you also see the xfs errors in de brick logs?
>
> yes, I can see this log lines related to "health-check failed" items
>
> [root@ovirt-hci02 ~]# grep "aio_read" /var/log/glusterfs/bricks/*
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-07
> 07:13:37.408010] W [MSGID: 113075]
> [posix-helpers.c:2135:posix_fs_health_check] 0-vms-posix:
> aio_read_cmp_buf() on /gluster_bricks/vms2/vms2/.glusterfs/health_check
> returned ret is -1 error is Structure needs cleaning
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-07
> 16:11:14.518844] W [MSGID: 113075]
> [posix-helpers.c:2135:posix_fs_health_check] 0-vms-posix:
> aio_read_cmp_buf() on /gluster_bricks/vms2/vms2/.glusterfs/health_check
> returned ret is -1 error is Structure needs cleaning
>
> [root@ovirt-hci01 ~]# grep "aio_read" /var/log/glusterfs/bricks/*
> /var/log/glusterfs/bricks/gluster_bricks-engine-engine.log:[2021-07-05
> 13:15:51.982938] W [MSGID: 113075]
> [posix-helpers.c:2135:posix_fs_health_check] 0-engine-posix:
> aio_read_cmp_buf() on
> /gluster_bricks/engine/engine/.glusterfs/health_check returned ret is -1
> error is Structure needs cleaning
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-05
> 01:53:35.768534] W [MSGID: 113075]
> [posix-helpers.c:2135:posix_fs_health_check] 0-vms-posix:
> aio_read_cmp_buf() on /gluster_bricks/vms2/vms2/.glusterfs/health_check
> returned ret is -1 error is Structure needs cleaning
>
> it looks very similar to your issue but in my case I don't use LVM cache
> and brick disks are JBOD (but connected through Broadcom / LSI MegaRAID
> SAS-3 3008 [Fury] (rev 02)).
>
> > For me the situation improved once i disabled brick multiplexing, but i
> > don't see that in your volume configuration.
>
> probably important is your note...
>
> > When i kill the brick process and start with "gluser v start x force" the
> > issue seems much more unlikely to occur, but when started from a fresh
> > reboot, or when killing the process and let it being started by glusterd
> > (e.g. service glusterd start) the error seems to arise after a couple of
> > minutes.
>
> ...because in the ovirt list Jayme replied this
>
>
> https://lists.ovirt.org/archives/list/us...@ovirt.org/message/BZRONK53OGWSOPUSGQ76GIXUM7J6HHMJ/
>
> and it looks to me like something you also observes.
>
> Cheers, Jiri
>
> >
> > Cheers Olaf
> >
> > Op do 8 jul. 2021 om 12:28 schreef Jiří Sléžka  > <mailto:jiri.sle...@slu.cz>>:
> >
> > Hello gluster community,
> >
> > I am new to this list but using glusterfs for log time as our SDS
> > solution for storing 80+TiB of data. I'm also using glusterfs for
> small
> > 3 node HCI cluster with oVirt 4.4.6 and CentOS 8 (not stream yet).
> > Glusterfs version here is 8.5-2.el8.x86_64.
> >
> > For time to time (I belive) random brick on random host goes down
> > because health-check. It looks like
> >
> > [root@ovirt-hci02 ~]# grep "posix_health_check"
> > /var/log/glusterfs/bricks/*
> > /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-07
> > 07:13:37.408184] M [MSGID: 113075]
> > [posix-helpers.c:2214:posix_health_check_thread_proc] 0-vms-posix:
> > health-check failed, going down
> > /var/log/gl

Re: [Gluster-users] glusterfs health-check failed, (brick) going down

[Olaf Buitelaar]

Hi Jiri,

your probleem looks pretty similar to mine, see;
https://lists.gluster.org/pipermail/gluster-users/2021-February/039134.html
Any chance you also see the xfs errors in de brick logs?
For me the situation improved once i disabled brick multiplexing, but i
don't see that in your volume configuration.

Cheers Olaf

Op do 8 jul. 2021 om 12:28 schreef Jiří Sléžka :

> Hello gluster community,
>
> I am new to this list but using glusterfs for log time as our SDS
> solution for storing 80+TiB of data. I'm also using glusterfs for small
> 3 node HCI cluster with oVirt 4.4.6 and CentOS 8 (not stream yet).
> Glusterfs version here is 8.5-2.el8.x86_64.
>
> For time to time (I belive) random brick on random host goes down
> because health-check. It looks like
>
> [root@ovirt-hci02 ~]# grep "posix_health_check"
> /var/log/glusterfs/bricks/*
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-07
> 07:13:37.408184] M [MSGID: 113075]
> [posix-helpers.c:2214:posix_health_check_thread_proc] 0-vms-posix:
> health-check failed, going down
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-07
> 07:13:37.408407] M [MSGID: 113075]
> [posix-helpers.c:2232:posix_health_check_thread_proc] 0-vms-posix: still
> alive! -> SIGTERM
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-07
> 16:11:14.518971] M [MSGID: 113075]
> [posix-helpers.c:2214:posix_health_check_thread_proc] 0-vms-posix:
> health-check failed, going down
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-07
> 16:11:14.519200] M [MSGID: 113075]
> [posix-helpers.c:2232:posix_health_check_thread_proc] 0-vms-posix: still
> alive! -> SIGTERM
>
> on other host
>
> [root@ovirt-hci01 ~]# grep "posix_health_check"
> /var/log/glusterfs/bricks/*
> /var/log/glusterfs/bricks/gluster_bricks-engine-engine.log:[2021-07-05
> 13:15:51.983327] M [MSGID: 113075]
> [posix-helpers.c:2214:posix_health_check_thread_proc] 0-engine-posix:
> health-check failed, going down
> /var/log/glusterfs/bricks/gluster_bricks-engine-engine.log:[2021-07-05
> 13:15:51.983728] M [MSGID: 113075]
> [posix-helpers.c:2232:posix_health_check_thread_proc] 0-engine-posix:
> still alive! -> SIGTERM
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-05
> 01:53:35.769129] M [MSGID: 113075]
> [posix-helpers.c:2214:posix_health_check_thread_proc] 0-vms-posix:
> health-check failed, going down
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-05
> 01:53:35.769819] M [MSGID: 113075]
> [posix-helpers.c:2232:posix_health_check_thread_proc] 0-vms-posix: still
> alive! -> SIGTERM
>
> I cannot link these errors to any storage/fs issue (in dmesg or
> /var/log/messages), brick devices looks healthy (smartd).
>
> I can force start brick with
>
> gluster volume start vms|engine force
>
> and after some healing all works fine for few days
>
> Did anybody observe this behavior?
>
> vms volume has this structure (two bricks per host, each is separate
> JBOD ssd disk), engine volume has one brick on each host...
>
> gluster volume info vms
>
> Volume Name: vms
> Type: Distributed-Replicate
> Volume ID: 52032ec6-99d4-4210-8fb8-ffbd7a1e0bf7
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 2 x 3 = 6
> Transport-type: tcp
> Bricks:
> Brick1: 10.0.4.11:/gluster_bricks/vms/vms
> Brick2: 10.0.4.13:/gluster_bricks/vms/vms
> Brick3: 10.0.4.12:/gluster_bricks/vms/vms
> Brick4: 10.0.4.11:/gluster_bricks/vms2/vms2
> Brick5: 10.0.4.13:/gluster_bricks/vms2/vms2
> Brick6: 10.0.4.12:/gluster_bricks/vms2/vms2
> Options Reconfigured:
> cluster.granular-entry-heal: enable
> performance.stat-prefetch: off
> cluster.eager-lock: enable
> performance.io-cache: off
> performance.read-ahead: off
> performance.quick-read: off
> user.cifs: off
> network.ping-timeout: 30
> network.remote-dio: off
> performance.strict-o-direct: on
> performance.low-prio-threads: 32
> features.shard: on
> storage.owner-gid: 36
> storage.owner-uid: 36
> transport.address-family: inet
> storage.fips-mode-rchecksum: on
> nfs.disable: on
> performance.client-io-threads: off
>
>
> Cheers,
>
> Jiri
>
> 
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster usage scenarios in HPC cluster management

[Olaf Buitelaar]

Hi Erik,

Thanks for sharing your unique use-case and solution. It was very
interesting to read your write-up.

I agree with your point; " * Growing volumes, replacing bricks, and
replacing servers work.
However, the process is very involved and quirky for us. I have."
in your use-case 1 last point.

I do seem to suffer from similar issues where glusterd just doesn't want to
start up correctly at first time, maybe also see:
https://lists.gluster.org/pipermail/gluster-users/2021-February/039134.html as
one possible cause for not starting up correctly.
And secondly indeed for a "molten to the floor" server it would be great if
gluster instead of the current replace-brick commands, would have something
to replace a complete host, and would just recreate the all bricks (or
better yet all missing bricks, in case some survived from another
RAID/disk), which it would expect on that host, and would heal them all.
right now indeed this process is quite involved, and sometimes feels a bit
like performing black magic.

Best Olaf

Op vr 19 mrt. 2021 om 16:10 schreef Erik Jacobson :

> A while back I was asked to make a blog or something similar to discuss
> the use cases the team I work on (HPCM cluster management) at HPE.
>
> If you are not interested in reading about what I'm up to, just delete
> this and move on.
>
> I really don't have a public blogging mechanism so I'll just describe
> what we're up to here. Some of this was posted in some form in the past.
> Since this contains the raw materials, I could make a wiki-ized version
> if there were a public place to put it.
>
>
>
> We currently use gluster in two parts of cluster management.
>
> In fact, gluster in our management node infrastructure is helping us to
> provide scaling and consistency to some of the largest clusters in the
> world, clusters in the TOP100 list. While I can get in to trouble by
> sharing too much, I will just say that trends are continuing and the
> future may have some exciting announcements on where on TOP100 certain
> new giant systems may end up in the coming 1-2 years.
>
> At HPE, HPCM is the "traditional cluster manager." There is another team
> that develops a more cloud-like solution and I am not discussing that
> solution here.
>
>
> Use Case #1: Leader Nodes and Scale Out
>
> --
> - Why?
>   * Scale out
>   * Redundancy (combined with CTDB, any leader can fail)
>   * Consistency (All servers and compute agree on what the content is)
>
> - Cluster manager has an admin or head node and zero or more leader nodes
>
> - Leader nodes are provisioned in groups of 3 to use distributed
>   replica-3 volumes (although at least one customer has interest
>   in replica-5)
>
> - We configure a few different volumes for different use cases
>
> - We use Gluster NFS still because, over a year ago, Ganesha was not
>   working with our workload and we haven't had time to re-test and
>   engage with the community. No blame - we would also owe making sure
>   our settings are right.
>
> - We use CTDB for a measure of HA and IP alias management. We use this
>   instead of pacemaker to reduce complexity.
>
> - The volume use cases are:
>   * Image sharing for diskless compute nodes (sometimes 6,000 nodes)
> -> Normally squashFS image files for speed/efficiency exported NFS
> -> Expanded ("chrootable") traditional NFS trees for people who
>prefer that, but they don't scale as well and are slower to boot
> -> Squashfs images sit on a sharded volume while traditional gluster
>used for expanded tree.
>   * TFTP/HTTP for network boot/PXE including miniroot
> -> Spread across leaders too due so one node is not saturated with
>PXE/DHCP requests
> -> Miniroot is a "fatter initrd" that has our CM toolchain
>   * Logs/consoles
> -> For traditional logs and consoles (HCPM also uses
>elasticsearch/kafka/friends but we don't put that in gluster)
> -> Separate volume to have more non-cached friendly settings
>   * 4 total volumes used (one sharded, one heavily optimized for
> caching, one for ctdb lock, and one traditional for logging/etc)
>
> - Leader Setup
>   * Admin node installs the leaders like any other compute nodes
>   * A setup tool operates that configures gluster volumes and CTDB
>   * When ready, an admin/head node can be engaged with the leaders
>   * At that point, certain paths on the admin become gluster fuse mounts
> and bind mounts to gluster fuse mounts.
>
> - How images are deployed (squashfs mode)
>   * User creates an image using image creation tools that make a
> chrootable tree style image on the admin/head node
>   * mksquashfs will generate a squashfs image file on to a shared
> storage gluster mount
>   * Nodes will mount the filesystem with the squashfs images and then
> loop mount the squashfs as part of the boot process.
>
> - How are compute nodes tied to leaders
>   * 

[Gluster-users] brick process crashes on "Structure needs cleaning"

[Olaf Buitelaar]

Dear Users,

Somehow the brick processes seem to crash on xfs filesystem error's. It
seems it depends on the way the gluster process is started. Also gluster
sends on this occurrence a message to the console, informing the process
will go down, however it doesn't really seem to go down;

M [MSGID: 113075] [posix-helpers.c:2185:posix_health_check_thread_proc]
0-ovirt-engine-posix: health-check failed, going down
 M [MSGID: 113075] [posix-helpers.c:2203:posix_health_check_thread_proc]
0-ovirt-engine-posix: still alive! -> SIGTERM

in the brick log a message like this is logged;
[posix-helpers.c:2111:posix_fs_health_check] 0-ovirt-data-posix:
aio_read_cmp_buf() on
/data5/gfs/bricks/brick1/ovirt-data/.glusterfs/health_check returned ret is
-1 error is Structure needs cleaning

or like this;
 W [MSGID: 113075] [posix-helpers.c:2111:posix_fs_health_check]
0-ovirt-mon-2-posix: aio_read_buf() on
/data0/gfs/bricks/bricka/ovirt-mon-2/.glusterfs/health_check returned ret
is -1 error is Success

when i check the actual file it just seems to contain a timestamp;
cat /data0/gfs/bricks/bricka/ovirt-mon-2/.glusterfs/health_check
2021-01-28 09:08:01⏎

And don't see errors in DMESG about having issues accessing it.

When i unmount the filesystem and run xfs_repair on it, no error's/issues
are reported. Also when i mount the filesystem again, it's reported as a
clean mount;
[2478552.169540] XFS (dm-23): Mounting V5 Filesystem
[2478552.180645] XFS (dm-23): Ending clean mount

When i kill the brick process and start with "gluser v start x force" the
issue seems much more unlikely to occur, but when started from a fresh
reboot, or when killing the process and let it being started by glusterd
(e.g. service glusterd start) the error seems to arise after a couple of
minutes.

I am making use of LVM cache (in write through mode), maybe that's related.
Also the disks it self are backed by a hardware raid controller and i did
inspect all disks for SMART errors.

Does anybody has experience with this, and a clue on what might causing
this?

Thanks Olaf




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Tiered storage feature

[Olaf Buitelaar]

Dear Schlick,

It's indeed dropped, I think it's in favor of more native options. The go
to replacement is lvm-cache (
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/logical_volume_manager_administration/lvm_cache_volume_creation
).
This places just an amount of SSD storage as a cache layer in-front of your
HDDs, using device mapping.
If you're into ansible, you could use these play's;
https://github.com/gluster/gluster-ansible-infra
it seems to operate pretty stable.

Best

Olaf




Op di 26 jan. 2021 om 20:34 schreef Schlick Rupert :

> Dear List,
>
>
>
> could anyone enlighten me what happened to the tiered storage feature?
>
>
>
> When finding references to it online, it seemed to be the perfect solution
> for my problem, but further digging shows it was removed in gluster 6.
>
>
>
> What digging did not show is why it was removed and if there are any
> suggestions to solve the issue of having fast and slow storage with
> glusterfs.
>
>
>
> I’m looking for a distributed storage for a small cluster (for now 3
> nodes, 4 fast GPUs each). Space on NVMe SSDs is limited, the nodes are
> interconnected with 100Gb Infiniband.
>
> Gluster seemed to be the perfect solution to mirror between the nodes,
> until I realized that tiered volumes are not supported anymore.
>
>
>
> The removal baffles me a bit, since alternatives available under
> open-source licenses like MooseFS, BeeGFS, CEPH, Lustre have also solutions
> for hierarchical storage management in some form and the use case seems to
> be a common one.
>
>
>
> Best
>
>
>
> Rupert
>
>
> 
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Add more capacity to existing gulsterfs replicate

[Olaf Buitelaar]

Hi,

please see;
https://docs.gluster.org/en/latest/Administrator-Guide/Managing-Volumes/

Gluster offers online expansion of the volume you can add bricks and/or
nodes without taking mariadb offline if you want.

just use the; gluster volume add-brick [vol] [bricks] (bricks must be added
according your replication count, in your case 2)
or use; gluster volume replace-brick [vol] [brick] to replace a single brick

Best Olaf

Op do 24 dec. 2020 om 23:35 schreef Bambang Sumitra <
bambang.sumi...@gmail.com>:

> Hi,
>
> I have small datawarehouse using mariadb columnstore setup as 2 instance
> server with glusterfs as storage backend, i follow setup from this guide
> https://mariadb.com/kb/en/installing-and-configuring-a-multi-server-columnstore-system-11x
>
> Now our server almost running out of space, i have attached new disk to
> both server and plan to add more storage capacity to the glusterfs volume
> and remove old brick (if possible)
>
> Questions :
> Can i do this step :
> 1. stop mariadb
> 2. stop glusterfs
> 3. mount new disk to /mnt/newdisk
> 4. copy data from old brick to to /mnt/newdisk
> 5. unmount brick
> 6. mount new disk to /usr/local/mariadb/columnstore/gluster (existing
> glusterfs mount)
>
> or is there any easy and better way to add capacity? i dont mine to keep
> or remove old brick
>
> Thank you,
>
> *command output from host 10.1.1.60*
> root@mDWDB01:~# gluster volume status
> Status of volume: dbroot1
> Gluster process TCP Port  RDMA Port  Online
>  Pid
>
> --
> Brick 10.1.1.60:/usr/local/mariadb/columnst
> ore/gluster/brick1  49152 0  Y
> 1541
> Brick 10.1.1.61:/usr/local/mariadb/columnst
> ore/gluster/brick1  49152 0  Y
> 1499
> Self-heal Daemon on localhost   N/A   N/AY
> 1425
> Self-heal Daemon on 10.1.1.61   N/A   N/AY
> 1367
>
> Task Status of Volume dbroot1
>
> --
> There are no active volume tasks
>
> Status of volume: dbroot2
> Gluster process TCP Port  RDMA Port  Online
>  Pid
>
> --
> Brick 10.1.1.60:/usr/local/mariadb/columnst
> ore/gluster/brick2  49153 0  Y
> 1550
> Brick 10.1.1.61:/usr/local/mariadb/columnst
> ore/gluster/brick2  49153 0  Y
> 1508
> Self-heal Daemon on localhost   N/A   N/AY
> 1425
> Self-heal Daemon on 10.1.1.61   N/A   N/AY
> 1367
>
> Task Status of Volume dbroot2
>
> --
> There are no active volume tasks
>
> mis@mDWDB01:~$ sudo gluster volume info
> [sudo] password for mis:
>
> Volume Name: dbroot1
> Type: Replicate
> Volume ID: 22814201-3fae-4904-b0b7-d6e1716365ec
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: 10.1.1.60:/usr/local/mariadb/columnstore/gluster/brick1
> Brick2: 10.1.1.61:/usr/local/mariadb/columnstore/gluster/brick1
> Options Reconfigured:
> performance.client-io-threads: off
> nfs.disable: on
> transport.address-family: inet
>
> Volume Name: dbroot2
> Type: Replicate
> Volume ID: 6443b073-754d-440b-89e9-49c085114f46
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: 10.1.1.60:/usr/local/mariadb/columnstore/gluster/brick2
> Brick2: 10.1.1.61:/usr/local/mariadb/columnstore/gluster/brick2
> Options Reconfigured:
> performance.client-io-threads: off
> nfs.disable: on
> transport.address-family: inet
>
> mis@mDWDB01:~$ mount |grep column
> /dev/sdb1 on /usr/local/mariadb/columnstore/gluster type xfs
> (rw,relatime,attr2,inode64,noquota)
> 10.1.1.60:/dbroot2 on /usr/local/mariadb/columnstore/data2 type
> fuse.glusterfs
> (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
>
>
> *command output from host 10.1.1.61*
> mis@mDWUM01:~$ sudo gluster volume info
> [sudo] password for mis:
>
> Volume Name: dbroot1
> Type: Replicate
> Volume ID: 22814201-3fae-4904-b0b7-d6e1716365ec
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: 10.1.1.60:/usr/local/mariadb/columnstore/gluster/brick1
> Brick2: 10.1.1.61:/usr/local/mariadb/columnstore/gluster/brick1
> Options Reconfigured:
> performance.client-io-threads: off
> nfs.disable: on
> transport.address-family: inet
>
> Volume Name: dbroot2
> Type: Replicate
> Volume ID: 6443b073-754d-440b-89e9-49c085114f46
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: 10.1.1.60:/usr/local/mariadb/columnstore/gluster/brick2
> Brick2: 10.1.1.61:/usr/local/mariadb

Re: [Gluster-users] Gluster 7 or 8?

[Olaf Buitelaar]

It is in their release notes;
https://docs.gluster.org/en/latest/release-notes/7.9/

Ok, we've usually good experiences with inplace update's. But taking the
safe route is always a good idea!

Op do 17 dec. 2020 om 19:49 schreef WK :

>
> On 12/17/2020 10:13 AM, Olaf Buitelaar wrote:
> > Hi WK,
> >
> > i believe gluster 7 just received it's
> > latest maintenance patch @version 7.9. so that's a dead end from there
> on.
> >
> Hmm, ok, the website should reflect that then.
>
>
> > I'm not sure if you can skip a whole version going straight to 8.
>
> well it would be a forklift upgrade.
>
> We build up new boxes and migrate over the data.
>
> The one area we have been burned before on Gluster is doing in place
> upgrades. Since we don't have all our eggs in one basket we can migrate
> one cluster at a time that way.
>
>
>




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster 7 or 8?

[Olaf Buitelaar]

Hi WK,

i believe gluster 7 just received it's latest maintenance patch @version
7.9. so that's a dead end from there on.

I'm not sure if you can skip a whole version going straight to 8.

Unfortunately I cannot answer your other questions. Hopefully somebody else
could.

Best Olaf


Op do 17 dec. 2020 om 18:23 schreef WK :

> Greetings:
>
> We have several  replica 2 + Arb clusters using 6.x.
>
> They have worked wonderfully with the only glitch being traced back to a
> faulty network card.
>
> We are considering doing a fork lift upgrade for some them as we have
> some down time during the North American holidays and some available kit
> that is newer/better.
>
> According to the Website, Gluster 7 is the 'maintained version' and G8
> is the current.
>
> Any thoughts about being conservative and just going to G7 or should we
> go ahead with G8 since we have fairly simple setups?
>
> Also what is the status with Thin Arbiter?
>
> Is it considered Safe?
>
> How much of an improvement of people seen with it?
>
> For Us, Classic Arbiter works great, so we would need a reaon to go with
> Thin, but speed would be one of them if we are giving up safety.
>
> -wk
>
>
> 
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] possible memory leak in client/fuse mount

[Olaf Buitelaar]

Hi Ravi,

I'm not sure what was bothering the SHD, but i tried again to restart the
glusterd/glusterfsd's and now the SHD came up. I'm sorry i don't have any
more details about what could have caused the issue.
If you want me to check/search for certain log messages on what could have
caused it, please let me know.

Thanks for your assistance.

Best Olaf

Op do 26 nov. 2020 om 11:53 schreef Ravishankar N :

>
> On 26/11/20 4:00 pm, Olaf Buitelaar wrote:
>
> Hi Ravi,
>
> I could try that, but i can only try a setup on VM's, and will not be able
> to setup an environment like our production environment.
> Which runs on physical machines, and has actual production load etc. So
> the 2 setups would be quite different.
> Personally i think it would be best debug the actual machines instead of
> trying to reproduce it. Since the reproduction of the issue on the
> physical machines is just swap the repositories and upgrade the packages.
> Let me know what you think?
>
> Physical machines or VMs - anything is fine. The only thing is I cannot
> guarantee quick responses , so if it is a production machine, it will be an
> issue for you. So any set up you can use for experimenting is fine. You
> don't need any clients for the testing. Just create a 1x2  replica volume
> using 2 nodes and start it. Then upgrade one node and see if shd and bricks
> come up on that node.
>
> -Ravi
>
>
> Thanks Olaf
>
> Op do 26 nov. 2020 om 02:43 schreef Ravishankar N  >:
>
>>
>> On 25/11/20 7:17 pm, Olaf Buitelaar wrote:
>>
>> Hi Ravi,
>>
>> Thanks for checking. Unfortunately this is our production system, what
>> i've done is simple change the yum repo from gluter-6 to
>> http://mirror.centos.org/centos/$releasever/storage/$basearch/gluster-7/.
>> Did a yum upgrade. And did restart the glusterd process several times, i've
>> also tried rebooting the machine. And didn't touch the op-version yet,
>> which is still at (6), usually i only do this when all nodes are
>> upgraded, and are running stable.
>> We're running multiple volumes with different configurations, but for
>> none of the volumes the shd starts on the upgraded nodes.
>> Is there anything further i could check/do to get to the bottom of this?
>>
>> Hi Olaf, like I said, would it be possible to create a test setup to see
>> if you can recreate it?
>> Regards,
>> Ravi
>>
>>
>> Thanks Olaf
>>
>> Op wo 25 nov. 2020 om 14:14 schreef Ravishankar N > >:
>>
>>>
>>> On 25/11/20 5:50 pm, Olaf Buitelaar wrote:
>>>
>>> Hi Ashish,
>>>
>>> Thank you for looking into this. I indeed also suspect it has something
>>> todo with the 7.X client, because on the 6.X clients the issue doesn't
>>> really seem to occur.
>>> I would love to update everything to 7.X, But since the self-heal
>>> daemons (
>>> https://lists.gluster.org/pipermail/gluster-users/2020-November/038917.html)
>>> won't start, i halted the full upgrade.
>>>
>>> Olaf, based on your email. I did try to upgrade a 1 node of a 3-node
>>> replica 3 setup from 6.10 to 7.8 on my test VMs and I found that the
>>> self-heal daemon (and the bricks) came online after I restarted glusterd
>>> post-upgrade on that node. (I did not touch the op-version), and I did not
>>> spend time on it further.  So I don't think the problem is related to the
>>> shd mux changes I referred to. But if you have a test setup where you can
>>> reproduce this, please raise a github issue with the details.
>>> Thanks,
>>> Ravi
>>>
>>> Hopefully that issue will be addressed in the upcoming release. Once
>>> i've everything running on the same version i'll check if the issue still
>>> occurs and reach out, if that's the case.
>>>
>>> Thanks Olaf
>>>
>>> Op wo 25 nov. 2020 om 10:42 schreef Ashish Pandey :
>>>
>>>>
>>>> Hi,
>>>>
>>>> I checked the statedump and found some very high memory allocations.
>>>> grep -rwn "num_allocs" glusterdump.17317.dump.1605* | cut -d'=' -f2 |
>>>> sort
>>>>
>>>> 30003616
>>>> 30003616
>>>> 3305
>>>> 3305
>>>> 36960008
>>>> 36960008
>>>> 38029944
>>>> 38029944
>>>> 38450472
>>>> 38450472
>>>> 39566824
>>>> 39566824
>>>> 4
>>>> I did check the lines on statedump a

Re: [Gluster-users] possible memory leak in client/fuse mount

[Olaf Buitelaar]

Hi Ravi,

I could try that, but i can only try a setup on VM's, and will not be able
to setup an environment like our production environment.
Which runs on physical machines, and has actual production load etc. So the
2 setups would be quite different.
Personally i think it would be best debug the actual machines instead of
trying to reproduce it. Since the reproduction of the issue on the
physical machines is just swap the repositories and upgrade the packages.
Let me know what you think?

Thanks Olaf

Op do 26 nov. 2020 om 02:43 schreef Ravishankar N :

>
> On 25/11/20 7:17 pm, Olaf Buitelaar wrote:
>
> Hi Ravi,
>
> Thanks for checking. Unfortunately this is our production system, what
> i've done is simple change the yum repo from gluter-6 to
> http://mirror.centos.org/centos/$releasever/storage/$basearch/gluster-7/.
> Did a yum upgrade. And did restart the glusterd process several times, i've
> also tried rebooting the machine. And didn't touch the op-version yet,
> which is still at (6), usually i only do this when all nodes are
> upgraded, and are running stable.
> We're running multiple volumes with different configurations, but for none
> of the volumes the shd starts on the upgraded nodes.
> Is there anything further i could check/do to get to the bottom of this?
>
> Hi Olaf, like I said, would it be possible to create a test setup to see
> if you can recreate it?
> Regards,
> Ravi
>
>
> Thanks Olaf
>
> Op wo 25 nov. 2020 om 14:14 schreef Ravishankar N  >:
>
>>
>> On 25/11/20 5:50 pm, Olaf Buitelaar wrote:
>>
>> Hi Ashish,
>>
>> Thank you for looking into this. I indeed also suspect it has something
>> todo with the 7.X client, because on the 6.X clients the issue doesn't
>> really seem to occur.
>> I would love to update everything to 7.X, But since the self-heal daemons
>> (
>> https://lists.gluster.org/pipermail/gluster-users/2020-November/038917.html)
>> won't start, i halted the full upgrade.
>>
>> Olaf, based on your email. I did try to upgrade a 1 node of a 3-node
>> replica 3 setup from 6.10 to 7.8 on my test VMs and I found that the
>> self-heal daemon (and the bricks) came online after I restarted glusterd
>> post-upgrade on that node. (I did not touch the op-version), and I did not
>> spend time on it further.  So I don't think the problem is related to the
>> shd mux changes I referred to. But if you have a test setup where you can
>> reproduce this, please raise a github issue with the details.
>> Thanks,
>> Ravi
>>
>> Hopefully that issue will be addressed in the upcoming release. Once i've
>> everything running on the same version i'll check if the issue still occurs
>> and reach out, if that's the case.
>>
>> Thanks Olaf
>>
>> Op wo 25 nov. 2020 om 10:42 schreef Ashish Pandey :
>>
>>>
>>> Hi,
>>>
>>> I checked the statedump and found some very high memory allocations.
>>> grep -rwn "num_allocs" glusterdump.17317.dump.1605* | cut -d'=' -f2 |
>>> sort
>>>
>>> 30003616
>>> 30003616
>>> 3305
>>> 3305
>>> 36960008
>>> 36960008
>>> 38029944
>>> 38029944
>>> 38450472
>>> 38450472
>>> 39566824
>>> 39566824
>>> 4
>>> I did check the lines on statedump and it could be happening in
>>> protocol/clinet. However, I did not find anything suspicious in my quick
>>> code exploration.
>>> I would suggest to upgrade all the nodes on latest version and the start
>>> your work and see if there is any high usage of memory .
>>> That way it will also be easier to debug this issue.
>>>
>>> ---
>>> Ashish
>>>
>>> --
>>> *From: *"Olaf Buitelaar" 
>>> *To: *"gluster-users" 
>>> *Sent: *Thursday, November 19, 2020 10:28:57 PM
>>> *Subject: *[Gluster-users] possible memory leak in client/fuse mount
>>>
>>> Dear Gluster Users,
>>>
>>> I've a glusterfs process which consumes about all memory of the machine
>>> (~58GB);
>>>
>>> # ps -faxu|grep 17317
>>> root 17317  3.1 88.9 59695516 58479708 ?   Ssl  Oct31 839:36
>>> /usr/sbin/glusterfs --process-name fuse --volfile-server=10.201.0.1
>>> --volfile-server=10.201.0.8:10.201.0.5:10.201.0.6:10.201.0.7:10.201.0.9
>>> --volfile-id=/docker2 /mnt/docker2
>>>
>>> The gluster version on this machine is 7.8, but i'm currently running a
>

Re: [Gluster-users] possible memory leak in client/fuse mount

[Olaf Buitelaar]

Hi Ravi,

Thanks for checking. Unfortunately this is our production system, what i've
done is simple change the yum repo from gluter-6 to
http://mirror.centos.org/centos/$releasever/storage/$basearch/gluster-7/.
Did a yum upgrade. And did restart the glusterd process several times, i've
also tried rebooting the machine. And didn't touch the op-version yet,
which is still at (6), usually i only do this when all nodes are
upgraded, and are running stable.
We're running multiple volumes with different configurations, but for none
of the volumes the shd starts on the upgraded nodes.
Is there anything further i could check/do to get to the bottom of this?

Thanks Olaf

Op wo 25 nov. 2020 om 14:14 schreef Ravishankar N :

>
> On 25/11/20 5:50 pm, Olaf Buitelaar wrote:
>
> Hi Ashish,
>
> Thank you for looking into this. I indeed also suspect it has something
> todo with the 7.X client, because on the 6.X clients the issue doesn't
> really seem to occur.
> I would love to update everything to 7.X, But since the self-heal daemons (
> https://lists.gluster.org/pipermail/gluster-users/2020-November/038917.html)
> won't start, i halted the full upgrade.
>
> Olaf, based on your email. I did try to upgrade a 1 node of a 3-node
> replica 3 setup from 6.10 to 7.8 on my test VMs and I found that the
> self-heal daemon (and the bricks) came online after I restarted glusterd
> post-upgrade on that node. (I did not touch the op-version), and I did not
> spend time on it further.  So I don't think the problem is related to the
> shd mux changes I referred to. But if you have a test setup where you can
> reproduce this, please raise a github issue with the details.
> Thanks,
> Ravi
>
> Hopefully that issue will be addressed in the upcoming release. Once i've
> everything running on the same version i'll check if the issue still occurs
> and reach out, if that's the case.
>
> Thanks Olaf
>
> Op wo 25 nov. 2020 om 10:42 schreef Ashish Pandey :
>
>>
>> Hi,
>>
>> I checked the statedump and found some very high memory allocations.
>> grep -rwn "num_allocs" glusterdump.17317.dump.1605* | cut -d'=' -f2 | sort
>>
>> 30003616
>> 30003616
>> 3305
>> 3305
>> 36960008
>> 36960008
>> 38029944
>> 38029944
>> 38450472
>> 38450472
>> 39566824
>> 39566824
>> 4
>> I did check the lines on statedump and it could be happening in
>> protocol/clinet. However, I did not find anything suspicious in my quick
>> code exploration.
>> I would suggest to upgrade all the nodes on latest version and the start
>> your work and see if there is any high usage of memory .
>> That way it will also be easier to debug this issue.
>>
>> ---
>> Ashish
>>
>> --
>> *From: *"Olaf Buitelaar" 
>> *To: *"gluster-users" 
>> *Sent: *Thursday, November 19, 2020 10:28:57 PM
>> *Subject: *[Gluster-users] possible memory leak in client/fuse mount
>>
>> Dear Gluster Users,
>>
>> I've a glusterfs process which consumes about all memory of the machine
>> (~58GB);
>>
>> # ps -faxu|grep 17317
>> root 17317  3.1 88.9 59695516 58479708 ?   Ssl  Oct31 839:36
>> /usr/sbin/glusterfs --process-name fuse --volfile-server=10.201.0.1
>> --volfile-server=10.201.0.8:10.201.0.5:10.201.0.6:10.201.0.7:10.201.0.9
>> --volfile-id=/docker2 /mnt/docker2
>>
>> The gluster version on this machine is 7.8, but i'm currently running a
>> mixed cluster of 6.10 and 7.8, while awaiting to proceed to upgrade for the
>> issue mentioned earlier with the self-heal daemon.
>>
>> The affected volume info looks like;
>>
>> # gluster v info docker2
>>
>> Volume Name: docker2
>> Type: Distributed-Replicate
>> Volume ID: 4e0670a0-3d00-4360-98bd-3da844cedae5
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 3 x (2 + 1) = 9
>> Transport-type: tcp
>> Bricks:
>> Brick1: 10.201.0.5:/data0/gfs/bricks/brick1/docker2
>> Brick2: 10.201.0.9:/data0/gfs/bricks/brick1/docker2
>> Brick3: 10.201.0.3:/data0/gfs/bricks/bricka/docker2 (arbiter)
>> Brick4: 10.201.0.6:/data0/gfs/bricks/brick1/docker2
>> Brick5: 10.201.0.7:/data0/gfs/bricks/brick1/docker2
>> Brick6: 10.201.0.4:/data0/gfs/bricks/bricka/docker2 (arbiter)
>> Brick7: 10.201.0.1:/data0/gfs/bricks/brick1/docker2
>> Brick8: 10.201.0.8:/data0/gfs/bricks/brick1/docker2
>> Brick9: 10.201.0.2:/data0/gfs/bricks/bricka/docker2 (arbiter)
>> Options Reconfigured:
>> performance.cache-size: 128MB
>> transport.addre

Re: [Gluster-users] possible memory leak in client/fuse mount

[Olaf Buitelaar]

Hi Ashish,

Thank you for looking into this. I indeed also suspect it has something
todo with the 7.X client, because on the 6.X clients the issue doesn't
really seem to occur.
I would love to update everything to 7.X, But since the self-heal daemons (
https://lists.gluster.org/pipermail/gluster-users/2020-November/038917.html)
won't start, i halted the full upgrade.
Hopefully that issue will be addressed in the upcoming release. Once i've
everything running on the same version i'll check if the issue still occurs
and reach out, if that's the case.

Thanks Olaf

Op wo 25 nov. 2020 om 10:42 schreef Ashish Pandey :

>
> Hi,
>
> I checked the statedump and found some very high memory allocations.
> grep -rwn "num_allocs" glusterdump.17317.dump.1605* | cut -d'=' -f2 | sort
>
> 30003616
> 30003616
> 3305
> 3305
> 36960008
> 36960008
> 38029944
> 38029944
> 38450472
> 38450472
> 39566824
> 39566824
> 4
> I did check the lines on statedump and it could be happening in
> protocol/clinet. However, I did not find anything suspicious in my quick
> code exploration.
> I would suggest to upgrade all the nodes on latest version and the start
> your work and see if there is any high usage of memory .
> That way it will also be easier to debug this issue.
>
> ---
> Ashish
>
> --
> *From: *"Olaf Buitelaar" 
> *To: *"gluster-users" 
> *Sent: *Thursday, November 19, 2020 10:28:57 PM
> *Subject: *[Gluster-users] possible memory leak in client/fuse mount
>
> Dear Gluster Users,
>
> I've a glusterfs process which consumes about all memory of the machine
> (~58GB);
>
> # ps -faxu|grep 17317
> root 17317  3.1 88.9 59695516 58479708 ?   Ssl  Oct31 839:36
> /usr/sbin/glusterfs --process-name fuse --volfile-server=10.201.0.1
> --volfile-server=10.201.0.8:10.201.0.5:10.201.0.6:10.201.0.7:10.201.0.9
> --volfile-id=/docker2 /mnt/docker2
>
> The gluster version on this machine is 7.8, but i'm currently running a
> mixed cluster of 6.10 and 7.8, while awaiting to proceed to upgrade for the
> issue mentioned earlier with the self-heal daemon.
>
> The affected volume info looks like;
>
> # gluster v info docker2
>
> Volume Name: docker2
> Type: Distributed-Replicate
> Volume ID: 4e0670a0-3d00-4360-98bd-3da844cedae5
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 3 x (2 + 1) = 9
> Transport-type: tcp
> Bricks:
> Brick1: 10.201.0.5:/data0/gfs/bricks/brick1/docker2
> Brick2: 10.201.0.9:/data0/gfs/bricks/brick1/docker2
> Brick3: 10.201.0.3:/data0/gfs/bricks/bricka/docker2 (arbiter)
> Brick4: 10.201.0.6:/data0/gfs/bricks/brick1/docker2
> Brick5: 10.201.0.7:/data0/gfs/bricks/brick1/docker2
> Brick6: 10.201.0.4:/data0/gfs/bricks/bricka/docker2 (arbiter)
> Brick7: 10.201.0.1:/data0/gfs/bricks/brick1/docker2
> Brick8: 10.201.0.8:/data0/gfs/bricks/brick1/docker2
> Brick9: 10.201.0.2:/data0/gfs/bricks/bricka/docker2 (arbiter)
> Options Reconfigured:
> performance.cache-size: 128MB
> transport.address-family: inet
> nfs.disable: on
> cluster.brick-multiplex: on
>
> The issue seems to be triggered by a program called zammad, which has an
> init process, which runs in a loop. on cycle it re-compiles the
> ruby-on-rails application.
>
> I've attached 2 statedumps, but as i only recently noticed the high memory
> usage, i believe both statedumps already show an escalated state of the
> glusterfs process. If it's needed to also have them from the beginning let
> me know. The dumps are taken about an hour apart.
> Also i've included the glusterd.log. I couldn't include mnt-docker2.log
> since it's too large, since it's littered with: " I [MSGID: 109066]
> [dht-rename.c:1951:dht_rename] 0-docker2-dht"
> However i've inspected the log and it contains no Error message's all are
> of the Info kind;
> which look like these;
> [2020-11-19 03:29:05.406766] I [glusterfsd-mgmt.c:2282:mgmt_getspec_cbk]
> 0-glusterfs: No change in volfile,continuing
> [2020-11-19 03:29:21.271886] I [socket.c:865:__socket_shutdown]
> 0-docker2-client-8: intentional socket shutdown(5)
> [2020-11-19 03:29:24.479738] I [socket.c:865:__socket_shutdown]
> 0-docker2-client-2: intentional socket shutdown(5)
> [2020-11-19 03:30:12.318146] I [socket.c:865:__socket_shutdown]
> 0-docker2-client-5: intentional socket shutdown(5)
> [2020-11-19 03:31:27.381720] I [socket.c:865:__socket_shutdown]
> 0-docker2-client-8: intentional socket shutdown(5)
> [2020-11-19 03:31:30.579630] I [socket.c:865:__socket_shutdown]
> 0-docker2-client-2: intentional socket shutdown(5)
> [2020-11-19 03:32:18.427364] I [socket.c:865:__socket_sh

Re: [Gluster-users] Self-Heal Daemon not starting after upgrade 6.10 to 7.8

[Olaf Buitelaar]

Hi Ravi,

I would like to avoid an offline upgrade, since it would disrupt quite some
services.
Is there anything further I can investigate or do?

Thanks Olaf

Op di 3 nov. 2020 om 12:17 schreef Ravishankar N :

>
> On 02/11/20 8:35 pm, Olaf Buitelaar wrote:
>
> Dear Gluster users,
>
> I'm trying to upgrade from gluster 6.10 to 7.8, i've currently tried this
> on 2 hosts, but on both the Self-Heal Daemon refuses to start.
> It could be because not all not are updated yet, but i'm a bit hesitant to
> continue, without the Self-Heal Daemon running.
> I'm not using quata's and i'm not seeing the peer reject messages, as
> other users reported in the mailing list.
> In fact gluster peer status and gluster pool list, display all nodes as
> connected.
> Also gluster v heal  info shows all nodes as Status: connected,
> however some report pending heals, which don't really seem to progress.
> Only in gluster v status  the 2 upgraded nodes report not running;
>
> Self-heal Daemon on localhost   N/A   N/AN
> N/A
> Self-heal Daemon on 10.32.9.5   N/A   N/AY
> 24022
> Self-heal Daemon on 10.201.0.4  N/A   N/AY
> 26704
> Self-heal Daemon on 10.201.0.3  N/A   N/AN
> N/A
> Self-heal Daemon on 10.32.9.4   N/A   N/AY
> 46294
> Self-heal Daemon on 10.32.9.3   N/A   N/AY
> 22194
> Self-heal Daemon on 10.201.0.9  N/A   N/AY
> 14902
> Self-heal Daemon on 10.201.0.6  N/A   N/AY
> 5358
> Self-heal Daemon on 10.201.0.5  N/A   N/AY
> 28073
> Self-heal Daemon on 10.201.0.7  N/A   N/AY
> 15385
> Self-heal Daemon on 10.201.0.1  N/A   N/AY
> 8917
> Self-heal Daemon on 10.201.0.12 N/A   N/AY
> 56796
> Self-heal Daemon on 10.201.0.8  N/A   N/AY
> 7990
> Self-heal Daemon on 10.201.0.11 N/A   N/AY
> 68223
> Self-heal Daemon on 10.201.0.10 N/A   N/AY
> 20828
>
> After the upgrade i see the
> file /var/lib/glusterd/vols//-shd.vol being created, which
> doesn't exists on the 6.10 nodes.
>
> in the logs i see these relevant messages;
> log: glusterd.log
> 0-management: Regenerating volfiles due to a max op-version mismatch or
> glusterd.upgrade file not being present, op_version retrieved:6, max
> op_version: 70200
>
> I think this is because of the shd multiplex (
> https://bugzilla.redhat.com/show_bug.cgi?id=1659708) added by Rafi.
>
> Rafi, is there any workaround which can work for rolling upgrades? Or
> should we just do an offline upgrade of all server nodes for the shd to
> come online?
>
> -Ravi
>
>
>
> [2020-10-31 21:48:42.256193] W [MSGID: 106204]
> [glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown
> key: tier-enabled
> [2020-10-31 21:48:42.256232] W [MSGID: 106204]
> [glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown
> key: brick-0
> [2020-10-31 21:48:42.256240] W [MSGID: 106204]
> [glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown
> key: brick-1
> [2020-10-31 21:48:42.256246] W [MSGID: 106204]
> [glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown
> key: brick-2
> [2020-10-31 21:48:42.256251] W [MSGID: 106204]
> [glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown
> key: brick-3
> [2020-10-31 21:48:42.256256] W [MSGID: 106204]
> [glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown
> key: brick-4
> [2020-10-31 21:48:42.256261] W [MSGID: 106204]
> [glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown
> key: brick-5
> [2020-10-31 21:48:42.256266] W [MSGID: 106204]
> [glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown
> key: brick-6
> [2020-10-31 21:48:42.256271] W [MSGID: 106204]
> [glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown
> key: brick-7
> [2020-10-31 21:48:42.256276] W [MSGID: 106204]
> [glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown
> key: brick-8
>
> [2020-10-31 21:51:36.049009] W [MSGID: 106617]
> [glusterd-svc-helper.c:948:glusterd_attach_svc] 0-glusterd: attach failed
> for glustershd(volume=backups)
> [2020-10-31 21:51:36.049055] E [MSGID: 106048]
> [glusterd-shd-svc.c:482:glusterd_shdsvc_start] 0-glusterd: Failed to attach
> shd svc(volume=backups) to pid=9262
> [2020-10-31 21:51:36.049138] E [MSGID: 106615]
> [glusterd-shd-svc.c:638:glusterd_s

[Gluster-users] Self-Heal Daemon not starting after upgrade 6.10 to 7.8

[Olaf Buitelaar]

Dear Gluster users,

I'm trying to upgrade from gluster 6.10 to 7.8, i've currently tried this
on 2 hosts, but on both the Self-Heal Daemon refuses to start.
It could be because not all not are updated yet, but i'm a bit hesitant to
continue, without the Self-Heal Daemon running.
I'm not using quata's and i'm not seeing the peer reject messages, as other
users reported in the mailing list.
In fact gluster peer status and gluster pool list, display all nodes as
connected.
Also gluster v heal  info shows all nodes as Status: connected,
however some report pending heals, which don't really seem to progress.
Only in gluster v status  the 2 upgraded nodes report not running;

Self-heal Daemon on localhost   N/A   N/AN   N/A
Self-heal Daemon on 10.32.9.5   N/A   N/AY
24022
Self-heal Daemon on 10.201.0.4  N/A   N/AY
26704
Self-heal Daemon on 10.201.0.3  N/A   N/AN   N/A
Self-heal Daemon on 10.32.9.4   N/A   N/AY
46294
Self-heal Daemon on 10.32.9.3   N/A   N/AY
22194
Self-heal Daemon on 10.201.0.9  N/A   N/AY
14902
Self-heal Daemon on 10.201.0.6  N/A   N/AY
5358
Self-heal Daemon on 10.201.0.5  N/A   N/AY
28073
Self-heal Daemon on 10.201.0.7  N/A   N/AY
15385
Self-heal Daemon on 10.201.0.1  N/A   N/AY
8917
Self-heal Daemon on 10.201.0.12 N/A   N/AY
56796
Self-heal Daemon on 10.201.0.8  N/A   N/AY
7990
Self-heal Daemon on 10.201.0.11 N/A   N/AY
68223
Self-heal Daemon on 10.201.0.10 N/A   N/AY
20828

After the upgrade i see the file /var/lib/glusterd/vols//-shd.vol
being created, which doesn't exists on the 6.10 nodes.

in the logs i see these relevant messages;
log: glusterd.log
0-management: Regenerating volfiles due to a max op-version mismatch or
glusterd.upgrade file not being present, op_version retrieved:6, max
op_version: 70200

[2020-10-31 21:48:42.256193] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown
key: tier-enabled
[2020-10-31 21:48:42.256232] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown
key: brick-0
[2020-10-31 21:48:42.256240] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown
key: brick-1
[2020-10-31 21:48:42.256246] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown
key: brick-2
[2020-10-31 21:48:42.256251] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown
key: brick-3
[2020-10-31 21:48:42.256256] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown
key: brick-4
[2020-10-31 21:48:42.256261] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown
key: brick-5
[2020-10-31 21:48:42.256266] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown
key: brick-6
[2020-10-31 21:48:42.256271] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown
key: brick-7
[2020-10-31 21:48:42.256276] W [MSGID: 106204]
[glusterd-store.c:3275:glusterd_store_update_volinfo] 0-management: Unknown
key: brick-8

[2020-10-31 21:51:36.049009] W [MSGID: 106617]
[glusterd-svc-helper.c:948:glusterd_attach_svc] 0-glusterd: attach failed
for glustershd(volume=backups)
[2020-10-31 21:51:36.049055] E [MSGID: 106048]
[glusterd-shd-svc.c:482:glusterd_shdsvc_start] 0-glusterd: Failed to attach
shd svc(volume=backups) to pid=9262
[2020-10-31 21:51:36.049138] E [MSGID: 106615]
[glusterd-shd-svc.c:638:glusterd_shdsvc_restart] 0-management: Couldn't
start shd for vol: backups on restart
[2020-10-31 21:51:36.183133] I [MSGID: 106618]
[glusterd-svc-helper.c:901:glusterd_attach_svc] 0-glusterd: adding svc
glustershd (volume=backups) to existing process with pid 9262

log: glustershd.log

[2020-10-31 21:49:55.976120] I [MSGID: 100041]
[glusterfsd-mgmt.c::glusterfs_handle_svc_attach] 0-glusterfs: received
attach request for volfile-id=shd/backups
[2020-10-31 21:49:55.976136] W [MSGID: 100042]
[glusterfsd-mgmt.c:1137:glusterfs_handle_svc_attach] 0-glusterfs: got
attach for shd/backups but no active graph [Invalid argument]

So i suspect something in the logic for the self-heal daemon has
changed, since it has the new *.vol configuration for the shd. Question is,
is this just a transitional state, till all nodes are upgraded. And thus
safe to continue the update. Or is this something that should be fixed, and
if so, any clues how?

Thanks Olaf




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing 

Re: [Gluster-users] Change arbiter brick mount point

[Olaf Buitelaar]

Hi Andreas,

Only step 5 should be enough. Remounting would only be required, when all
the bricks change their destination.

Best Olaf

Op di 13 okt. 2020 om 11:20 schreef Andreas Kirbach :

> Hi,
>
> I'd like to change the mount point of an arbiter brick in a  1 x (2 + 1)
> = 3 replicated volume, but I can't seem to find any information on how
> to do this.
>
> My idea is
> 1) Stop the arbiter brick
> 2) Unmount the brick
> 3) Change the mount point for the brick
> 4) Mount the brick
> 5) gluster volume replace-brick  :/old/brick/path
> :/new/brick/path commit force
>
> Would that be the correct process?
>
> Kind regards
> Andreas
>
> --
>
> forumhome GmbH & Co. KG
> Bruchstr. 54a
> 67098 Bad Dürkheim
> Tel.: +49-6322-91 995-15
> Fax:  +49-6322-91 995-19
>
> Andreas Kirbach
> Technischer Leiter
>
> akirb...@forumhome.com
> www.forumhome.com
>
> Geschäftsführer: Carsten Grentrup
>
> Handelsregister: Ludwigshafen/ Rhein HRA 60968
> USt-Id: DE 285 908 418
>
> --
> --
> Forumhome sucht Praktikanten im Bereich Online-Marketing, Webdesign und
> Webbasierte Softwareentwicklung!
> Interesse? Informieren Sie sich unter
> http://de.forumhome.com/content/10-jobs und schicken Sie uns Ihre
> Bewerbungsunterlagen an j...@forumhome.com
> Wir freuen uns auf Sie!
> --
> --
>
> Diese E-Mail enthält vertrauliche oder rechtlich geschützte Informationen.
> Wenn Sie nicht der beabsichtigte Empfänger oder dessen Vertreter sind,
> informieren Sie bitte sofort den Absender und löschen Sie diese E-Mail.
> Das unbefugte Kopieren dieser E-Mail oder die unbefugte Weitergabe der
> enthaltenen Informationen ist nicht gestattet.
>
> The information contained in this message is confidential or protected
> by law. If you are not the intended recipient, please contact the sender
> and delete this message.
> Any unauthorised copying of this message or unauthorised distribution of
> the information contained herein is prohibited.
>
>
> --
> Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
> https://www.avast.com/antivirus
>
> 
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Glusterfs as databse store

[Olaf Buitelaar]

Hi Alex,

I've been running databases both directly and indirectly through qemu
images vms (managed by oVirt), and since the recent gluster versions (6+,
haven't tested 7-8) I'm generally happy with the stability. I'm running
mostly write intensive workloads.
For mariadb, any gluster volume seems to workfine, i've both running shared
and none-sharded volumes (using none-sharded for backup slave's to keep the
file's as a whole).
For postgresql it's required to enable the volume
option; performance.strict-o-direct: on.  but both shared and none-sharded
work in that case too.
none the less i would advise to run any database with strict-o-direct on.

Best Olaf


Op ma 12 okt. 2020 om 20:10 schreef Alex K :

>
>
> On Mon, Oct 12, 2020, 19:24 Strahil Nikolov  wrote:
>
>> Hi Alex,
>>
>> I can share that oVirt is using Gluster as a HCI solution and many people
>> are hosting DBs in their Virtual Machines.Yet, oVirt bypasses any file
>> system caches and uses Direct I/O in order to ensure consistency.
>>
>> As you will be using pacemaker, drbd is a viable solution that can be
>> controlled easily.
>>
> Thank you Strahil. I am using ovirt with glusterfs successfully for the
> last 5 years and I'm very happy about it. Though the vms gluster volume has
> sharding enabled by default and I suspect this is different if you run DB
> directly on top glusterfs. I assume there are optimizations one could apply
> at gluster volumes (use direct io?, small file workload optimizations, etc)
> and was hoping that there were success stories of DBs on top glusterfs.  I
> might go with drbd as the latest version is much more scalable and
> simplified.
>
>>
>> Best Regards,
>> Strahil Nikolov
>>
>>
>>
>>
>>
>>
>>
>> В понеделник, 12 октомври 2020 г., 12:12:18 Гринуич+3, Alex K <
>> rightkickt...@gmail.com> написа:
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Oct 12, 2020 at 9:47 AM Diego Zuccato 
>> wrote:
>> > Il 10/10/20 16:53, Alex K ha scritto:
>> >
>> >> Reading from the docs i see that this is not recommended?
>> > IIUC the risk of having partially-unsynced data is is too high.
>> > DB replication is not easy to configure because it's hard to do well,
>> > even active/passive.
>> > But I can tell you that a 3-node mariadb (galera) cluster is not hard to
>> > setup. Just follow one of the tutorials. It's nearly as easy as setting
>> > up a replica3 gluster volume :)
>> > And "guarantees" consinstency in the DB data.
>> I see. Since I will not have only mariadb, then I have to setup the same
>> replication for postgresql and later influxdb, which adds into the
>> complexity.
>> For cluster management I will be using pacemaker/corosync.
>>
>> Thanx for your feedback
>>
>> >
>> > --
>> > Diego Zuccato
>> > DIFA - Dip. di Fisica e Astronomia
>> > Servizi Informatici
>> > Alma Mater Studiorum - Università di Bologna
>> > V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
>> > tel.: +39 051 20 95786
>> >
>> 
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
> 
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>




Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Image File Owner change Situation. (root:root)

[Olaf Buitelaar]

Hi Robert,

there were serveral issues with ownership in ovirt, for example see;
https://bugzilla.redhat.com/show_bug.cgi?id=1666795
Maybe you're encountering these issues during the upgrade process. Also if
you're using gluster as backend storage, there might be some permission
issues in the 6.7 or below branches. not sure about the newest version's.

Best Olaf

Op vr 13 mrt. 2020 om 16:55 schreef Robert O'Kane :

> Hello @All,
>
> This has happened to us for the first time and only on One VM.
>
> I believe it happened with the switch from Fuse to "LibgfApi" in Ovirt.
>
> I was using LibgfApiSupported=True on 4.2.8 . I upgraded to 4.3.8 and did
> NOT restart all of my VMs (30+)
>
> but only some VMs, no problem. Eventually I noticed that
> LibgfApiSupported=False  and reset it to True.
>
> The VM was Running WindowsServer2016 and we did a Cold-Reboot (not VM
> Restart). It did not come back online
> due to "Invalid Volume" which was eventually due to the IMAGEs (Boot and
> Data) being  user:group=root but
> not the meta,lease or directory. Nor any other VM have/had this problem
> but they were (if at all) completely
> stopped and restarted.
>
> I am looking for another VM that has not yet been restarted to test this
> theory. I thought this would be
> interesting for others looking into this problem.
>
> (I will ask my Colleague next week what he means with Cold-Reboot vs
> Warm-reboot)
>
> Stay Healthy.
>
> Cheers,
>
> Robert O'Kane
>
>
>
> --
> Robert O'Kane
> Systems Administrator
> Kunsthochschule für Medien Köln
> Peter-Welter-Platz 2
> 50676 Köln
>
> fon: +49(221)20189-223
> fax: +49(221)20189-49223
> 
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>




Community Meeting Calendar:

Schedule -
Every Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Errno 107] Transport endpoint is not connected

[Olaf Buitelaar]

>
> Hi Strahil,
>

in fact i'm running more bricks per host, around 12 bricks per host.
Nonetheless the feature doesn't seem to work really for me, since it's
starting a separate glusterfsd processes for each brick anyway, actually
after a reboot or restart of glusterd, multiple glusterfsd for the same
brick. It's probably because of this line; "allowing compatible bricks to
use the same process and port", but not sure about that, as that would mean
my bricks aren't compatible for the multiplex feature. Also I'm not sure if
12 is considered "many", but i can always test again with multiplexing off,
if that's considered to be better in this case. Only really stopping and
starting the volumes is not really an option, so ideally restarting them
one-by-one would also be sufficient.

Since i ever so often encounter qcow corruptions, i moved from having
multiple smaller ones in a raid0. as if i have to deal with the corruption,
it's easier to have it as simple as possible, and also having to consider a
raid0 layer to rebuild, only complicates the recovery. Also the live
migration doesn't really work for these VM's anyway since they have so much
RAM, and we don't have enough bandwidth (only 10Gbit) to transfer that fast
enough. It's actually faster to shut the VM down, and start it on another
node.

Ok running only another VM on the HostedEngine never caused me any trouble,
except with the recent restore, all vm's on this domain need to be down,
since it wanted to detach the domain. Can't remember that was always the
case, with prior restores (this was pre ansible deploy's).
Searching through https://lists.ovirt.org/ for the discussion you're
referring at seems no easy feat..so many threads about the HostedEngine..if
you would know a more detailed pointer, that would be great.

Thanks Olaf


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Errno 107] Transport endpoint is not connected

[Olaf Buitelaar]

Hi Strahil,

Thank you for your reply. I found the issue, the not connected errors seem
to appear from the ACL layer. and somehow it received a permission denied,
and this was translated to a not connected error.
while the file permission were listed as owner=vdsm and group=kvm, somehow
ACL saw this differently. I ran "chown -R vdsm.kvm
/rhev/data-center/mnt/glusterSD/10.201.0.11\:_ovirt-mon-2/" on the mount,
and suddenly things started working again.

I indeed have (or now had, since for the restore procedure i needed to
provide an empty domain) 1 other VM on the HostedEngine domain, this other
VM had other critical services like VPN. Since i see the HostedEngine
domain as one of the most reliable domains, i used it for critical
services.
All other VM's have their own domains.

I'm a bit surprised by your comment about brick multiplexing, i understood
this should actually improve performance, by sharing resources? Would you
have some extra information about this?

To answer your questions;

We currently have 15 physical hosts.

1) there are no pending heals
2) yes i'm able to connect to the ports
3) all peers report as connected
4) Actually i had a setup like this before, i had multiple smaller qcow
disks in a raid0 with LVM. But this did appeared not to be reliable, so i
switched to 1 single large disk. Would you know if there is some
documentation about this?
5) i'm running about the latest and greatest stable; 4.3.7.2-1.el7. Only
had trouble with the restore, because the cluster was still in
compatibility mode 4.2 and there were 2 older VM's which had snapshots from
prior versions, while the leaf was in compatibility level 4.2. note; the
backup was taken on the engine running 4.3.

Thanks Olaf



Op di 28 jan. 2020 om 17:31 schreef Strahil Nikolov :

> On January 27, 2020 11:49:08 PM GMT+02:00, Olaf Buitelaar <
> olaf.buitel...@gmail.com> wrote:
> >Dear Gluster users,
> >
> >i'm a bit at a los here, and any help would be appreciated.
> >
> >I've lost a couple, since the disks suffered from severe XFS error's
> >and of
> >virtual machines and some won't boot because they can't resolve the
> >size of
> >the image as reported by vdsm:
> >"VM kube-large-01 is down with error. Exit message: Unable to get
> >volume
> >size for domain 5f17d41f-d617-48b8-8881-a53460b02829 volume
> >f16492a6-2d0e-4657-88e3-a9f4d8e48e74."
> >
> >which is also reported by the vdsm-client;  vdsm-client Volume getSize
> >storagepoolID=59cd53a9-0003-02d7-00eb-01e3
> >storagedomainID=5f17d41f-d617-48b8-8881-a53460b02829
> >imageID=2f96fd46-1851-49c8-9f48-78bb50dbdffd
> >volumeID=f16492a6-2d0e-4657-88e3-a9f4d8e48e74
> >vdsm-client: Command Volume.getSize with args {'storagepoolID':
> >'59cd53a9-0003-02d7-00eb-01e3', 'storagedomainID':
> >'5f17d41f-d617-48b8-8881-a53460b02829', 'volumeID':
> >'f16492a6-2d0e-4657-88e3-a9f4d8e48e74', 'imageID':
> >'2f96fd46-1851-49c8-9f48-78bb50dbdffd'} failed:
> >(code=100, message=[Errno 107] Transport endpoint is not connected)
> >
> >with corresponding gluster mount log;
> >[2020-01-27 19:42:22.678793] W [MSGID: 114031]
> >[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk]
> >0-ovirt-data-client-14:
> >remote operation failed. Path:
>
> >/5f17d41f-d617-48b8-8881-a53460b02829/images/2f96fd46-1851-49c8-9f48-78bb50dbdffd/f16492a6-2d0e-4657-88e3-a9f4d8e48e74
> >(a19abb2f-8e7e-42f0-a3c1-dad1eeb3a851) [Permission denied]
> >[2020-01-27 19:42:22.678828] W [MSGID: 114031]
> >[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk]
> >0-ovirt-data-client-13:
> >remote operation failed. Path:
>
> >/5f17d41f-d617-48b8-8881-a53460b02829/images/2f96fd46-1851-49c8-9f48-78bb50dbdffd/f16492a6-2d0e-4657-88e3-a9f4d8e48e74
> >(a19abb2f-8e7e-42f0-a3c1-dad1eeb3a851) [Permission denied]
> >[2020-01-27 19:42:22.679806] W [MSGID: 114031]
> >[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk]
> >0-ovirt-data-client-14:
> >remote operation failed. Path: (null)
> >(----) [Permission denied]
> >[2020-01-27 19:42:22.679862] W [MSGID: 114031]
> >[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk]
> >0-ovirt-data-client-13:
> >remote operation failed. Path: (null)
> >(----) [Permission denied]
> >[2020-01-27 19:42:22.679981] W [MSGID: 108027]
> >[afr-common.c:2274:afr_attempt_readsubvol_set]
> >0-ovirt-data-replicate-3: no
> >read subvols for
>
> >/5f17d41f-d617-48b8-8881-a53460b02829/images/2f96fd46-1851-49c8-9f48-78bb50dbdffd/f16492a6-2d0e-4657-88e3-a9f4d8e48e74
> >[2020-

[Gluster-users] [Errno 107] Transport endpoint is not connected

[Olaf Buitelaar]

Dear Gluster users,

i'm a bit at a los here, and any help would be appreciated.

I've lost a couple, since the disks suffered from severe XFS error's and of
virtual machines and some won't boot because they can't resolve the size of
the image as reported by vdsm:
"VM kube-large-01 is down with error. Exit message: Unable to get volume
size for domain 5f17d41f-d617-48b8-8881-a53460b02829 volume
f16492a6-2d0e-4657-88e3-a9f4d8e48e74."

which is also reported by the vdsm-client;  vdsm-client Volume getSize
storagepoolID=59cd53a9-0003-02d7-00eb-01e3
storagedomainID=5f17d41f-d617-48b8-8881-a53460b02829
imageID=2f96fd46-1851-49c8-9f48-78bb50dbdffd
volumeID=f16492a6-2d0e-4657-88e3-a9f4d8e48e74
vdsm-client: Command Volume.getSize with args {'storagepoolID':
'59cd53a9-0003-02d7-00eb-01e3', 'storagedomainID':
'5f17d41f-d617-48b8-8881-a53460b02829', 'volumeID':
'f16492a6-2d0e-4657-88e3-a9f4d8e48e74', 'imageID':
'2f96fd46-1851-49c8-9f48-78bb50dbdffd'} failed:
(code=100, message=[Errno 107] Transport endpoint is not connected)

with corresponding gluster mount log;
[2020-01-27 19:42:22.678793] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-ovirt-data-client-14:
remote operation failed. Path:
/5f17d41f-d617-48b8-8881-a53460b02829/images/2f96fd46-1851-49c8-9f48-78bb50dbdffd/f16492a6-2d0e-4657-88e3-a9f4d8e48e74
(a19abb2f-8e7e-42f0-a3c1-dad1eeb3a851) [Permission denied]
[2020-01-27 19:42:22.678828] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-ovirt-data-client-13:
remote operation failed. Path:
/5f17d41f-d617-48b8-8881-a53460b02829/images/2f96fd46-1851-49c8-9f48-78bb50dbdffd/f16492a6-2d0e-4657-88e3-a9f4d8e48e74
(a19abb2f-8e7e-42f0-a3c1-dad1eeb3a851) [Permission denied]
[2020-01-27 19:42:22.679806] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-ovirt-data-client-14:
remote operation failed. Path: (null)
(----) [Permission denied]
[2020-01-27 19:42:22.679862] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-ovirt-data-client-13:
remote operation failed. Path: (null)
(----) [Permission denied]
[2020-01-27 19:42:22.679981] W [MSGID: 108027]
[afr-common.c:2274:afr_attempt_readsubvol_set] 0-ovirt-data-replicate-3: no
read subvols for
/5f17d41f-d617-48b8-8881-a53460b02829/images/2f96fd46-1851-49c8-9f48-78bb50dbdffd/f16492a6-2d0e-4657-88e3-a9f4d8e48e74
[2020-01-27 19:42:22.680606] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-ovirt-data-client-14:
remote operation failed. Path:
/5f17d41f-d617-48b8-8881-a53460b02829/images/2f96fd46-1851-49c8-9f48-78bb50dbdffd/f16492a6-2d0e-4657-88e3-a9f4d8e48e74
(----) [Permission denied]
[2020-01-27 19:42:22.680622] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-ovirt-data-client-13:
remote operation failed. Path:
/5f17d41f-d617-48b8-8881-a53460b02829/images/2f96fd46-1851-49c8-9f48-78bb50dbdffd/f16492a6-2d0e-4657-88e3-a9f4d8e48e74
(----) [Permission denied]
[2020-01-27 19:42:22.681742] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-ovirt-data-client-13:
remote operation failed. Path: (null)
(----) [Permission denied]
[2020-01-27 19:42:22.681871] W [MSGID: 108027]
[afr-common.c:2274:afr_attempt_readsubvol_set] 0-ovirt-data-replicate-3: no
read subvols for
/5f17d41f-d617-48b8-8881-a53460b02829/images/2f96fd46-1851-49c8-9f48-78bb50dbdffd/f16492a6-2d0e-4657-88e3-a9f4d8e48e74
[2020-01-27 19:42:22.682344] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-ovirt-data-client-14:
remote operation failed. Path:
/5f17d41f-d617-48b8-8881-a53460b02829/images/2f96fd46-1851-49c8-9f48-78bb50dbdffd/f16492a6-2d0e-4657-88e3-a9f4d8e48e74
(----) [Permission denied]
The message "W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-ovirt-data-client-14:
remote operation failed. Path: (null)
(----) [Permission denied]" repeated 2
times between [2020-01-27 19:42:22.679806] and [2020-01-27 19:42:22.683308]
[2020-01-27 19:42:22.683327] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-ovirt-data-client-13:
remote operation failed. Path: (null)
(----) [Permission denied]
[2020-01-27 19:42:22.683438] W [MSGID: 108027]
[afr-common.c:2274:afr_attempt_readsubvol_set] 0-ovirt-data-replicate-3: no
read subvols for
/5f17d41f-d617-48b8-8881-a53460b02829/images/2f96fd46-1851-49c8-9f48-78bb50dbdffd/f16492a6-2d0e-4657-88e3-a9f4d8e48e74
[2020-01-27 19:42:22.683495] I [dict.c:560:dict_get]
(-->/usr/lib64/glusterfs/6.7/xlator/cluster/replicate.so(+0x6e92b)
[0x7fdeb92b]
-->/usr/lib64/glusterfs/6.7/xlator/cluster/distribute.so(+0x45c78)
[0x7fb08c78] -->/lib64/libglusterfs.so.0(dict_get+0x94)
[0x7faab36ac254] ) 0-dict: !this || key=trusted.glusterfs.dht.mds [Inva

Re: [Gluster-users] Stale File Handle Errors During Heavy Writes

[Olaf Buitelaar]

Yeah..zo the right procedure should be to setup a new volume without
sharding and copy everything over.

On Thu, 28 Nov 2019, 06:45 Strahil,  wrote:

> I have already tried disabling sharding on a test oVirt volume... The
> results were devastating for the app, so please do not disable sharding.
>
> Best Regards,
> Strahil Nikolov
> On Nov 27, 2019 20:55, Olaf Buitelaar  wrote:
>
> Hi Tim,
>
> That issue also seems to point to a stale file. Best i suppose is first to
> determine if you indeed have the same shard on different sub-volumes, where
> on one of the sub-volumes the file size is 0KB and has the stick bit set.
> if so we suffer from the same issue, and you can clean those files up, so
> the `rm` command should start working again.
> Essentially you should consider the volume unhealty until you have
> resolved the stale files, before you can continue file operations.
> Remounting the client shouldn't make a difference since the issue is at
> brick/sub-volume level.
>
> the last comment i received from Krutika;
> "I haven't had the chance to look into the attachments yet. I got another
> customer case on me.
> But from the description, it seems like the linkto file (the one with a
> 'T') and the original file don't have the same gfid?
> It's not wrong for those 'T' files to exist. But they're supposed to have
> the same gfid.
> This is something that needs DHT team's attention.
> Do you mind raising a bug in bugzilla.redhat.com against glusterfs and
> component 'distribute' or 'DHT'?"
>
>
> For me replicating it was easiest with running xfs_fsr (which is very
> write intensive in fragmented volumes) from within a VM, but it could
> happen with a simple yum install.. docker run (with new image)..general
> test with dd, mkfs.xfs or just random, which was the normal case. But i've
> to say my workload is mostly write intensive, like yours.
>
> Sharding in general is a nice feature, it allows your files to be broken
> up into peaces, which is also it's biggest danger..if anything goes
> haywire, it's currently practically impossible to stitch all those peaces
> together again, since no tool for this seems to exists..which is the nice
> thing about none-sharded volumes, they are just files..but if you really
> wanted i suppose it could be done. But would be very painful..i suppose.
> With the files being in shard's it allows  for much more equal
> distribution. Also heals seem to resolve much quicker.
> I'm also running none sharded volumes, with files of 100GB+ and those
> heals can take significantly longer. And those none sharded volumes i also
> sometime's have issues with..however not remembering any stale files.
> But if you don't need it you might be better of disabling it. However i
> believe you're never allowed to turn of sharding on a sharded volumes since
> it will corrupt your data.
>
> Best Olaf
>
> Op wo 27 nov. 2019 om 19:19 schreef Timothy Orme :
>
> Hi Olaf,
>
> Thanks so much for sharing this, it's hugely helpful, if only to make me
> feel less like I'm going crazy.  I'll see if theres anything I can add to
> the bug report.  I'm trying to develop a test to reproduce the issue now.
>
> We're running this in a sort of interactive HPC environment, so these
> error are a bit hard for us to systematically handle, and they have a
> tendency to be quite disruptive to folks work.
>
> I've run into other issues with sharding as well, such as this:
> <https://lists.gluster.org/pipermail/gluster-users/2019-October/037241.html>
> https://lists.gluster.org/pipermail/gluster-users/2019-October/037241.html
>
> I'm wondering then, if maybe sharding isn't quite stable yet and it's more
> sensible for me to just disable this feature for now?  I'm not quite sure
> what other implications that might have but so far all the issues I've run
> into so far as a new gluster user seem like they're related to shards.
>
> Thanks,
> Tim
> --
> *From:* Olaf Buitelaar 
> *Sent:* Wednesday, November 27, 2019 9:50 AM
> *To:* Timothy Orme 
> *Cc:* gluster-users 
> *Subject:* [EXTERNAL] Re: [Gluster-users] Stale File Handle Errors During
> Heavy Writes
>
> Hi Tim,
>
> i've been suffering from this also for a long time, not sure if it's exact
> the same situation since your setup is different. But it seems similar.
> i've filed this bug report;
> https://bugzilla.redhat.com/show_bug.cgi?id=1732961
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.redhat.com_show-5Fbug.cgi-3Fid-3D1732961&d=DwMFaQ&c=k

Re: [Gluster-users] Stale File Handle Errors During Heavy Writes

[Olaf Buitelaar]

Hi Tim,

That issue also seems to point to a stale file. Best i suppose is first to
determine if you indeed have the same shard on different sub-volumes, where
on one of the sub-volumes the file size is 0KB and has the stick bit set.
if so we suffer from the same issue, and you can clean those files up, so
the `rm` command should start working again.
Essentially you should consider the volume unhealty until you have resolved
the stale files, before you can continue file operations. Remounting the
client shouldn't make a difference since the issue is at brick/sub-volume
level.

the last comment i received from Krutika;
"I haven't had the chance to look into the attachments yet. I got another
customer case on me.
But from the description, it seems like the linkto file (the one with a
'T') and the original file don't have the same gfid?
It's not wrong for those 'T' files to exist. But they're supposed to have
the same gfid.
This is something that needs DHT team's attention.
Do you mind raising a bug in bugzilla.redhat.com against glusterfs and
component 'distribute' or 'DHT'?"


For me replicating it was easiest with running xfs_fsr (which is very write
intensive in fragmented volumes) from within a VM, but it could happen with
a simple yum install.. docker run (with new image)..general test with dd,
mkfs.xfs or just random, which was the normal case. But i've to say my
workload is mostly write intensive, like yours.

Sharding in general is a nice feature, it allows your files to be broken up
into peaces, which is also it's biggest danger..if anything goes haywire,
it's currently practically impossible to stitch all those peaces together
again, since no tool for this seems to exists..which is the nice thing
about none-sharded volumes, they are just files..but if you really wanted i
suppose it could be done. But would be very painful..i suppose.
With the files being in shard's it allows  for much more equal
distribution. Also heals seem to resolve much quicker.
I'm also running none sharded volumes, with files of 100GB+ and those heals
can take significantly longer. And those none sharded volumes i also
sometime's have issues with..however not remembering any stale files.
But if you don't need it you might be better of disabling it. However i
believe you're never allowed to turn of sharding on a sharded volumes since
it will corrupt your data.

Best Olaf

Op wo 27 nov. 2019 om 19:19 schreef Timothy Orme :

> Hi Olaf,
>
> Thanks so much for sharing this, it's hugely helpful, if only to make me
> feel less like I'm going crazy.  I'll see if theres anything I can add to
> the bug report.  I'm trying to develop a test to reproduce the issue now.
>
> We're running this in a sort of interactive HPC environment, so these
> error are a bit hard for us to systematically handle, and they have a
> tendency to be quite disruptive to folks work.
>
> I've run into other issues with sharding as well, such as this:
> https://lists.gluster.org/pipermail/gluster-users/2019-October/037241.html
>
> I'm wondering then, if maybe sharding isn't quite stable yet and it's more
> sensible for me to just disable this feature for now?  I'm not quite sure
> what other implications that might have but so far all the issues I've run
> into so far as a new gluster user seem like they're related to shards.
>
> Thanks,
> Tim
> --
> *From:* Olaf Buitelaar 
> *Sent:* Wednesday, November 27, 2019 9:50 AM
> *To:* Timothy Orme 
> *Cc:* gluster-users 
> *Subject:* [EXTERNAL] Re: [Gluster-users] Stale File Handle Errors During
> Heavy Writes
>
> Hi Tim,
>
> i've been suffering from this also for a long time, not sure if it's exact
> the same situation since your setup is different. But it seems similar.
> i've filed this bug report;
> https://bugzilla.redhat.com/show_bug.cgi?id=1732961
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__bugzilla.redhat.com_show-5Fbug.cgi-3Fid-3D1732961&d=DwMFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=d0SJB4ihnau-Oyws6GEzcipkV9DfxCuMbgdSRgXeuxM&m=Nh3Ca9VCh4XnpEF6imXwTa2NUUglz-XZQhfG8-AyOVU&s=GbJiS8pLGORzLwdgt0ypnnQxQgRhrTHdGXEizatE9g0&e=>
>  for
> which you might be able to enrich.
> To solve the stale files i've made this bash script;
> https://gist.github.com/olafbuitelaar/ff6fe9d4ab39696d9ad6ca689cc89986
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_olafbuitelaar_ff6fe9d4ab39696d9ad6ca689cc89986&d=DwMFaQ&c=kKqjBR9KKWaWpMhASkPbOg&r=d0SJB4ihnau-Oyws6GEzcipkV9DfxCuMbgdSRgXeuxM&m=Nh3Ca9VCh4XnpEF6imXwTa2NUUglz-XZQhfG8-AyOVU&s=CvN0yMFI03czcHgzTeexTfP9h4woiAO_XVyn1umHR8g&e=>
>  (it's
> sli

Re: [Gluster-users] Stale File Handle Errors During Heavy Writes

[Olaf Buitelaar]

Hi Tim,

i've been suffering from this also for a long time, not sure if it's exact
the same situation since your setup is different. But it seems similar.
i've filed this bug report;
https://bugzilla.redhat.com/show_bug.cgi?id=1732961 for which you might be
able to enrich.
To solve the stale files i've made this bash script;
https://gist.github.com/olafbuitelaar/ff6fe9d4ab39696d9ad6ca689cc89986 (it's
slightly outdated) which you could use as inspiration, it basically removes
the stale files as suggested here;
https://lists.gluster.org/pipermail/gluster-users/2018-March/033785.html .
Please be aware the script won't work if you have  2 (or more) bricks of
the same volume on the same server (since it always takes the first path
found).
I invoke the script via ansible like this (since the script needs to run on
all bricks);
- hosts: host1,host2,host3
  tasks:
- shell: 'bash /root/clean-stale-gluster-fh.sh --host="{{ intif.ip |
first }}" --volume=ovirt-data --backup="/backup/stale/gfs/ovirt-data"
--shard="{{ item }}" --force'
  with_items:
- 1b0ba5c2-dd2b-45d0-9c4b-a39b2123cc13.14451

fortunately for me the issue seems to be disappeared, since it's now about
1 month i received one, while before it was about every other day.
The biggest thing the seemed to resolve it was more disk space. while
before there was also plenty the gluster volume was at about 85% full, and
the individual disk had about 20-30% free of 8TB disk array, but had
servers in the mix with smaller disk array's but with similar available
space (in percents). i'm now at much lower percentage.
So my latest running theory is that it has something todo with how gluster
allocates the shared's, since it's based on it's hash it might want to
place it in a certain sub-volume, but than comes to the conclusion it has
not enough space there, writes a marker to redirect it to another
sub-volume (thinking this is the stale file). However rebalances don't fix
this issue.  Also this still doesn't seem explain that most stale files
always end up in the first sub-volume.
Unfortunate i've no proof this is actually the root cause, besides that the
symptom "disappeared" once gluster had more space to work with.

Best Olaf

Op wo 27 nov. 2019 om 02:38 schreef Timothy Orme :

> Hi All,
>
> I'm running a 3x2 cluster, v6.5.  Not sure if its relevant, but also have
> sharding enabled.
>
> I've found that when under heavy write load, clients start erroring out
> with "stale file handle" errors, on files not related to the writes.
>
> For instance, when a user is running a simple wc against a file, it will
> bail during that operation with "stale file"
>
> When I check the client logs, I see errors like:
>
> [2019-11-26 22:41:33.565776] E [MSGID: 109040]
> [dht-helper.c:1336:dht_migration_complete_check_task] 3-scratch-dht:
> 24d53a0e-c28d-41e0-9dbc-a75e823a3c7d: failed to lookup the file on
> scratch-dht  [Stale file handle]
> [2019-11-26 22:41:33.565853] W [fuse-bridge.c:2827:fuse_readv_cbk]
> 0-glusterfs-fuse: 33112038: READ => -1
> gfid=147040e2-a6b8-4f54-8490-f0f3df29ee50 fd=0x7f95d8d0b3f8 (Stale file
> handle)
>
> I've seen some bugs or other threads referencing similar issues, but
> couldn't really discern a solution from them.
>
> Is this caused by some consistency issue with metadata while under load or
> something else?  I dont see the issue when heavy reads are occurrring.
>
> Any help is greatly appreciated!
>
> Thanks!
> Tim
> 
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/441850968
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>


Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] glusterd crashes on Assertion failed: rsp.op == txn_op_info.op

[Olaf Buitelaar]

Hi Sanju,

going through the stacks i noticed that this function was in
between; glusterd_volume_rebalance_use_rsp_dict
So it might after all have todo something with the rebalancing logic.
I've checked the cmd_history.log and exactly on the time of crash time
command was executed;
[2019-06-19 07:25:03.108360]  : volume rebalance ovirt-data status :
SUCCESS preceding a couple of other status checks of rebalancing. The
complete batch of 2 mins before, all reported success.
These commands are executed by ovirt about every 2 minutes, to pull for the
status of gluster.
I'm sure no actual rebalancing tasks were running, also checked the last
time that was @2019-06-08 21:13:02 and was completed successfully
Hopefully this is additional useful info.

Thanks Olaf

Op do 20 jun. 2019 om 14:52 schreef Olaf Buitelaar :

> Hi Sanju,
>
> you can download the coredump here;
> http://edgecastcdn.net/0004FA/files/core_dump.zip (around 20MB)
>
> Thanks Olaf
>
> Op do 20 jun. 2019 om 08:35 schreef Sanju Rakonde :
>
>> Olaf,
>>
>> Can you please paste complete backtrace from the core file, so that we
>> can analyse what is wrong here.
>>
>> On Wed, Jun 19, 2019 at 10:31 PM Olaf Buitelaar 
>> wrote:
>>
>>> Hi Atin,
>>>
>>> Thank you for pointing out this bug report, however no rebalancing task
>>> was running during this event. So maybe something else is causing this?
>>> According the report this should be fixed in gluster 6, unfortunate
>>> ovirt doesn't seem to officially support that version, so i'm stuck on the
>>> 5 branch for now.
>>> Any chance this will be back ported?
>>>
>>> Thanks Olaf
>>>
>>>
>>> Op wo 19 jun. 2019 om 17:57 schreef Atin Mukherjee >> >:
>>>
>>>> Please see - https://bugzilla.redhat.com/show_bug.cgi?id=1655827
>>>>
>>>>
>>>>
>>>> On Wed, Jun 19, 2019 at 5:52 PM Olaf Buitelaar <
>>>> olaf.buitel...@gmail.com> wrote:
>>>>
>>>>> Dear All,
>>>>>
>>>>> Has anybody seen this error on gluster 5.6;
>>>>> [glusterd-rpc-ops.c:1388:__glusterd_commit_op_cbk]
>>>>> (-->/lib64/libgfrpc.so.0(+0xec60) [0x7fbfb7801c60]
>>>>> -->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x79b7a)
>>>>> [0x7fbfac50db7a]
>>>>> -->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x77393)
>>>>> [0x7fbfac50b393] ) 0-: Assertion failed: rsp.op == txn_op_info.op
>>>>>
>>>>> checking the code;
>>>>> https://github.com/gluster/glusterfs/blob/6fd8281ac9af58609979f660ece58c2ed1100e72/xlators/mgmt/glusterd/src/glusterd-rpc-ops.c#L1388
>>>>>
>>>>> doesn't seem to reveal much on what could causing this.
>>>>>
>>>>> It's the second time this occurs.
>>>>>
>>>>> Attached the full stack.
>>>>>
>>>>> Thanks Olaf
>>>>> ___
>>>>> Gluster-users mailing list
>>>>> Gluster-users@gluster.org
>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>> --
>> Thanks,
>> Sanju
>>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] glusterd crashes on Assertion failed: rsp.op == txn_op_info.op

[Olaf Buitelaar]

Hi Sanju,

you can download the coredump here;
http://edgecastcdn.net/0004FA/files/core_dump.zip (around 20MB)

Thanks Olaf

Op do 20 jun. 2019 om 08:35 schreef Sanju Rakonde :

> Olaf,
>
> Can you please paste complete backtrace from the core file, so that we can
> analyse what is wrong here.
>
> On Wed, Jun 19, 2019 at 10:31 PM Olaf Buitelaar 
> wrote:
>
>> Hi Atin,
>>
>> Thank you for pointing out this bug report, however no rebalancing task
>> was running during this event. So maybe something else is causing this?
>> According the report this should be fixed in gluster 6, unfortunate ovirt
>> doesn't seem to officially support that version, so i'm stuck on the 5
>> branch for now.
>> Any chance this will be back ported?
>>
>> Thanks Olaf
>>
>>
>> Op wo 19 jun. 2019 om 17:57 schreef Atin Mukherjee :
>>
>>> Please see - https://bugzilla.redhat.com/show_bug.cgi?id=1655827
>>>
>>>
>>>
>>> On Wed, Jun 19, 2019 at 5:52 PM Olaf Buitelaar 
>>> wrote:
>>>
>>>> Dear All,
>>>>
>>>> Has anybody seen this error on gluster 5.6;
>>>> [glusterd-rpc-ops.c:1388:__glusterd_commit_op_cbk]
>>>> (-->/lib64/libgfrpc.so.0(+0xec60) [0x7fbfb7801c60]
>>>> -->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x79b7a)
>>>> [0x7fbfac50db7a]
>>>> -->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x77393)
>>>> [0x7fbfac50b393] ) 0-: Assertion failed: rsp.op == txn_op_info.op
>>>>
>>>> checking the code;
>>>> https://github.com/gluster/glusterfs/blob/6fd8281ac9af58609979f660ece58c2ed1100e72/xlators/mgmt/glusterd/src/glusterd-rpc-ops.c#L1388
>>>>
>>>> doesn't seem to reveal much on what could causing this.
>>>>
>>>> It's the second time this occurs.
>>>>
>>>> Attached the full stack.
>>>>
>>>> Thanks Olaf
>>>> ___
>>>> Gluster-users mailing list
>>>> Gluster-users@gluster.org
>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
> --
> Thanks,
> Sanju
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] glusterd crashes on Assertion failed: rsp.op == txn_op_info.op

[Olaf Buitelaar]

Hi Atin,

Thank you for pointing out this bug report, however no rebalancing task was
running during this event. So maybe something else is causing this?
According the report this should be fixed in gluster 6, unfortunate ovirt
doesn't seem to officially support that version, so i'm stuck on the 5
branch for now.
Any chance this will be back ported?

Thanks Olaf


Op wo 19 jun. 2019 om 17:57 schreef Atin Mukherjee :

> Please see - https://bugzilla.redhat.com/show_bug.cgi?id=1655827
>
>
>
> On Wed, Jun 19, 2019 at 5:52 PM Olaf Buitelaar 
> wrote:
>
>> Dear All,
>>
>> Has anybody seen this error on gluster 5.6;
>> [glusterd-rpc-ops.c:1388:__glusterd_commit_op_cbk]
>> (-->/lib64/libgfrpc.so.0(+0xec60) [0x7fbfb7801c60]
>> -->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x79b7a)
>> [0x7fbfac50db7a]
>> -->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x77393)
>> [0x7fbfac50b393] ) 0-: Assertion failed: rsp.op == txn_op_info.op
>>
>> checking the code;
>> https://github.com/gluster/glusterfs/blob/6fd8281ac9af58609979f660ece58c2ed1100e72/xlators/mgmt/glusterd/src/glusterd-rpc-ops.c#L1388
>>
>> doesn't seem to reveal much on what could causing this.
>>
>> It's the second time this occurs.
>>
>> Attached the full stack.
>>
>> Thanks Olaf
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] glusterd crashes on Assertion failed: rsp.op == txn_op_info.op

[Olaf Buitelaar]

Dear All,

Has anybody seen this error on gluster 5.6;
[glusterd-rpc-ops.c:1388:__glusterd_commit_op_cbk]
(-->/lib64/libgfrpc.so.0(+0xec60) [0x7fbfb7801c60]
-->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x79b7a)
[0x7fbfac50db7a]
-->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x77393)
[0x7fbfac50b393] ) 0-: Assertion failed: rsp.op == txn_op_info.op

checking the code;
https://github.com/gluster/glusterfs/blob/6fd8281ac9af58609979f660ece58c2ed1100e72/xlators/mgmt/glusterd/src/glusterd-rpc-ops.c#L1388

doesn't seem to reveal much on what could causing this.

It's the second time this occurs.

Attached the full stack.

Thanks Olaf
[2019-06-19 07:25:03.289265] E 
[glusterd-rpc-ops.c:1388:__glusterd_commit_op_cbk] 
(-->/lib64/libgfrpc.so.0(+0xec60) [0x7fbfb7801c60] 
-->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x79b7a) [0x7fbfac50db7a] 
-->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x77393) [0x7fbfac50b393] 
) 0-: Assertion failed: rsp.op == txn_op_info.op
[2019-06-19 07:25:03.289410] E 
[glusterd-rpc-ops.c:1388:__glusterd_commit_op_cbk] 
(-->/lib64/libgfrpc.so.0(+0xec60) [0x7fbfb7801c60] 
-->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x79b7a) [0x7fbfac50db7a] 
-->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x77393) [0x7fbfac50b393] 
) 0-: Assertion failed: rsp.op == txn_op_info.op
[2019-06-19 07:25:03.290103] E 
[glusterd-rpc-ops.c:1388:__glusterd_commit_op_cbk] 
(-->/lib64/libgfrpc.so.0(+0xec60) [0x7fbfb7801c60] 
-->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x79b7a) [0x7fbfac50db7a] 
-->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x77393) [0x7fbfac50b393] 
) 0-: Assertion failed: rsp.op == txn_op_info.op
[2019-06-19 07:25:03.291407] E 
[glusterd-rpc-ops.c:1388:__glusterd_commit_op_cbk] 
(-->/lib64/libgfrpc.so.0(+0xec60) [0x7fbfb7801c60] 
-->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x79b7a) [0x7fbfac50db7a] 
-->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x77393) [0x7fbfac50b393] 
) 0-: Assertion failed: rsp.op == txn_op_info.op
[2019-06-19 07:25:03.299574] E 
[glusterd-rpc-ops.c:1388:__glusterd_commit_op_cbk] 
(-->/lib64/libgfrpc.so.0(+0xec60) [0x7fbfb7801c60] 
-->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x79b7a) [0x7fbfac50db7a] 
-->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x77393) [0x7fbfac50b393] 
) 0-: Assertion failed: rsp.op == txn_op_info.op
pending frames:
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(

Re: [Gluster-users] [ovirt-users] Re: Announcing Gluster release 5.5

[Olaf Buitelaar]

Dear  Mohit,

Thanks for backporting this issue. Hopefully we can address the others as
well, if i can do anything let me know.
On my side i've tested with: gluster volume reset 
cluster.choose-local, but haven't noticed really a change in performance.
On the good side, the brick processes didn't crash with updating this
config.
I'll experiment with the other changes as well, and see how the
combinations affect performance.
I also saw this commit; https://review.gluster.org/#/c/glusterfs/+/21333/
which looks very useful, will this be an recommended option for VM/block
workloads?

Thanks Olaf


Op wo 3 apr. 2019 om 17:56 schreef Mohit Agrawal :

>
> Hi,
>
> Thanks Olaf for sharing the relevant logs.
>
> @Atin,
> You are right patch https://review.gluster.org/#/c/glusterfs/+/22344/
> will resolve the issue running multiple brick instance for same brick.
>
> As we can see in below logs glusterd is trying to start the same brick
> instance twice at the same time
>
> [2019-04-01 10:23:21.752401] I
> [glusterd-utils.c:6301:glusterd_brick_start] 0-management: starting a fresh
> brick process for brick /data/gfs/bricks/brick1/ovirt-engine
> [2019-04-01 10:23:30.348091] I
> [glusterd-utils.c:6301:glusterd_brick_start] 0-management: starting a fresh
> brick process for brick /data/gfs/bricks/brick1/ovirt-engine
> [2019-04-01 10:24:13.353396] I
> [glusterd-utils.c:6301:glusterd_brick_start] 0-management: starting a fresh
> brick process for brick /data/gfs/bricks/brick1/ovirt-engine
> [2019-04-01 10:24:24.253764] I
> [glusterd-utils.c:6301:glusterd_brick_start] 0-management: starting a fresh
> brick process for brick /data/gfs/bricks/brick1/ovirt-engine
>
> We are seeing below message between starting of two instances
> The message "E [MSGID: 101012] [common-utils.c:4075:gf_is_service_running]
> 0-: Unable to read pidfile:
> /var/run/gluster/vols/ovirt-engine/10.32.9.5-data-gfs-bricks-brick1-ovirt-engine.pid"
> repeated 2 times between [2019-04-01 10:23:21.748492] and [2019-04-01
> 10:23:21.752432]
>
> I will backport the same.
> Thanks,
> Mohit Agrawal
>
> On Wed, Apr 3, 2019 at 3:58 PM Olaf Buitelaar 
> wrote:
>
>> Dear Mohit,
>>
>> Sorry i thought Krutika was referring to the ovirt-kube brick logs. due
>> the large size (18MB compressed), i've placed the files here;
>> https://edgecastcdn.net/0004FA/files/bricklogs.tar.bz2
>> Also i see i've attached the wrong files, i intended to
>> attach profile_data4.txt | profile_data3.txt
>> Sorry for the confusion.
>>
>> Thanks Olaf
>>
>> Op wo 3 apr. 2019 om 04:56 schreef Mohit Agrawal :
>>
>>> Hi Olaf,
>>>
>>>   As per current attached "multi-glusterfsd-vol3.txt |
>>> multi-glusterfsd-vol4.txt" it is showing multiple processes are running
>>>   for "ovirt-core ovirt-engine" brick names but there are no logs
>>> available in bricklogs.zip specific to this bricks, bricklogs.zip
>>>   has a dump of ovirt-kube logs only
>>>
>>>   Kindly share brick logs specific to the bricks "ovirt-core
>>> ovirt-engine" and share glusterd logs also.
>>>
>>> Regards
>>> Mohit Agrawal
>>>
>>> On Tue, Apr 2, 2019 at 9:18 PM Olaf Buitelaar 
>>> wrote:
>>>
>>>> Dear Krutika,
>>>>
>>>> 1.
>>>> I've changed the volume settings, write performance seems to increased
>>>> somewhat, however the profile doesn't really support that since latencies
>>>> increased. However read performance has diminished, which does seem to be
>>>> supported by the profile runs (attached).
>>>> Also the IO does seem to behave more consistent than before.
>>>> I don't really understand the idea behind them, maybe you can explain
>>>> why these suggestions are good?
>>>> These settings seems to avoid as much local caching and access as
>>>> possible and push everything to the gluster processes. While i would expect
>>>> local access and local caches are a good thing, since it would lead to
>>>> having less network access or disk access.
>>>> I tried to investigate these settings a bit more, and this is what i
>>>> understood of them;
>>>> - network.remote-dio; when on it seems to ignore the O_DIRECT flag in
>>>> the client, thus causing the files to be cached and buffered in the page
>>>> cache on the client, i would expect this to be a good thing especially if
>>>> the server process would access the same page cache?
>>>> At least that is what gra

Re: [Gluster-users] [Stale file handle] in shard volume

[Olaf Buitelaar]

@Krutika if you need any further information, please let me know.

Thanks Olaf

Op vr 4 jan. 2019 om 07:51 schreef Nithya Balachandran :

> Adding Krutika.
>
> On Wed, 2 Jan 2019 at 20:56, Olaf Buitelaar 
> wrote:
>
>> Hi Nithya,
>>
>> Thank you for your reply.
>>
>> the VM's using the gluster volumes keeps on getting paused/stopped on
>> errors like these;
>> [2019-01-02 02:33:44.469132] E [MSGID: 133010]
>> [shard.c:1724:shard_common_lookup_shards_cbk] 0-ovirt-kube-shard: Lookup on
>> shard 101487 failed. Base file gfid = a38d64bc-a28b-4ee1-a0bb-f919e7a1022c
>> [Stale file handle]
>> [2019-01-02 02:33:44.563288] E [MSGID: 133010]
>> [shard.c:1724:shard_common_lookup_shards_cbk] 0-ovirt-kube-shard: Lookup on
>> shard 101488 failed. Base file gfid = a38d64bc-a28b-4ee1-a0bb-f919e7a1022c
>> [Stale file handle]
>>
>> Krutika, Can you take a look at this?
>
>
>>
>> What i'm trying to find out, if i can purge all gluster volumes from all
>> possible stale file handles (and hopefully find a method to prevent this in
>> the future), so the VM's can start running stable again.
>> For this i need to know when the "shard_common_lookup_shards_cbk"
>> function considers a file as stale.
>> The statement; "Stale file handle errors show up when a file with a
>> specified gfid is not found." doesn't seem to cover it all, as i've shown
>> in earlier mails the shard file and glusterfs/xx/xx/uuid file do both
>> exist, and have the same inode.
>> If the criteria i'm using aren't correct, could you please tell me which
>> criteria i should use to determine if a file is stale or not?
>> these criteria are just based observations i made, moving the stale files
>> manually. After removing them i was able to start the VM again..until some
>> time later it hangs on another stale shard file unfortunate.
>>
>> Thanks Olaf
>>
>> Op wo 2 jan. 2019 om 14:20 schreef Nithya Balachandran <
>> nbala...@redhat.com>:
>>
>>>
>>>
>>> On Mon, 31 Dec 2018 at 01:27, Olaf Buitelaar 
>>> wrote:
>>>
>>>> Dear All,
>>>>
>>>> till now a selected group of VM's still seem to produce new stale
>>>> file's and getting paused due to this.
>>>> I've not updated gluster recently, however i did change the op version
>>>> from 31200 to 31202 about a week before this issue arose.
>>>> Looking at the .shard directory, i've 100.000+ files sharing the same
>>>> characteristics as a stale file. which are found till now,
>>>> they all have the sticky bit set, e.g. file permissions; -T.
>>>> are 0kb in size, and have the trusted.glusterfs.dht.linkto attribute.
>>>>
>>>
>>> These are internal files used by gluster and do not necessarily mean
>>> they are stale. They "point" to data files which may be on different bricks
>>> (same name, gfid etc but no linkto xattr and no T permissions).
>>>
>>>
>>>> These files range from long a go (beginning of the year) till now.
>>>> Which makes me suspect this was laying dormant for some time now..and
>>>> somehow recently surfaced.
>>>> Checking other sub-volumes they contain also 0kb files in the .shard
>>>> directory, but don't have the sticky bit and the linkto attribute.
>>>>
>>>> Does anybody else experience this issue? Could this be a bug or an
>>>> environmental issue?
>>>>
>>> These are most likely valid files- please do not delete them without
>>> double-checking.
>>>
>>> Stale file handle errors show up when a file with a specified gfid is
>>> not found. You will need to debug the files for which you see this error by
>>> checking the bricks to see if they actually exist.
>>>
>>>>
>>>> Also i wonder if there is any tool or gluster command to clean all
>>>> stale file handles?
>>>> Otherwise i'm planning to make a simple bash script, which iterates
>>>> over the .shard dir, checks each file for the above mentioned criteria, and
>>>> (re)moves the file and the corresponding .glusterfs file.
>>>> If there are other criteria needed to identify a stale file handle, i
>>>> would like to hear that.
>>>> If this is a viable and safe operati

Re: [Gluster-users] [Stale file handle] in shard volume

[Olaf Buitelaar]

Hi Nithya,

Thank you for your reply.

the VM's using the gluster volumes keeps on getting paused/stopped on
errors like these;
[2019-01-02 02:33:44.469132] E [MSGID: 133010]
[shard.c:1724:shard_common_lookup_shards_cbk] 0-ovirt-kube-shard: Lookup on
shard 101487 failed. Base file gfid = a38d64bc-a28b-4ee1-a0bb-f919e7a1022c
[Stale file handle]
[2019-01-02 02:33:44.563288] E [MSGID: 133010]
[shard.c:1724:shard_common_lookup_shards_cbk] 0-ovirt-kube-shard: Lookup on
shard 101488 failed. Base file gfid = a38d64bc-a28b-4ee1-a0bb-f919e7a1022c
[Stale file handle]


What i'm trying to find out, if i can purge all gluster volumes from all
possible stale file handles (and hopefully find a method to prevent this in
the future), so the VM's can start running stable again.
For this i need to know when the "shard_common_lookup_shards_cbk" function
considers a file as stale.
The statement; "Stale file handle errors show up when a file with a
specified gfid is not found." doesn't seem to cover it all, as i've shown
in earlier mails the shard file and glusterfs/xx/xx/uuid file do both
exist, and have the same inode.
If the criteria i'm using aren't correct, could you please tell me which
criteria i should use to determine if a file is stale or not?
these criteria are just based observations i made, moving the stale files
manually. After removing them i was able to start the VM again..until some
time later it hangs on another stale shard file unfortunate.

Thanks Olaf

Op wo 2 jan. 2019 om 14:20 schreef Nithya Balachandran :

>
>
> On Mon, 31 Dec 2018 at 01:27, Olaf Buitelaar 
> wrote:
>
>> Dear All,
>>
>> till now a selected group of VM's still seem to produce new stale file's
>> and getting paused due to this.
>> I've not updated gluster recently, however i did change the op version
>> from 31200 to 31202 about a week before this issue arose.
>> Looking at the .shard directory, i've 100.000+ files sharing the same
>> characteristics as a stale file. which are found till now,
>> they all have the sticky bit set, e.g. file permissions; -T. are
>> 0kb in size, and have the trusted.glusterfs.dht.linkto attribute.
>>
>
> These are internal files used by gluster and do not necessarily mean they
> are stale. They "point" to data files which may be on different bricks
> (same name, gfid etc but no linkto xattr and no T permissions).
>
>
>> These files range from long a go (beginning of the year) till now. Which
>> makes me suspect this was laying dormant for some time now..and somehow
>> recently surfaced.
>> Checking other sub-volumes they contain also 0kb files in the .shard
>> directory, but don't have the sticky bit and the linkto attribute.
>>
>> Does anybody else experience this issue? Could this be a bug or an
>> environmental issue?
>>
> These are most likely valid files- please do not delete them without
> double-checking.
>
> Stale file handle errors show up when a file with a specified gfid is not
> found. You will need to debug the files for which you see this error by
> checking the bricks to see if they actually exist.
>
>>
>> Also i wonder if there is any tool or gluster command to clean all stale
>> file handles?
>> Otherwise i'm planning to make a simple bash script, which iterates over
>> the .shard dir, checks each file for the above mentioned criteria, and
>> (re)moves the file and the corresponding .glusterfs file.
>> If there are other criteria needed to identify a stale file handle, i
>> would like to hear that.
>> If this is a viable and safe operation to do of course.
>>
>> Thanks Olaf
>>
>>
>>
>> Op do 20 dec. 2018 om 13:43 schreef Olaf Buitelaar <
>> olaf.buitel...@gmail.com>:
>>
>>> Dear All,
>>>
>>> I figured it out, it appeared to be the exact same issue as described
>>> here;
>>> https://lists.gluster.org/pipermail/gluster-users/2018-March/033785.html
>>> Another subvolume also had the shard file, only were all 0 bytes and had
>>> the dht.linkto
>>>
>>> for reference;
>>> [root@lease-04 ovirt-backbone-2]# getfattr -d -m . -e hex
>>> .shard/f28cabcb-d169-41fc-a633-9bef4c4a8e40.51500
>>> # file: .shard/f28cabcb-d169-41fc-a633-9bef4c4a8e40.51500
>>>
>>> security.selinux=0x73797374656d5f753a6f626a6563745f723a6574635f72756e74696d655f743a733000
>>> trusted.gfid=0x298147e49f9748b2baf1c8fff897244d
>>>
>>> trusted.gfid2path.b48064c78d7a85c9=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f66323863616263622d643136392d3431666

Re: [Gluster-users] [Stale file handle] in shard volume

[Olaf Buitelaar]

Dear All,

The bash file i'm planning to run can be found here;
https://gist.github.com/olafbuitelaar/ff6fe9d4ab39696d9ad6ca689cc89986
It would be nice to receive some feedback from the community before i would
actually run the clean-up of all stale file handles.

Thanks Olaf

Op zo 30 dec. 2018 om 20:56 schreef Olaf Buitelaar :

> Dear All,
>
> till now a selected group of VM's still seem to produce new stale file's
> and getting paused due to this.
> I've not updated gluster recently, however i did change the op version
> from 31200 to 31202 about a week before this issue arose.
> Looking at the .shard directory, i've 100.000+ files sharing the same
> characteristics as a stale file. which are found till now,
> they all have the sticky bit set, e.g. file permissions; -T. are
> 0kb in size, and have the trusted.glusterfs.dht.linkto attribute.
> These files range from long a go (beginning of the year) till now. Which
> makes me suspect this was laying dormant for some time now..and somehow
> recently surfaced.
> Checking other sub-volumes they contain also 0kb files in the .shard
> directory, but don't have the sticky bit and the linkto attribute.
>
> Does anybody else experience this issue? Could this be a bug or an
> environmental issue?
>
> Also i wonder if there is any tool or gluster command to clean all stale
> file handles?
> Otherwise i'm planning to make a simple bash script, which iterates over
> the .shard dir, checks each file for the above mentioned criteria, and
> (re)moves the file and the corresponding .glusterfs file.
> If there are other criteria needed to identify a stale file handle, i
> would like to hear that.
> If this is a viable and safe operation to do of course.
>
> Thanks Olaf
>
>
>
> Op do 20 dec. 2018 om 13:43 schreef Olaf Buitelaar <
> olaf.buitel...@gmail.com>:
>
>> Dear All,
>>
>> I figured it out, it appeared to be the exact same issue as described
>> here;
>> https://lists.gluster.org/pipermail/gluster-users/2018-March/033785.html
>> Another subvolume also had the shard file, only were all 0 bytes and had
>> the dht.linkto
>>
>> for reference;
>> [root@lease-04 ovirt-backbone-2]# getfattr -d -m . -e hex
>> .shard/f28cabcb-d169-41fc-a633-9bef4c4a8e40.51500
>> # file: .shard/f28cabcb-d169-41fc-a633-9bef4c4a8e40.51500
>>
>> security.selinux=0x73797374656d5f753a6f626a6563745f723a6574635f72756e74696d655f743a733000
>> trusted.gfid=0x298147e49f9748b2baf1c8fff897244d
>>
>> trusted.gfid2path.b48064c78d7a85c9=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f66323863616263622d643136392d343166632d61362d3962656634633461386534302e3531353030
>>
>> trusted.glusterfs.dht.linkto=0x6f766972742d6261636b626f6e652d322d7265706c69636174652d3100
>>
>> [root@lease-04 ovirt-backbone-2]# getfattr -d -m . -e hex
>> .glusterfs/29/81/298147e4-9f97-48b2-baf1-c8fff897244d
>> # file: .glusterfs/29/81/298147e4-9f97-48b2-baf1-c8fff897244d
>>
>> security.selinux=0x73797374656d5f753a6f626a6563745f723a6574635f72756e74696d655f743a733000
>> trusted.gfid=0x298147e49f9748b2baf1c8fff897244d
>>
>> trusted.gfid2path.b48064c78d7a85c9=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f66323863616263622d643136392d343166632d61362d3962656634633461386534302e3531353030
>>
>> trusted.glusterfs.dht.linkto=0x6f766972742d6261636b626f6e652d322d7265706c69636174652d3100
>>
>> [root@lease-04 ovirt-backbone-2]# stat
>> .glusterfs/29/81/298147e4-9f97-48b2-baf1-c8fff897244d
>>   File: ‘.glusterfs/29/81/298147e4-9f97-48b2-baf1-c8fff897244d’
>>   Size: 0   Blocks: 0  IO Block: 4096   regular empty
>> file
>> Device: fd01h/64769dInode: 1918631406  Links: 2
>> Access: (1000/-T)  Uid: (0/root)   Gid: (0/root)
>> Context: system_u:object_r:etc_runtime_t:s0
>> Access: 2018-12-17 21:43:36.405735296 +
>> Modify: 2018-12-17 21:43:36.405735296 +
>> Change: 2018-12-17 21:43:36.405735296 +
>>  Birth: -
>>
>> removing the shard file and glusterfs file from each node resolved the
>> issue.
>>
>> I also found this thread;
>> https://lists.gluster.org/pipermail/gluster-users/2018-December/035460.html
>> Maybe he suffers from the same issue.
>>
>> Best Olaf
>>
>>
>> Op wo 19 dec. 2018 om 21:56 schreef Olaf Buitelaar <
>> olaf.buitel...@gmail.com>:
>>
>>> Dear All,
>>>
>>> It appears i've a stale file in one of the volumes, on 2 files. These
>>> files are qemu images (1 raw and 1 qcow2).
>&g

Re: [Gluster-users] [Stale file handle] in shard volume

[Olaf Buitelaar]

Dear All,

till now a selected group of VM's still seem to produce new stale file's
and getting paused due to this.
I've not updated gluster recently, however i did change the op version from
31200 to 31202 about a week before this issue arose.
Looking at the .shard directory, i've 100.000+ files sharing the same
characteristics as a stale file. which are found till now,
they all have the sticky bit set, e.g. file permissions; -T. are
0kb in size, and have the trusted.glusterfs.dht.linkto attribute.
These files range from long a go (beginning of the year) till now. Which
makes me suspect this was laying dormant for some time now..and somehow
recently surfaced.
Checking other sub-volumes they contain also 0kb files in the .shard
directory, but don't have the sticky bit and the linkto attribute.

Does anybody else experience this issue? Could this be a bug or an
environmental issue?

Also i wonder if there is any tool or gluster command to clean all stale
file handles?
Otherwise i'm planning to make a simple bash script, which iterates over
the .shard dir, checks each file for the above mentioned criteria, and
(re)moves the file and the corresponding .glusterfs file.
If there are other criteria needed to identify a stale file handle, i would
like to hear that.
If this is a viable and safe operation to do of course.

Thanks Olaf



Op do 20 dec. 2018 om 13:43 schreef Olaf Buitelaar :

> Dear All,
>
> I figured it out, it appeared to be the exact same issue as described
> here;
> https://lists.gluster.org/pipermail/gluster-users/2018-March/033785.html
> Another subvolume also had the shard file, only were all 0 bytes and had
> the dht.linkto
>
> for reference;
> [root@lease-04 ovirt-backbone-2]# getfattr -d -m . -e hex
> .shard/f28cabcb-d169-41fc-a633-9bef4c4a8e40.51500
> # file: .shard/f28cabcb-d169-41fc-a633-9bef4c4a8e40.51500
>
> security.selinux=0x73797374656d5f753a6f626a6563745f723a6574635f72756e74696d655f743a733000
> trusted.gfid=0x298147e49f9748b2baf1c8fff897244d
>
> trusted.gfid2path.b48064c78d7a85c9=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f66323863616263622d643136392d343166632d61362d3962656634633461386534302e3531353030
>
> trusted.glusterfs.dht.linkto=0x6f766972742d6261636b626f6e652d322d7265706c69636174652d3100
>
> [root@lease-04 ovirt-backbone-2]# getfattr -d -m . -e hex
> .glusterfs/29/81/298147e4-9f97-48b2-baf1-c8fff897244d
> # file: .glusterfs/29/81/298147e4-9f97-48b2-baf1-c8fff897244d
>
> security.selinux=0x73797374656d5f753a6f626a6563745f723a6574635f72756e74696d655f743a733000
> trusted.gfid=0x298147e49f9748b2baf1c8fff897244d
>
> trusted.gfid2path.b48064c78d7a85c9=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f66323863616263622d643136392d343166632d61362d3962656634633461386534302e3531353030
>
> trusted.glusterfs.dht.linkto=0x6f766972742d6261636b626f6e652d322d7265706c69636174652d3100
>
> [root@lease-04 ovirt-backbone-2]# stat
> .glusterfs/29/81/298147e4-9f97-48b2-baf1-c8fff897244d
>   File: ‘.glusterfs/29/81/298147e4-9f97-48b2-baf1-c8fff897244d’
>   Size: 0   Blocks: 0  IO Block: 4096   regular empty
> file
> Device: fd01h/64769dInode: 1918631406  Links: 2
> Access: (1000/-T)  Uid: (0/root)   Gid: (0/root)
> Context: system_u:object_r:etc_runtime_t:s0
> Access: 2018-12-17 21:43:36.405735296 +
> Modify: 2018-12-17 21:43:36.405735296 +
> Change: 2018-12-17 21:43:36.405735296 +
>  Birth: -
>
> removing the shard file and glusterfs file from each node resolved the
> issue.
>
> I also found this thread;
> https://lists.gluster.org/pipermail/gluster-users/2018-December/035460.html
> Maybe he suffers from the same issue.
>
> Best Olaf
>
>
> Op wo 19 dec. 2018 om 21:56 schreef Olaf Buitelaar <
> olaf.buitel...@gmail.com>:
>
>> Dear All,
>>
>> It appears i've a stale file in one of the volumes, on 2 files. These
>> files are qemu images (1 raw and 1 qcow2).
>> I'll just focus on 1 file since the situation on the other seems the
>> same.
>>
>> The VM get's paused more or less directly after being booted with error;
>> [2018-12-18 14:05:05.275713] E [MSGID: 133010]
>> [shard.c:1724:shard_common_lookup_shards_cbk] 0-ovirt-backbone-2-shard:
>> Lookup on shard 51500 failed. Base file gfid =
>> f28cabcb-d169-41fc-a633-9bef4c4a8e40 [Stale file handle]
>>
>> investigating the shard;
>>
>> #on the arbiter node:
>>
>> [root@lease-05 ovirt-backbone-2]# getfattr -n glusterfs.gfid.string
>> /mnt/ovirt-backbone-2/b1c2c949-aef4-4aec-999b-b179efeef732/images/f6ac9660-a84e-469e-a17c-c6dbc538af4b/d6b09501-5b79-4c92-bf10-2ed3bda1b425
>> getfattr: Rem

Re: [Gluster-users] [Stale file handle] in shard volume

[Olaf Buitelaar]

Dear All,

I figured it out, it appeared to be the exact same issue as described
here;
https://lists.gluster.org/pipermail/gluster-users/2018-March/033785.html
Another subvolume also had the shard file, only were all 0 bytes and had
the dht.linkto

for reference;
[root@lease-04 ovirt-backbone-2]# getfattr -d -m . -e hex
.shard/f28cabcb-d169-41fc-a633-9bef4c4a8e40.51500
# file: .shard/f28cabcb-d169-41fc-a633-9bef4c4a8e40.51500
security.selinux=0x73797374656d5f753a6f626a6563745f723a6574635f72756e74696d655f743a733000
trusted.gfid=0x298147e49f9748b2baf1c8fff897244d
trusted.gfid2path.b48064c78d7a85c9=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f66323863616263622d643136392d343166632d61362d3962656634633461386534302e3531353030
trusted.glusterfs.dht.linkto=0x6f766972742d6261636b626f6e652d322d7265706c69636174652d3100

[root@lease-04 ovirt-backbone-2]# getfattr -d -m . -e hex
.glusterfs/29/81/298147e4-9f97-48b2-baf1-c8fff897244d
# file: .glusterfs/29/81/298147e4-9f97-48b2-baf1-c8fff897244d
security.selinux=0x73797374656d5f753a6f626a6563745f723a6574635f72756e74696d655f743a733000
trusted.gfid=0x298147e49f9748b2baf1c8fff897244d
trusted.gfid2path.b48064c78d7a85c9=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f66323863616263622d643136392d343166632d61362d3962656634633461386534302e3531353030
trusted.glusterfs.dht.linkto=0x6f766972742d6261636b626f6e652d322d7265706c69636174652d3100

[root@lease-04 ovirt-backbone-2]# stat
.glusterfs/29/81/298147e4-9f97-48b2-baf1-c8fff897244d
  File: ‘.glusterfs/29/81/298147e4-9f97-48b2-baf1-c8fff897244d’
  Size: 0   Blocks: 0  IO Block: 4096   regular empty
file
Device: fd01h/64769dInode: 1918631406  Links: 2
Access: (1000/-T)  Uid: (0/root)   Gid: (0/root)
Context: system_u:object_r:etc_runtime_t:s0
Access: 2018-12-17 21:43:36.405735296 +
Modify: 2018-12-17 21:43:36.405735296 +
Change: 2018-12-17 21:43:36.405735296 +
 Birth: -

removing the shard file and glusterfs file from each node resolved the
issue.

I also found this thread;
https://lists.gluster.org/pipermail/gluster-users/2018-December/035460.html
Maybe he suffers from the same issue.

Best Olaf


Op wo 19 dec. 2018 om 21:56 schreef Olaf Buitelaar :

> Dear All,
>
> It appears i've a stale file in one of the volumes, on 2 files. These
> files are qemu images (1 raw and 1 qcow2).
> I'll just focus on 1 file since the situation on the other seems the same.
>
> The VM get's paused more or less directly after being booted with error;
> [2018-12-18 14:05:05.275713] E [MSGID: 133010]
> [shard.c:1724:shard_common_lookup_shards_cbk] 0-ovirt-backbone-2-shard:
> Lookup on shard 51500 failed. Base file gfid =
> f28cabcb-d169-41fc-a633-9bef4c4a8e40 [Stale file handle]
>
> investigating the shard;
>
> #on the arbiter node:
>
> [root@lease-05 ovirt-backbone-2]# getfattr -n glusterfs.gfid.string
> /mnt/ovirt-backbone-2/b1c2c949-aef4-4aec-999b-b179efeef732/images/f6ac9660-a84e-469e-a17c-c6dbc538af4b/d6b09501-5b79-4c92-bf10-2ed3bda1b425
> getfattr: Removing leading '/' from absolute path names
> # file:
> mnt/ovirt-backbone-2/b1c2c949-aef4-4aec-999b-b179efeef732/images/f6ac9660-a84e-469e-a17c-c6dbc538af4b/d6b09501-5b79-4c92-bf10-2ed3bda1b425
> glusterfs.gfid.string="f28cabcb-d169-41fc-a633-9bef4c4a8e40"
>
> [root@lease-05 ovirt-backbone-2]# getfattr -d -m . -e hex
> .shard/f28cabcb-d169-41fc-a633-9bef4c4a8e40.51500
> # file: .shard/f28cabcb-d169-41fc-a633-9bef4c4a8e40.51500
>
> security.selinux=0x73797374656d5f753a6f626a6563745f723a6574635f72756e74696d655f743a733000
> trusted.afr.dirty=0x
> trusted.gfid=0x1f86b4328ec6424699aa48cc6d7b5da0
>
> trusted.gfid2path.b48064c78d7a85c9=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f66323863616263622d643136392d343166632d61362d3962656634633461386534302e3531353030
>
> [root@lease-05 ovirt-backbone-2]# getfattr -d -m . -e hex
> .glusterfs/1f/86/1f86b432-8ec6-4246-99aa-48cc6d7b5da0
> # file: .glusterfs/1f/86/1f86b432-8ec6-4246-99aa-48cc6d7b5da0
>
> security.selinux=0x73797374656d5f753a6f626a6563745f723a6574635f72756e74696d655f743a733000
> trusted.afr.dirty=0x
> trusted.gfid=0x1f86b4328ec6424699aa48cc6d7b5da0
>
> trusted.gfid2path.b48064c78d7a85c9=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f66323863616263622d643136392d343166632d61362d3962656634633461386534302e3531353030
>
> [root@lease-05 ovirt-backbone-2]# stat
> .glusterfs/1f/86/1f86b432-8ec6-4246-99aa-48cc6d7b5da0
>   File: ‘.glusterfs/1f/86/1f86b432-8ec6-4246-99aa-48cc6d7b5da0’
>   Size: 0   Blocks: 0  IO Block: 4096   regular empty
> file
> Device: fd01h/64769dInode: 537277306   Links: 2
> Access: (0660/-rw-rw)

[Gluster-users] [Stale file handle] in shard volume

[Olaf Buitelaar]

Dear All,

It appears i've a stale file in one of the volumes, on 2 files. These files
are qemu images (1 raw and 1 qcow2).
I'll just focus on 1 file since the situation on the other seems the same.

The VM get's paused more or less directly after being booted with error;
[2018-12-18 14:05:05.275713] E [MSGID: 133010]
[shard.c:1724:shard_common_lookup_shards_cbk] 0-ovirt-backbone-2-shard:
Lookup on shard 51500 failed. Base file gfid =
f28cabcb-d169-41fc-a633-9bef4c4a8e40 [Stale file handle]

investigating the shard;

#on the arbiter node:

[root@lease-05 ovirt-backbone-2]# getfattr -n glusterfs.gfid.string
/mnt/ovirt-backbone-2/b1c2c949-aef4-4aec-999b-b179efeef732/images/f6ac9660-a84e-469e-a17c-c6dbc538af4b/d6b09501-5b79-4c92-bf10-2ed3bda1b425
getfattr: Removing leading '/' from absolute path names
# file:
mnt/ovirt-backbone-2/b1c2c949-aef4-4aec-999b-b179efeef732/images/f6ac9660-a84e-469e-a17c-c6dbc538af4b/d6b09501-5b79-4c92-bf10-2ed3bda1b425
glusterfs.gfid.string="f28cabcb-d169-41fc-a633-9bef4c4a8e40"

[root@lease-05 ovirt-backbone-2]# getfattr -d -m . -e hex
.shard/f28cabcb-d169-41fc-a633-9bef4c4a8e40.51500
# file: .shard/f28cabcb-d169-41fc-a633-9bef4c4a8e40.51500
security.selinux=0x73797374656d5f753a6f626a6563745f723a6574635f72756e74696d655f743a733000
trusted.afr.dirty=0x
trusted.gfid=0x1f86b4328ec6424699aa48cc6d7b5da0
trusted.gfid2path.b48064c78d7a85c9=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f66323863616263622d643136392d343166632d61362d3962656634633461386534302e3531353030

[root@lease-05 ovirt-backbone-2]# getfattr -d -m . -e hex
.glusterfs/1f/86/1f86b432-8ec6-4246-99aa-48cc6d7b5da0
# file: .glusterfs/1f/86/1f86b432-8ec6-4246-99aa-48cc6d7b5da0
security.selinux=0x73797374656d5f753a6f626a6563745f723a6574635f72756e74696d655f743a733000
trusted.afr.dirty=0x
trusted.gfid=0x1f86b4328ec6424699aa48cc6d7b5da0
trusted.gfid2path.b48064c78d7a85c9=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f66323863616263622d643136392d343166632d61362d3962656634633461386534302e3531353030

[root@lease-05 ovirt-backbone-2]# stat
.glusterfs/1f/86/1f86b432-8ec6-4246-99aa-48cc6d7b5da0
  File: ‘.glusterfs/1f/86/1f86b432-8ec6-4246-99aa-48cc6d7b5da0’
  Size: 0   Blocks: 0  IO Block: 4096   regular empty
file
Device: fd01h/64769dInode: 537277306   Links: 2
Access: (0660/-rw-rw)  Uid: (0/root)   Gid: (0/root)
Context: system_u:object_r:etc_runtime_t:s0
Access: 2018-12-17 21:43:36.361984810 +
Modify: 2018-12-17 21:43:36.361984810 +
Change: 2018-12-18 20:55:29.908647417 +
 Birth: -

[root@lease-05 ovirt-backbone-2]# find . -inum 537277306
./.glusterfs/1f/86/1f86b432-8ec6-4246-99aa-48cc6d7b5da0
./.shard/f28cabcb-d169-41fc-a633-9bef4c4a8e40.51500

#on the data nodes:

[root@lease-08 ~]# getfattr -n glusterfs.gfid.string
/mnt/ovirt-backbone-2/b1c2c949-aef4-4aec-999b-b179efeef732/images/f6ac9660-a84e-469e-a17c-c6dbc538af4b/d6b09501-5b79-4c92-bf10-2ed3bda1b425
getfattr: Removing leading '/' from absolute path names
# file:
mnt/ovirt-backbone-2/b1c2c949-aef4-4aec-999b-b179efeef732/images/f6ac9660-a84e-469e-a17c-c6dbc538af4b/d6b09501-5b79-4c92-bf10-2ed3bda1b425
glusterfs.gfid.string="f28cabcb-d169-41fc-a633-9bef4c4a8e40"

[root@lease-08 ovirt-backbone-2]# getfattr -d -m . -e hex
.shard/f28cabcb-d169-41fc-a633-9bef4c4a8e40.51500
# file: .shard/f28cabcb-d169-41fc-a633-9bef4c4a8e40.51500
security.selinux=0x73797374656d5f753a6f626a6563745f723a6574635f72756e74696d655f743a733000
trusted.afr.dirty=0x
trusted.gfid=0x1f86b4328ec6424699aa48cc6d7b5da0
trusted.gfid2path.b48064c78d7a85c9=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f66323863616263622d643136392d343166632d61362d3962656634633461386534302e3531353030

[root@lease-08 ovirt-backbone-2]# getfattr -d -m . -e hex
.glusterfs/1f/86/1f86b432-8ec6-4246-99aa-48cc6d7b5da0
# file: .glusterfs/1f/86/1f86b432-8ec6-4246-99aa-48cc6d7b5da0
security.selinux=0x73797374656d5f753a6f626a6563745f723a6574635f72756e74696d655f743a733000
trusted.afr.dirty=0x
trusted.gfid=0x1f86b4328ec6424699aa48cc6d7b5da0
trusted.gfid2path.b48064c78d7a85c9=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f66323863616263622d643136392d343166632d61362d3962656634633461386534302e3531353030

[root@lease-08 ovirt-backbone-2]# stat
.glusterfs/1f/86/1f86b432-8ec6-4246-99aa-48cc6d7b5da0
  File: ‘.glusterfs/1f/86/1f86b432-8ec6-4246-99aa-48cc6d7b5da0’
  Size: 2166784 Blocks: 4128   IO Block: 4096   regular file
Device: fd03h/64771dInode: 12893624759  Links: 3
Access: (0660/-rw-rw)  Uid: (0/root)   Gid: (0/root)
Context: system_u:object_r:etc_runtime_t:s0
Access: 2018-12-18 18:52:38.070776585 +
Modify: 2018-12-17 21:43:36.388054443 +
Change: 2018-12-18 21:01:47.810506528 +
 Birth: -

[root@lease-08 ovirt-backbone-2]# find . -in