[Gluster-users] Does not look like 8.4 is pushed to CentOS mirrors.
Hi Community, Do we know if the GlusterFS builds will be pushed to Centos mirrors again ? E.g. to http://mirror.centos.org/centos/7/storage/ (or the CentOS 8 repo). Thanx, Claus. -- *Claus Jeppesen* Manager, Network Services Datto, Inc. p +45 6170 5901 | Copenhagen Office www.datto.com Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Sharding on 7.x - file sizes are wrong after a large copy.
In April of this year I reported the problem using sharding on gluster 7.4: We're using GlusterFS in a replicated brick setup with 2 bricks with sharding turned on (shardsize 128MB). There is something funny going on as we can see that if we copy large VM files to the volume we can end up with files that are a bit larger than the source files DEPENDING on the speed with which we copied the files - e.g.: dd if=SOURCE bs=1M | pv -L NNm | ssh gluster_server "dd of=/gluster/VOL_NAME/TARGET bs=1M" It seems that if NN is <= 25 (i.e. 25 MB/s) the size of SOURCE and TARGET will be the same. If we crank NN to, say, 50 we sometimes risk that a 25G file ends up having a slightly larger size, e.g. 26844413952 or 26844233728 - larger than the expected 26843545600. Unfortunately this is not an illusion ! If we dd the files out of Gluster we will receive the amount of data that 'ls' showed us. In the brick directory (incl .shard directory) we have the expected amount of shards for a 25G files (200) with size precisely equal to 128MB - but there is an additional 0 size shard file created. Has anyone else seen a phenomenon like this ? After upgrade to 7.6 we're still seeing this problem - now, the extra bytes that are appearing can be removed using truncate in the mounted gluster volume, and md5sum can confirm that after truncate the content is identical to the source - however, it may point to an underlying issue. I hope someone can reproduce this behaviour, Thanx, Claus. -- *Claus Jeppesen* Manager, Network Services Datto, Inc. p +45 6170 5901 | Copenhagen Office www.datto.com Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Sharding on 7.4 - filesizes may be wrong
We're using GlusterFS in a replicated brick setup with 2 bricks with sharding turned on (shardsize 128MB). There is something funny going on as we can see that if we copy large VM files to the volume we can end up with files that are a bit larger than the source files DEPENDING on the speed with which we copied the files - e.g.: dd if=SOURCE bs=1M | pv -L NNm | ssh gluster_server "dd of=/gluster/VOL_NAME/TARGET bs=1M" It seems that if NN is <= 25 (i.e. 25 MB/s) the size of SOURCE and TARGET will be the same. If we crank NN to, say, 50 we sometimes risk that a 25G file ends up having a slightly larger size, e.g. 26844413952 or 26844233728 - larger than the expected 26843545600. Unfortunately this is not an illusion ! If we dd the files out of Gluster we will receive the amount of data that 'ls' showed us. In the brick directory (incl .shard directory) we have the expected amount of shards for a 25G files (200) with size precisely equal to 128MB - but there is an additional 0 size shard file created. Has anyone else seen a phenomenon like this ? Thanx, Claus. -- *Claus Jeppesen* Manager, Network Services Datto, Inc. p +45 6170 5901 | Copenhagen Office www.datto.com Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
Re: [Gluster-users] KVM lockups on Gluster 4.1.1
Hi Amar, Unfortunately I do not have the GlusterFS brick logs anymore - however I do have a hint: I have 2 gluster (4.1.1) glusterfs volumes where I saw the issue - each has about 10-12 VMs active. I also have 2 addl. gluster (4.1.1) glusterfs volumes, but with only 3-4 VMs, where I did not see the issue (and they had been running for 1-2 months). Thanx, Claus. P.S. We are talking about using Gluster "URI" with qemu - I hope - e.g. like On Mon, Aug 20, 2018 at 5:39 PM Amar Tumballi wrote: > > > On Mon, Aug 20, 2018 at 6:20 PM, Walter Deignan > wrote: > >> I upgraded late last week to 4.1.2. Since then I've seen several posix >> health checks fail and bricks drop offline but I'm not sure if that's >> related or a different root issue. >> >> I haven't seen the issue described below re-occur on 4.1.2 yet but it was >> intermittent to begin with so I'll probably need to run for a week or more >> to be confident. >> >> > Thanks for the update! We will be trying to reproduce the issue, and also > root cause based on analysis of code, but if you get us brick logs around > the time this happens, it may fasttrack the issue. > > Thanks again, > Amar > > >> -Walter Deignan >> -Uline IT, Systems Architect >> >> >> >> From:"Claus Jeppesen" >> To:wdeig...@uline.com >> Cc:gluster-users@gluster.org >> Date:08/20/2018 07:20 AM >> Subject:Re: [Gluster-users] KVM lockups on Gluster 4.1.1 >> -- >> >> >> >> I think I have seen this also on our CentOS 7.5 systems using GlusterFS >> 4.1.1 (*) - has an upgrade to 4.1.2 helped out ? I'm trying this now. >> >> Thanx, >> >> Claus. >> >> (*) libvirt/quemu log: >> [2018-08-19 16:45:54.275830] E [MSGID: 114031] >> [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk] >> 0-glu-vol01-lab-client-0: remote operation failed [Invalid argument] >> [2018-08-19 16:45:54.276156] E [MSGID: 114031] >> [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk] >> 0-glu-vol01-lab-client-1: remote operation failed [Invalid argument] >> [2018-08-19 16:45:54.276159] E [MSGID: 108010] >> [afr-lk-common.c:284:afr_unlock_inodelk_cbk] 0-glu-vol01-lab-replicate-0: >> path=(null) gfid=----: unlock failed on >> subvolume glu-vol >> 01-lab-client-0 with lock owner 28ae49704956 [Invalid argument] >> [2018-08-19 16:45:54.276183] E [MSGID: 108010] >> [afr-lk-common.c:284:afr_unlock_inodelk_cbk] 0-glu-vol01-lab-replicate-0: >> path=(null) gfid=----: unlock failed on >> subvolume glu-vol >> 01-lab-client-1 with lock owner 28ae49704956 [Invalid argument] >> [2018-08-19 17:16:03.690808] E [rpc-clnt.c:184:call_bail] >> 0-glu-vol01-lab-client-0: bailing out frame type(GlusterFS 4.x v1) >> op(FINODELK(30)) xid = 0x3071a5 sent = 2018-08-19 16:45:54.276560. timeout >> = 1800 for >> *192.168.13.131:49152* <http://192.168.13.131:49152/> >> [2018-08-19 17:16:03.691113] E [MSGID: 114031] >> [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk] >> 0-glu-vol01-lab-client-0: remote operation failed [Transport endpoint is >> not connected] >> [2018-08-19 17:46:03.855909] E [rpc-clnt.c:184:call_bail] >> 0-glu-vol01-lab-client-1: bailing out frame type(GlusterFS 4.x v1) >> op(FINODELK(30)) xid = 0x301d0f sent = 2018-08-19 17:16:03.691174. timeout >> = 1800 for >> *192.168.13.132:49152* <http://192.168.13.132:49152/> >> [2018-08-19 17:46:03.856170] E [MSGID: 114031] >> [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk] >> 0-glu-vol01-lab-client-1: remote operation failed [Transport endpoint is >> not connected] >> block I/O error in device 'drive-virtio-disk0': Operation not permitted >> (1) >> ... many repeats ... >> block I/O error in device 'drive-virtio-disk0': Operation not permitted >> (1) >> [2018-08-19 18:16:04.022526] E [rpc-clnt.c:184:call_bail] >> 0-glu-vol01-lab-client-0: bailing out frame type(GlusterFS 4.x v1) >> op(FINODELK(30)) xid = 0x307221 sent = 2018-08-19 17:46:03.861005. timeout >> = 1800 for >> *192.168.13.131:49152* <http://192.168.13.131:49152/> >> [2018-08-19 18:16:04.022788] E [MSGID: 114031] >> [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk] >> 0-glu-vol01-lab-client-0: remote operation failed [Transport endpoint is >> not connected] >> [2018-08-19 18:46:04.195590] E [rpc-clnt.c:184:call_bail] >> 0-glu-vol01-lab-client-1: bailing out frame type(GlusterFS 4.x v1) >>
Re: [Gluster-users] KVM lockups on Gluster 4.1.1
-: unlock failed on subvolume > gv1-client-5 with lock owner d89caca92b7f [Invalid argument] > [2018-08-14 14:52:18.726219] E [rpc-clnt.c:184:call_bail] 2-gv1-client-4: > bailing out frame type(GlusterFS 4.x v1) op(FINODELK(30)) xid = 0xc5e00 > sent = 2018-08-14 14:22:15.699082. timeout = 1800 for 10.35.20.106:49159 > [2018-08-14 14:52:18.726254] E [MSGID: 114031] > [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk] 2-gv1-client-4: remote > operation failed [Transport endpoint is not connected] > [2018-08-14 15:22:25.962546] E [rpc-clnt.c:184:call_bail] 2-gv1-client-5: > bailing out frame type(GlusterFS 4.x v1) op(FINODELK(30)) xid = 0xc4a6d > sent = 2018-08-14 14:52:18.726329. timeout = 1800 for 10.35.20.107:49164 > [2018-08-14 15:22:25.962587] E [MSGID: 114031] > [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk] 2-gv1-client-5: remote > operation failed [Transport endpoint is not connected] > [2018-08-14 15:22:25.962618] W [MSGID: 108019] > [afr-lk-common.c:601:is_blocking_locks_count_sufficient] 2-gv1-replicate-2: > Unable to obtain blocking inode lock on even one child for > gfid:24a48cae-53fe-4634-8fb7-0254c85ad672. > [2018-08-14 15:22:25.962668] W [fuse-bridge.c:1441:fuse_err_cbk] > 0-glusterfs-fuse: 3715808: FSYNC() ERR => -1 (Transport endpoint is not > connected) > > Volume configuration - > > Volume Name: gv1 > Type: Distributed-Replicate > Volume ID: 66ad703e-3bae-4e79-a0b7-29ea38e8fcfc > Status: Started > Snapshot Count: 0 > Number of Bricks: 5 x 2 = 10 > Transport-type: tcp > Bricks: > Brick1: dc-vihi44:/gluster/bricks/megabrick/data > Brick2: dc-vihi45:/gluster/bricks/megabrick/data > Brick3: dc-vihi44:/gluster/bricks/brick1/data > Brick4: dc-vihi45:/gluster/bricks/brick1/data > Brick5: dc-vihi44:/gluster/bricks/brick2_1/data > Brick6: dc-vihi45:/gluster/bricks/brick2/data > Brick7: dc-vihi44:/gluster/bricks/brick3/data > Brick8: dc-vihi45:/gluster/bricks/brick3/data > Brick9: dc-vihi44:/gluster/bricks/brick4/data > Brick10: dc-vihi45:/gluster/bricks/brick4/data > Options Reconfigured: > cluster.min-free-inodes: 6% > performance.client-io-threads: off > nfs.disable: on > transport.address-family: inet > performance.quick-read: off > performance.read-ahead: off > performance.io-cache: off > performance.low-prio-threads: 32 > network.remote-dio: enable > cluster.eager-lock: enable > cluster.server-quorum-type: server > cluster.data-self-heal-algorithm: full > cluster.locking-scheme: granular > cluster.shd-max-threads: 8 > cluster.shd-wait-qlength: 1 > user.cifs: off > cluster.choose-local: off > features.shard: on > cluster.server-quorum-ratio: 51% > > -Walter Deignan > -Uline IT, Systems Architect > ___ > Gluster-users mailing list > Gluster-users@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -- *Claus Jeppesen* Manager, Network Services Datto, Inc. p +45 6170 5901 | Copenhagen Office www.datto.com <http://www.datto.com/datto-signature/> ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] Gluster 4.1 servers - KVM Centos 7.5 clients - replica 2 - continuous healing taking place.
We have a replica 2 setup using glusterfs-4.1.1-1 which we use to store VM files. The Gluster clients are Centos 7.5 using gluster "url" setup for the VM disks - e.g. like: ... We have the problem that gluster on the 2 brick server keeps on running healing operations - it's a bit hard to understand why as we cannot see any drops in network traffic between the 2 bricks (either errors or bandwidth issue). - Is there a fundamental versioning conflict ? (libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.5.x86_64 vs. gluster 4.1) - Is there a limit to how many clients a gluster server can handle ? - Some setting we need to adjust ? I hope someone has an idea, Thanx, Claus. -- *Claus Jeppesen* Manager, Network Services Datto, Inc. p +45 6170 5901 | Copenhagen Office www.datto.com <http://www.datto.com/datto-signature/> ___ Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
