On Tue, Jan 16, 2018 at 10:47 PM, Ing. Luca Lazzeroni - Trend Servizi Srl < [email protected]> wrote:
> I've made the test with raw image format (preallocated too) and the > corruption problem is still there (but without errors in bricks' log file). > > What does the "link" error in bricks log files means ? > > I've seen the source code looking for the lines where it happens and it > seems a warning (it doesn't imply a failure). > Indeed, it only represents a transient state when the shards are created for the first time and does not indicate a failure. Could you also get the logs of the gluster fuse mount process? It should be under /var/log/glusterfs of your client machine with the filename as a hyphenated mount point path. For example, if your volume was mounted at /mnt/glusterfs, then your log file would be named mnt-glusterfs.log. -Krutika > > Il 16/01/2018 17:39, Ing. Luca Lazzeroni - Trend Servizi Srl ha scritto: > > An update: > > I've tried, for my tests, to create the vm volume as > > qemu-img create -f qcow2 -o preallocation=full > gluster://gluster1/Test/Test-vda.img 20G > > et voila ! > > No errors at all, neither in bricks' log file (the "link failed" message > disappeared), neither in VM (no corruption and installed succesfully). > > I'll do another test with a fully preallocated raw image. > > > > Il 16/01/2018 16:31, Ing. Luca Lazzeroni - Trend Servizi Srl ha scritto: > > I've just done all the steps to reproduce the problem. > > Tha VM volume has been created via "qemu-img create -f qcow2 > Test-vda2.qcow2 20G" on the gluster volume mounted via FUSE. I've tried > also to create the volume with preallocated metadata, which moves the > problem a bit far away (in time). The volume is a replice 3 arbiter 1 > volume hosted on XFS bricks. > > Here are the informations: > > [root@ovh-ov1 bricks]# gluster volume info gv2a2 > > Volume Name: gv2a2 > Type: Replicate > Volume ID: 83c84774-2068-4bfc-b0b9-3e6b93705b9f > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x (2 + 1) = 3 > Transport-type: tcp > Bricks: > Brick1: gluster1:/bricks/brick2/gv2a2 > Brick2: gluster3:/bricks/brick3/gv2a2 > Brick3: gluster2:/bricks/arbiter_brick_gv2a2/gv2a2 (arbiter) > Options Reconfigured: > storage.owner-gid: 107 > storage.owner-uid: 107 > user.cifs: off > features.shard: on > cluster.shd-wait-qlength: 10000 > cluster.shd-max-threads: 8 > cluster.locking-scheme: granular > cluster.data-self-heal-algorithm: full > cluster.server-quorum-type: server > cluster.quorum-type: auto > cluster.eager-lock: enable > network.remote-dio: enable > performance.low-prio-threads: 32 > performance.io-cache: off > performance.read-ahead: off > performance.quick-read: off > transport.address-family: inet > nfs.disable: off > performance.client-io-threads: off > > /var/log/glusterfs/glusterd.log: > > [2018-01-15 14:17:50.196228] I [MSGID: 106488] [glusterd-handler.c:1548:__ > glusterd_handle_cli_get_volume] 0-management: Received get vol req > [2018-01-15 14:25:09.555214] I [MSGID: 106488] [glusterd-handler.c:1548:__ > glusterd_handle_cli_get_volume] 0-management: Received get vol req > > (empty because today it's 2018-01-16) > > /var/log/glusterfs/glustershd.log: > > [2018-01-14 02:23:02.731245] I [glusterfsd-mgmt.c:1821:mgmt_getspec_cbk] > 0-glusterfs: No change in volfile,continuing > > (empty too) > > /var/log/glusterfs/bricks/brick-brick2-gv2a2.log (the interested volume): > > [2018-01-16 15:14:37.809965] I [MSGID: 115029] > [server-handshake.c:793:server_setvolume] > 0-gv2a2-server: accepted client from ovh-ov1-10302-2018/01/16-15: > 14:37:790306-gv2a2-client-0-0-0 (version: 3.12.4) > [2018-01-16 15:16:41.471751] E [MSGID: 113020] [posix.c:1485:posix_mknod] > 0-gv2a2-posix: setting gfid on /bricks/brick2/gv2a2/.shard/ > 62335cb9-c7b5-4735-a879-59cff93fe622.4 failed > [2018-01-16 15:16:41.471745] W [MSGID: 113096] > [posix-handle.c:770:posix_handle_hard] > 0-gv2a2-posix: link > /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4 > -> > /bricks/brick2/gv2a2/.glusterfs/a0/14/a0144df3-8d89-4aed-872e-5fef141e9e1efailed > [File exists] > [2018-01-16 15:16:42.593392] W [MSGID: 113096] > [posix-handle.c:770:posix_handle_hard] > 0-gv2a2-posix: link > /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5 > -> > /bricks/brick2/gv2a2/.glusterfs/eb/04/eb044e6e-3a23-40a4-9ce1-f13af148eb67failed > [File exists] > [2018-01-16 15:16:42.593426] E [MSGID: 113020] [posix.c:1485:posix_mknod] > 0-gv2a2-posix: setting gfid on /bricks/brick2/gv2a2/.shard/ > 62335cb9-c7b5-4735-a879-59cff93fe622.5 failed > [2018-01-16 15:17:04.129593] W [MSGID: 113096] > [posix-handle.c:770:posix_handle_hard] > 0-gv2a2-posix: link > /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8 > -> > /bricks/brick2/gv2a2/.glusterfs/dc/92/dc92bd0a-0d46-4826-a4c9-d073a924dd8dfailed > [File exists] > The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] > 0-gv2a2-posix: link > /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.8 > -> > /bricks/brick2/gv2a2/.glusterfs/dc/92/dc92bd0a-0d46-4826-a4c9-d073a924dd8dfailed > [File exists]" repeated 5 times between [2018-01-16 15:17:04.129593] and > [2018-01-16 15:17:04.129593] > [2018-01-16 15:17:04.129661] E [MSGID: 113020] [posix.c:1485:posix_mknod] > 0-gv2a2-posix: setting gfid on /bricks/brick2/gv2a2/.shard/ > 62335cb9-c7b5-4735-a879-59cff93fe622.8 failed > [2018-01-16 15:17:08.279162] W [MSGID: 113096] > [posix-handle.c:770:posix_handle_hard] > 0-gv2a2-posix: link > /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9 > -> > /bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed > [File exists] > [2018-01-16 15:17:08.279162] W [MSGID: 113096] > [posix-handle.c:770:posix_handle_hard] > 0-gv2a2-posix: link > /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9 > -> > /bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed > [File exists] > The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] > 0-gv2a2-posix: link > /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.9 > -> > /bricks/brick2/gv2a2/.glusterfs/c9/b7/c9b71b00-a09f-4df1-b874-041820ca8241failed > [File exists]" repeated 2 times between [2018-01-16 15:17:08.279162] and > [2018-01-16 15:17:08.279162] > > [2018-01-16 15:17:08.279177] E [MSGID: 113020] [posix.c:1485:posix_mknod] > 0-gv2a2-posix: setting gfid on /bricks/brick2/gv2a2/.shard/ > 62335cb9-c7b5-4735-a879-59cff93fe622.9 failed > The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] > 0-gv2a2-posix: link > /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.4 > -> > /bricks/brick2/gv2a2/.glusterfs/a0/14/a0144df3-8d89-4aed-872e-5fef141e9e1efailed > [File exists]" repeated 6 times between [2018-01-16 15:16:41.471745] and > [2018-01-16 15:16:41.471807] > The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] > 0-gv2a2-posix: link > /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.5 > -> > /bricks/brick2/gv2a2/.glusterfs/eb/04/eb044e6e-3a23-40a4-9ce1-f13af148eb67failed > [File exists]" repeated 2 times between [2018-01-16 15:16:42.593392] and > [2018-01-16 15:16:42.593430] > [2018-01-16 15:17:32.229689] W [MSGID: 113096] > [posix-handle.c:770:posix_handle_hard] > 0-gv2a2-posix: link > /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14 > -> > /bricks/brick2/gv2a2/.glusterfs/53/04/530449fa-d698-4928-a262-9a0234232323failed > [File exists] > [2018-01-16 15:17:32.229720] E [MSGID: 113020] [posix.c:1485:posix_mknod] > 0-gv2a2-posix: setting gfid on /bricks/brick2/gv2a2/.shard/ > 62335cb9-c7b5-4735-a879-59cff93fe622.14 failed > [2018-01-16 15:18:07.154330] W [MSGID: 113096] > [posix-handle.c:770:posix_handle_hard] > 0-gv2a2-posix: link > /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17 > -> > /bricks/brick2/gv2a2/.glusterfs/81/96/8196dd19-84bc-4c3d-909f-8792e9b4929dfailed > [File exists] > [2018-01-16 15:18:07.154375] E [MSGID: 113020] [posix.c:1485:posix_mknod] > 0-gv2a2-posix: setting gfid on /bricks/brick2/gv2a2/.shard/ > 62335cb9-c7b5-4735-a879-59cff93fe622.17 failed > The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] > 0-gv2a2-posix: link > /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.14 > -> > /bricks/brick2/gv2a2/.glusterfs/53/04/530449fa-d698-4928-a262-9a0234232323failed > [File exists]" repeated 7 times between [2018-01-16 15:17:32.229689] and > [2018-01-16 15:17:32.229806] > The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] > 0-gv2a2-posix: link > /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.17 > -> > /bricks/brick2/gv2a2/.glusterfs/81/96/8196dd19-84bc-4c3d-909f-8792e9b4929dfailed > [File exists]" repeated 3 times between [2018-01-16 15:18:07.154330] and > [2018-01-16 15:18:07.154357] > [2018-01-16 15:19:23.618794] W [MSGID: 113096] > [posix-handle.c:770:posix_handle_hard] > 0-gv2a2-posix: link > /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21 > -> > /bricks/brick2/gv2a2/.glusterfs/6d/02/6d02bd98-83de-43e8-a7af-b1d5f5160403failed > [File exists] > [2018-01-16 15:19:23.618827] E [MSGID: 113020] [posix.c:1485:posix_mknod] > 0-gv2a2-posix: setting gfid on /bricks/brick2/gv2a2/.shard/ > 62335cb9-c7b5-4735-a879-59cff93fe622.21 failed > The message "W [MSGID: 113096] [posix-handle.c:770:posix_handle_hard] > 0-gv2a2-posix: link > /bricks/brick2/gv2a2/.shard/62335cb9-c7b5-4735-a879-59cff93fe622.21 > -> > /bricks/brick2/gv2a2/.glusterfs/6d/02/6d02bd98-83de-43e8-a7af-b1d5f5160403failed > [File exists]" repeated 3 times between [2018-01-16 15:19:23.618794] and > [2018-01-16 15:19:23.618794] > > Thank you, > > Il 16/01/2018 11:40, Krutika Dhananjay ha scritto: > > Also to help isolate the component, could you answer these: > > 1. on a different volume with shard not enabled, do you see this issue? > 2. on a plain 3-way replicated volume (no arbiter), do you see this issue? > > > > On Tue, Jan 16, 2018 at 4:03 PM, Krutika Dhananjay <[email protected]> > wrote: > >> Please share the volume-info output and the logs under >> /var/log/glusterfs/ from all your nodes. for investigating the issue. >> >> -Krutika >> >> On Tue, Jan 16, 2018 at 1:30 PM, Ing. Luca Lazzeroni - Trend Servizi Srl >> <[email protected]> wrote: >> >>> Hi to everyone. >>> >>> I've got a strange problem with a gluster setup: 3 nodes with Centos >>> 7.4, Gluster 3.12.4 from Centos/Gluster repositories, QEMU-KVM version >>> 2.9.0 (compiled from RHEL sources). >>> >>> I'm running volumes in replica 3 arbiter 1 mode (but I've got a volume >>> in "pure" replica 3 mode too). I've applied the "virt" group settings to my >>> volumes since they host VM images. >>> >>> If I try to install something (eg: Ubuntu Server 16.04.3) on a VM (and >>> so I generate a bit of I/O inside it) and configure KVM to access gluster >>> volume directly (via libvirt), install fails after a while because the disk >>> content is corrupted. If I inspect the block inside the disk (by accessing >>> the image directly from outside) I can found many files filled with "^@". >>> >> > Also, what exactly do you mean by accessing the image directly from > outside? Was it from the brick directories directly? Was it from the mount > point of the volume? Could you elaborate? Which files exactly did you check? > > -Krutika > > >>> If, instead, I configure KVM to access VM images via a FUSE mount, >>> everything seems to work correctly. >>> >>> Note that the problem with install is verified 100% time with QCOW2 >>> image, while it appears only after with RAW disk images. >>> >>> Is there anyone who experienced the same problem ? >>> >>> Thank you, >>> >>> >>> -- >>> Ing. Luca Lazzeroni >>> Responsabile Ricerca e Sviluppo >>> Trend Servizi Srl >>> Tel: 0376/631761 >>> Web: https://www.trendservizi.it >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> [email protected] >>> http://lists.gluster.org/mailman/listinfo/gluster-users >>> >> >> > > -- > Ing. Luca Lazzeroni > Responsabile Ricerca e Sviluppo > Trend Servizi Srl > Tel: 0376/631761 > Web: https://www.trendservizi.it > > > > _______________________________________________ > Gluster-users mailing > [email protected]http://lists.gluster.org/mailman/listinfo/gluster-users > > > -- > Ing. Luca Lazzeroni > Responsabile Ricerca e Sviluppo > Trend Servizi Srl > Tel: 0376/631761 > Web: https://www.trendservizi.it > > > > _______________________________________________ > Gluster-users mailing > [email protected]http://lists.gluster.org/mailman/listinfo/gluster-users > > > -- > Ing. Luca Lazzeroni > Responsabile Ricerca e Sviluppo > Trend Servizi Srl > Tel: 0376/631761 > Web: https://www.trendservizi.it > > > _______________________________________________ > Gluster-users mailing list > [email protected] > http://lists.gluster.org/mailman/listinfo/gluster-users >
_______________________________________________ Gluster-users mailing list [email protected] http://lists.gluster.org/mailman/listinfo/gluster-users
