Re: [Gluster-users] Stale locks on shards

[Pranith Kumar Karampuri]

On Tue, Jan 23, 2018 at 1:04 PM, Pranith Kumar Karampuri <
pkara...@redhat.com> wrote:

>
>
> On Mon, Jan 22, 2018 at 12:33 AM, Samuli Heinonen 
> wrote:
>
>> Hi again,
>>
>> here is more information regarding issue described earlier
>>
>> It looks like self healing is stuck. According to "heal statistics" crawl
>> began at Sat Jan 20 12:56:19 2018 and it's still going on (It's around Sun
>> Jan 21 20:30 when writing this). However glustershd.log says that last heal
>> was completed at "2018-01-20 11:00:13.090697" (which is 13:00 UTC+2). Also
>> "heal info" has been running now for over 16 hours without any information.
>> In statedump I can see that storage nodes have locks on files and some of
>> those are blocked. Ie. Here again it says that ovirt8z2 is having active
>> lock even ovirt8z2 crashed after the lock was granted.:
>>
>> [xlator.features.locks.zone2-ssd1-vmstor1-locks.inode]
>> path=/.shard/3d55f8cc-cda9-489a-b0a3-fd0f43d67876.27
>> mandatory=0
>> inodelk-count=3
>> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0:self-heal
>> inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid =
>> 18446744073709551610, owner=d0c6d857a87f, client=0x7f885845efa0,
>> connection-id=sto2z2.xxx-10975-2018/01/20-10:56:14:649541-
>> zone2-ssd1-vmstor1-client-0-0-0, granted at 2018-01-20 10:59:52
>> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0:metadata
>> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0
>> inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid =
>> 3420, owner=d8b9372c397f, client=0x7f8858410be0, connection-id=
>> ovirt8z2.xxx.com-5652-2017/12/27-09:49:02:9468
>> 25-zone2-ssd1-vmstor1-client-0-7-0, granted at 2018-01-20 08:57:23
>> inodelk.inodelk[1](BLOCKED)=type=WRITE, whence=0, start=0, len=0, pid =
>> 18446744073709551610, owner=d0c6d857a87f, client=0x7f885845efa0,
>> connection-id=sto2z2.xxx-10975-2018/01/20-10:56:14:649541-
>> zone2-ssd1-vmstor1-client-0-0-0, blocked at 2018-01-20 10:59:52
>>
>> I'd also like to add that volume had arbiter brick before crash happened.
>> We decided to remove it because we thought that it was causing issues.
>> However now I think that this was unnecessary. After the crash arbiter logs
>> had lots of messages like this:
>> [2018-01-20 10:19:36.515717] I [MSGID: 115072]
>> [server-rpc-fops.c:1640:server_setattr_cbk] 0-zone2-ssd1-vmstor1-server:
>> 37374187: SETATTR 
>> (a52055bd-e2e9-42dd-92a3-e96b693bcafe) ==> (Operation not permitted)
>> [Operation not permitted]
>>
>> Is there anyways to force self heal to stop? Any help would be very much
>> appreciated :)
>>
>
> Exposing .shard to a normal mount is opening a can of worms. You should
> probably look at mounting the volume with gfid aux-mount where you can
> access a file with /.gfid/to clear locks on
> it.
>

Please use this mount only for doing just this work and unmount it after
that. But my recommendation would be to do an upgrade as soon as possible.
Your bricks will crash on the next disconnect from 'sto2z2.xxx' if you are
not lucky.


>
> Mount command:  mount -t glusterfs -o aux-gfid-mount vm1:test /mnt/testvol
>
> A gfid string will have some hyphens like: 
> 8443-1894-4273-9340-4b212fa1c0e4
>
> That said. Next disconnect on the brick where you successfully did the
> clear-locks will crash the brick. There was a bug in 3.8.x series with
> clear-locks which was fixed in 3.9.0 with a feature. The self-heal
> deadlocks that you witnessed also is fixed in 3.10 version of the release.
>
> 3.8.x is EOLed, so I recommend you to upgrade to a supported version soon.
>
>
>>
>> Best regards,
>> Samuli Heinonen
>>
>>
>>
>>
>>
>> Samuli Heinonen 
>> 20 January 2018 at 21.57
>> Hi all!
>>
>> One hypervisor on our virtualization environment crashed and now some of
>> the VM images cannot be accessed. After investigation we found out that
>> there was lots of images that still had active lock on crashed hypervisor.
>> We were able to remove locks from "regular files", but it doesn't seem
>> possible to remove locks from shards.
>>
>> We are running GlusterFS 3.8.15 on all nodes.
>>
>> Here is part of statedump that shows shard having active lock on crashed
>> node:
>> [xlator.features.locks.zone2-ssd1-vmstor1-locks.inode]
>> path=/.shard/75353c17-d6b8-485d-9baf-fd6c700e39a1.21
>> mandatory=0
>> inodelk-count=1
>> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0:metadata
>> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0:self-heal
>> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0
>> inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid =
>> 3568, owner=14ce372c397f, client=0x7f3198388770, connection-id
>> ovirt8z2.xxx-5652-2017/12/27-09:49:02:946825-zone2-ssd1-vmstor1-client-1-7-0,
>> granted at 2018-01-20 08:57:24
>>
>> If we try to run clear-locks we get following error message:
>> # gluster volume clear-locks zone2-ssd1-vmstor1
>> 

Re: [Gluster-users] Stale locks on shards

[Krutika Dhananjay]

On Mon, Jan 22, 2018 at 12:33 AM, Samuli Heinonen 
wrote:

> Hi again,
>
> here is more information regarding issue described earlier
>
> It looks like self healing is stuck. According to "heal statistics" crawl
> began at Sat Jan 20 12:56:19 2018 and it's still going on (It's around Sun
> Jan 21 20:30 when writing this). However glustershd.log says that last heal
> was completed at "2018-01-20 11:00:13.090697" (which is 13:00 UTC+2). Also
> "heal info" has been running now for over 16 hours without any information.
> In statedump I can see that storage nodes have locks on files and some of
> those are blocked. Ie. Here again it says that ovirt8z2 is having active
> lock even ovirt8z2 crashed after the lock was granted.:
>
> [xlator.features.locks.zone2-ssd1-vmstor1-locks.inode]
> path=/.shard/3d55f8cc-cda9-489a-b0a3-fd0f43d67876.27
> mandatory=0
> inodelk-count=3
> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0:self-heal
> inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid =
> 18446744073709551610, owner=d0c6d857a87f, client=0x7f885845efa0,
> connection-id=sto2z2.xxx-10975-2018/01/20-10:56:14:
> 649541-zone2-ssd1-vmstor1-client-0-0-0, granted at 2018-01-20 10:59:52
> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0:metadata
> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0
> inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid =
> 3420, owner=d8b9372c397f, client=0x7f8858410be0,
> connection-id=ovirt8z2.xxx.com-5652-2017/12/27-09:49:02:
> 946825-zone2-ssd1-vmstor1-client-0-7-0, granted at 2018-01-20 08:57:23
> inodelk.inodelk[1](BLOCKED)=type=WRITE, whence=0, start=0, len=0, pid =
> 18446744073709551610, owner=d0c6d857a87f, client=0x7f885845efa0,
> connection-id=sto2z2.xxx-10975-2018/01/20-10:56:14:
> 649541-zone2-ssd1-vmstor1-client-0-0-0, blocked at 2018-01-20 10:59:52
>
> I'd also like to add that volume had arbiter brick before crash happened.
> We decided to remove it because we thought that it was causing issues.
> However now I think that this was unnecessary. After the crash arbiter logs
> had lots of messages like this:
> [2018-01-20 10:19:36.515717] I [MSGID: 115072] 
> [server-rpc-fops.c:1640:server_setattr_cbk]
> 0-zone2-ssd1-vmstor1-server: 37374187: SETATTR
>  
> (a52055bd-e2e9-42dd-92a3-e96b693bcafe)
> ==> (Operation not permitted) [Operation not permitted]
>
> Is there anyways to force self heal to stop? Any help would be very much
> appreciated :)
>

The locks are contending in afr self-heal and data path domains. It's
possible that the deadlock is not caused by the hypervisor as if that were
the case, the locks should have been released when it crashed/disconnected.

Adding AFR devs to check what's causing the deadlock in the first place.

-Krutika



>
> Best regards,
> Samuli Heinonen
>
>
>
>
>
> Samuli Heinonen 
> 20 January 2018 at 21.57
> Hi all!
>
> One hypervisor on our virtualization environment crashed and now some of
> the VM images cannot be accessed. After investigation we found out that
> there was lots of images that still had active lock on crashed hypervisor.
> We were able to remove locks from "regular files", but it doesn't seem
> possible to remove locks from shards.
>
> We are running GlusterFS 3.8.15 on all nodes.
>
> Here is part of statedump that shows shard having active lock on crashed
> node:
> [xlator.features.locks.zone2-ssd1-vmstor1-locks.inode]
> path=/.shard/75353c17-d6b8-485d-9baf-fd6c700e39a1.21
> mandatory=0
> inodelk-count=1
> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0:metadata
> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0:self-heal
> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0
> inodelk.inodelk[0](ACTIVE)=type=WRITE, whence=0, start=0, len=0, pid =
> 3568, owner=14ce372c397f, client=0x7f3198388770, connection-id
> ovirt8z2.xxx-5652-2017/12/27-09:49:02:946825-zone2-ssd1-vmstor1-client-1-7-0,
> granted at 2018-01-20 08:57:24
>
> If we try to run clear-locks we get following error message:
> # gluster volume clear-locks zone2-ssd1-vmstor1 
> /.shard/75353c17-d6b8-485d-9baf-fd6c700e39a1.21
> kind all inode
> Volume clear-locks unsuccessful
> clear-locks getxattr command failed. Reason: Operation not permitted
>
> Gluster vol info if needed:
> Volume Name: zone2-ssd1-vmstor1
> Type: Replicate
> Volume ID: b6319968-690b-4060-8fff-b212d2295208
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 2 = 2
> Transport-type: rdma
> Bricks:
> Brick1: sto1z2.xxx:/ssd1/zone2-vmstor1/export
> Brick2: sto2z2.xxx:/ssd1/zone2-vmstor1/export
> Options Reconfigured:
> cluster.shd-wait-qlength: 1
> cluster.shd-max-threads: 8
> cluster.locking-scheme: granular
> performance.low-prio-threads: 32
> cluster.data-self-heal-algorithm: full
> performance.client-io-threads: off
> storage.linux-aio: off
> performance.readdir-ahead: on
> client.event-threads: 16
> server.event-threads: 16
> 

Re: [Gluster-users] [Possibile SPAM] Re: Strange messages in mnt-xxx.log

[Nithya Balachandran]

On 17 January 2018 at 16:04, Ing. Luca Lazzeroni - Trend Servizi Srl <
l...@trendservizi.it> wrote:

> Here's the volume info:
>
>
> Volume Name: gv2a2
> Type: Replicate
> Volume ID: 83c84774-2068-4bfc-b0b9-3e6b93705b9f
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x (2 + 1) = 3
> Transport-type: tcp
> Bricks:
> Brick1: gluster1:/bricks/brick2/gv2a2
> Brick2: gluster3:/bricks/brick3/gv2a2
> Brick3: gluster2:/bricks/arbiter_brick_gv2a2/gv2a2 (arbiter)
> Options Reconfigured:
> storage.owner-gid: 107
> storage.owner-uid: 107
> user.cifs: off
> features.shard: on
> cluster.shd-wait-qlength: 1
> cluster.shd-max-threads: 8
> cluster.locking-scheme: granular
> cluster.data-self-heal-algorithm: full
> cluster.server-quorum-type: server
> cluster.quorum-type: auto
> cluster.eager-lock: enable
> network.remote-dio: enable
> performance.low-prio-threads: 32
> performance.io-cache: off
> performance.read-ahead: off
> performance.quick-read: off
> transport.address-family: inet
> nfs.disable: off
> performance.client-io-threads: off
>
> The only client I'm using is FUSE client used to mount the gluster volume.
> On the gluster volume there is a maximum of 15 files ("big" files because
> they host VM images) accessed by qemu-kvm as normal files on the FUSE
> mounted volume.
>
> By inspecting the code I've found that the message is logged in 2
> situations:
>
> 1) A real "Hole" in DHT
>
> 2) A "virgin" file being created
>
> I think this is the second situation because that message appears only
> when I create a new qcow2 volume to host VM image.
>
>  These messages should ideally be seen only for directories and I have
never seen it with a null gfid so far. I'll try to reproduce this and get
back to you.

Regards,
Nithya


> Il 17/01/2018 04:54, Nithya Balachandran ha scritto:
>
> Hi,
>
>
> On 16 January 2018 at 18:56, Ing. Luca Lazzeroni - Trend Servizi Srl <
> l...@trendservizi.it> wrote:
>
>> Hi,
>>
>> I'm testing gluster 3.12.4 and, by inspecting log files
>> /var/log/glusterfs/mnt-gv0.log (gv0 is the volume name), I found many lines
>> saying:
>>
>> [2018-01-15 09:45:41.066914] I [MSGID: 109063]
>> [dht-layout.c:716:dht_layout_normalize] 0-gv0-dht: Found anomalies in
>> (null) (gfid = ----). Holes=1 overlaps=0
>> [2018-01-15 09:45:45.755021] I [MSGID: 109063]
>> [dht-layout.c:716:dht_layout_normalize] 0-gv0-dht: Found anomalies in
>> (null) (gfid = ----). Holes=1 overlaps=0
>> [2018-01-15 14:02:29.171437] I [MSGID: 109063]
>> [dht-layout.c:716:dht_layout_normalize] 0-gv0-dht: Found anomalies in
>> (null) (gfid = ----). Holes=1 overlaps=0
>>
>> What do they mean ? Is there any real problem ?
>>
>>
> Please provide the following details:
> gluster volume info
> what clients you are using and what operations being performed
> Any steps to reproduce this issue.
>
> Thanks,
> Nithya
>
> Thank you,
>>
>>
>> --
>> Ing. Luca Lazzeroni
>> Responsabile Ricerca e Sviluppo
>> Trend Servizi Srl
>> Tel: 0376/631761
>> Web: https://www.trendservizi.it
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
> --
> Ing. Luca Lazzeroni
> Responsabile Ricerca e Sviluppo
> Trend Servizi Srl
> Tel: 0376/631761
> Web: https://www.trendservizi.it
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] BUG: After stop and start wrong port is advertised

[Atin Mukherjee]

So from the logs what it looks to be a regression caused by commit 635c1c3
( and the good news is that this is now fixed in release-3.12 branch and
should be part of 3.12.5.

Commit which fixes this issue:

COMMIT: https://review.gluster.org/19146 committed in release-3.12 by
\"Atin Mukherjee\"  with a commit message-
glusterd: connect to an existing brick process when qourum status is
NOT_APPLICABLE_QUORUM

First of all, this patch reverts commit 635c1c3 as the same is causing a
regression with bricks not coming up on time when a node is rebooted.
This patch tries to fix the problem in a different way by just trying to
connect to an existing running brick when quorum status is not
applicable.
>mainline patch : https://review.gluster.org/#/c/19134/

Change-Id: I0efb5901832824b1c15dcac529bffac85173e097
BUG: 1511301
Signed-off-by: Atin Mukherjee 




On Mon, Jan 22, 2018 at 3:15 PM, Alan Orth  wrote:

> Ouch! Yes, I see two port-related fixes in the GlusterFS 3.12.3 release
> notes[0][1][2]. I've attached a tarball of all yesterday's logs from
> /var/log/glusterd on one the affected nodes (called "wingu3"). I hope
> that's what you need.
>
> [0] https://github.com/gluster/glusterfs/blob/release-3.12/
> doc/release-notes/3.12.3.md
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1507747
> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1507748
>
> Thanks,
>
>
> On Mon, Jan 22, 2018 at 6:34 AM Atin Mukherjee 
> wrote:
>
>> The patch was definitely there in 3.12.3. Do you have the glusterd and
>> brick logs handy with you when this happened?
>>
>> On Sun, Jan 21, 2018 at 10:21 PM, Alan Orth  wrote:
>>
>>> For what it's worth, I just updated some CentOS 7 servers from GlusterFS
>>> 3.12.1 to 3.12.4 and hit this bug. Did the patch make it into 3.12.4? I had
>>> to use Mike Hulsman's script to check the daemon port against the port in
>>> the volume's brick info, update the port, and restart glusterd on each
>>> node. Luckily I only have four servers! Hoping I don't have to do this
>>> every time I reboot!
>>>
>>> Regards,
>>>
>>> On Sat, Dec 2, 2017 at 5:23 PM Atin Mukherjee 
>>> wrote:
>>>
 On Sat, 2 Dec 2017 at 19:29, Jo Goossens 
 wrote:

> Hello Atin,
>
>
>
>
>
> Could you confirm this should have been fixed in 3.10.8? If so we'll
> test it for sure!
>

 Fix should be part of 3.10.8 which is awaiting release announcement.


>
> Regards
>
> Jo
>
>
>
>
>
>
> -Original message-
> *From:* Atin Mukherjee 
>
> *Sent:* Mon 30-10-2017 17:40
> *Subject:* Re: [Gluster-users] BUG: After stop and start wrong port
> is advertised
> *To:* Jo Goossens ;
> *CC:* gluster-users@gluster.org;
>
> On Sat, 28 Oct 2017 at 02:36, Jo Goossens <
> jo.gooss...@hosted-power.com> wrote:
>
> Hello Atin,
>
>
>
>
>
> I just read it and very happy you found the issue. We really hope this
> will be fixed in the next 3.10.7 version!
>
>
> 3.10.7 - no I guess as the patch is still in review and 3.10.7 is
> getting tagged today. You’ll get this fix in 3.10.8.
>
>
>
>
>
>
>
>
> PS: Wow nice all that c code and those "goto out" statements (not
> always considered clean but the best way often I think). Can remember the
> days I wrote kernel drivers myself in c :)
>
>
>
>
>
> Regards
>
> Jo Goossens
>
>
>
>
>
>
>
>
> -Original message-
> *From:* Atin Mukherjee 
> *Sent:* Fri 27-10-2017 21:01
> *Subject:* Re: [Gluster-users] BUG: After stop and start wrong port
> is advertised
> *To:* Jo Goossens ;
> *CC:* gluster-users@gluster.org;
>
> We (finally) figured out the root cause, Jo!
>
> Patch https://review.gluster.org/#/c/18579 posted upstream for review.
>
> On Thu, Sep 21, 2017 at 2:08 PM, Jo Goossens <
> jo.gooss...@hosted-power.com> wrote:
>
> Hi,
>
>
>
>
>
> We use glusterfs 3.10.5 on Debian 9.
>
>
>
> When we stop or restart the service, e.g.: service glusterfs-server
> restart
>
>
>
> We see that the wrong port get's advertised afterwards. For example:
>
>
>
> Before restart:
>
>
> Status of volume: public
> Gluster process TCP Port  RDMA Port
>  Online  Pid
> 
> --
> Brick 192.168.140.41:/gluster/public49153 0  Y
> 6364
> Brick 192.168.140.42:/gluster/public

Re: [Gluster-users] Segfaults after upgrade to GlusterFS 3.10.9

[Frank Wall]

On Fri, Jan 19, 2018 at 08:19:09PM +, Niklas Hambüchen wrote:
> What's /proc/sys/kernel/core_pattern set to for you? For me it is
> 
> % cat /proc/sys/kernel/core_pattern
> core
> 
> which will drop a core file in the working directory of the process.

Same here, still I'm unable to find any core files.

I've reverted to 3.10.8, because GlusterFS was close to unusable.


Regards
- Frank
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster endless heal

[Mahdi Adnan]

Okay so i've found that one of the hypervisors was only connected to 3 gluster 
nodes instead of 4.
is there a way to tell which nodes glusterfs client is connected to ? or list 
clients from glusterfs server ?


--

Respectfully
Mahdi A. Mahdi


From: gluster-users-boun...@gluster.org  on 
behalf of Mahdi Adnan 
Sent: Wednesday, January 17, 2018 9:50 PM
To: gluster-users@gluster.org
Subject: [Gluster-users] Gluster endless heal

Hi,

I have an issue with Gluster 3.8.14.
The cluster is 4 nodes with replica count 2, on of the nodes went offline for 
around 15 minutes, when it came back online, self heal triggered and it just 
did not stop afterward, it's been running for 3 days now, maxing the bricks 
utilization without actually healing anything.
The bricks are all SSDs, and the logs of the source node is spamming with the 
following messages;

[2018-01-17 18:37:11.815247] I [MSGID: 108026] 
[afr-self-heal-common.c:1254:afr_log_selfheal] 0-ovirt_imgs-replicate-0: 
Completed data selfheal on 450fb07a-e95d-48ef-a229-48917557c278. sources=[0]  
sinks=1
[2018-01-17 18:37:12.830887] I [MSGID: 108026] 
[afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do] 
0-ovirt_imgs-replicate-0: performing metadata selfheal on 
ce0f545d-635a-40c0-95eb-ccfc71971f78
[2018-01-17 18:37:12.845978] I [MSGID: 108026] 
[afr-self-heal-common.c:1254:afr_log_selfheal] 0-ovirt_imgs-replicate-0: 
Completed metadata selfheal on ce0f545d-635a-40c0-95eb-ccfc71971f78. 
sources=[0]  sinks=1

---

I tried restarting glusterd and rebooting the node after about 24 hours of 
healing, but it just did not help, i had like several bricks doing heal and 
after rebooting it's now only 4 bricks doing heal.

The volume is used for oVirt storage domain with sharding enabled.
No errors or warnings on both nodes, just info messages about afr healing.

any idea whats going on or where should i start looking ?

--

Respectfully
Mahdi A. Mahdi

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users