Re: [Gluster-users] Gluster 3.10.5: used disk size reported by quota and du mismatch

[Sanoj Unnikrishnan]

Hi Mauro,

A script issue did not capture all necessary xattr.
Could you provide the xattrs with..
find /tier2/CSP/ans004  | xargs getfattr -d -m. -e hex

Meanwhile, If you are being impacted, you could do the following
back up quota limits
disable quota
enable quota
freshly set the limits.

Please capture the xattr values first, so that we can get to know what went
wrong.
Regards,
Sanoj


On Tue, Jul 3, 2018 at 4:09 PM, Mauro Tridici  wrote:

> Dear Sanoj,
>
> thank you very much for your support.
> I just downloaded and executed the script you suggested.
>
> This is the full command I executed:
>
> ./quota_fsck_new.py --full-logs --sub-dir /tier2/CSP/ans004/ /gluster
>
> In attachment, you can find the logs generated by the script.
> What can I do now?
>
> Thank you very much for your patience.
> Mauro
>
>
>
>
> Il giorno 03 lug 2018, alle ore 11:34, Sanoj Unnikrishnan <
> sunni...@redhat.com> ha scritto:
>
> Hi Mauro,
>
> This may be an issue with update of backend xattrs.
> To RCA further and provide resolution could you provide me with the logs
> by running the following fsck script.
> https://review.gluster.org/#/c/19179/6/extras/quota/quota_fsck.py
>
> Try running the script and revert with the logs generated.
>
> Thanks,
> Sanoj
>
>
> On Mon, Jul 2, 2018 at 2:21 PM, Mauro Tridici 
> wrote:
>
>> Dear Users,
>>
>> I just noticed that, after some data deletions executed inside
>> "/tier2/CSP/ans004” folder, the amount of used disk reported by quota
>> command doesn’t reflect the value indicated by du command.
>> Surfing on the web, it seems that it is a bug of previous versions of
>> Gluster FS and it was already fixed.
>> In my case, the problem seems unfortunately still here.
>>
>> How can I solve this issue? Is it possible to do it without starting a
>> downtime period?
>>
>> Thank you very much in advance,
>> Mauro
>>
>> [root@s01 ~]# glusterfs -V
>> glusterfs 3.10.5
>> Repository revision: git://git.gluster.org/glusterfs.git
>> Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
>> GlusterFS comes with ABSOLUTELY NO WARRANTY.
>> It is licensed to you under your choice of the GNU Lesser
>> General Public License, version 3 or any later version (LGPLv3
>> or later), or the GNU General Public License, version 2 (GPLv2),
>> in all cases as published by the Free Software Foundation.
>>
>> [root@s01 ~]# gluster volume quota tier2 list /CSP/ans004
>>   Path   Hard-limit  Soft-limit
>> Used  Available  Soft-limit exceeded? Hard-limit exceeded?
>> 
>> ---
>> /CSP/ans0041.0TB 99%(1013.8GB)
>> *3.9TB*  0Bytes Yes  Yes
>>
>> [root@s01 ~]# du -hs /tier2/CSP/ans004/
>> *295G* /tier2/CSP/ans004/
>>
>>
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
>
> -
> Mauro Tridici
>
> Fondazione CMCC
> CMCC Supercomputing Center
> presso Complesso Ecotekne - Università del Salento -
> Strada Prov.le Lecce - Monteroni sn
> 73100 Lecce  IT
> http://www.cmcc.it
>
> mobile: (+39) 327 5630841
> email: mauro.trid...@cmcc.it
>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster 3.10.5: used disk size reported by quota and du mismatch

[Sanoj Unnikrishnan]

Hi Mauro,

This may be an issue with update of backend xattrs.
To RCA further and provide resolution could you provide me with the logs by
running the following fsck script.
https://review.gluster.org/#/c/19179/6/extras/quota/quota_fsck.py

Try running the script and revert with the logs generated.

Thanks,
Sanoj


On Mon, Jul 2, 2018 at 2:21 PM, Mauro Tridici  wrote:

> Dear Users,
>
> I just noticed that, after some data deletions executed inside
> "/tier2/CSP/ans004” folder, the amount of used disk reported by quota
> command doesn’t reflect the value indicated by du command.
> Surfing on the web, it seems that it is a bug of previous versions of
> Gluster FS and it was already fixed.
> In my case, the problem seems unfortunately still here.
>
> How can I solve this issue? Is it possible to do it without starting a
> downtime period?
>
> Thank you very much in advance,
> Mauro
>
> [root@s01 ~]# glusterfs -V
> glusterfs 3.10.5
> Repository revision: git://git.gluster.org/glusterfs.git
> Copyright (c) 2006-2016 Red Hat, Inc. 
> GlusterFS comes with ABSOLUTELY NO WARRANTY.
> It is licensed to you under your choice of the GNU Lesser
> General Public License, version 3 or any later version (LGPLv3
> or later), or the GNU General Public License, version 2 (GPLv2),
> in all cases as published by the Free Software Foundation.
>
> [root@s01 ~]# gluster volume quota tier2 list /CSP/ans004
>   Path   Hard-limit  Soft-limit  Used
> Available  Soft-limit exceeded? Hard-limit exceeded?
> 
> ---
> /CSP/ans0041.0TB 99%(1013.8GB)
> *3.9TB*  0Bytes Yes  Yes
>
> [root@s01 ~]# du -hs /tier2/CSP/ans004/
> *295G* /tier2/CSP/ans004/
>
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Quotas not working after adding arbiter brick to replica 2

[Sanoj Unnikrishnan]

It will be fixed in 3.11

On Fri, Aug 4, 2017 at 7:30 PM, mabi <m...@protonmail.ch> wrote:

> Thank you very much Sanoj, I ran your script once and it worked. I now
> have quotas again...
>
> Question: do you know in which release this issue will be fixed?
>
>
>
>  Original Message 
> Subject: Re: [Gluster-users] Quotas not working after adding arbiter brick
> to replica 2
> Local Time: August 4, 2017 3:28 PM
> UTC Time: August 4, 2017 1:28 PM
> From: sunni...@redhat.com
> To: mabi <m...@protonmail.ch>
> Gluster Users <gluster-users@gluster.org>
>
> Hi mabi,
> This is a likely issue where the last gfid entry in the quota.conf file is
> stale (because the directory was deleted with quota limit on it being
> removed)
> (https://review.gluster.org/#/c/16507/)
>
> To fix the issue, we need to remove the last entry (last 17 bytes/ 16bytes
> based on quota version) in the file.
> Please use the below work around for the same until next upgrade.
> you only need to change $vol to the name of volume.
>
> ===
> vol=
> qconf=/var/lib/glusterd/vols/$vol/quota.conf
> qconf_bk="$qconf".bk
> cp $qconf $qconf_bk
>
> grep "GlusterFS Quota conf | version: v1.2" /var/lib/glusterd/vols/v5/
> quota.conf
> if [ $? -eq 0 ];
> then
> entry_size=17;
> else
> entry_size=16;
> fi
>
> size=`ls -l $qconf | awk '{print $5}'`
> (( size_new = size - entry_size ))
> dd if=$qconf_bk of=$qconf bs=1 count=$size_new
> gluster v quota v5 list
> 
> In the unlikely case that there are multiple stale entries in the end of
> file you may have to run it multiple times
> to fix the issue (each time one stale entry at the end is removed)
>
>
> On Thu, Aug 3, 2017 at 1:17 PM, mabi <m...@protonmail.ch> wrote:
>
>> I tried to re-create manually my quotas but not even that works now.
>> Running the "limit-usage" command as showed below returns success:
>>
>> $ sudo gluster volume quota myvolume limit-usage /userdirectory 50GB
>> volume quota : success
>>
>>
>>
>> but when I list the quotas using "list" nothing appears.
>>
>> What can I do to fix that issue with the quotas?
>>
>>  Original Message 
>> Subject: Re: [Gluster-users] Quotas not working after adding arbiter
>> brick to replica 2
>> Local Time: August 2, 2017 2:35 PM
>> UTC Time: August 2, 2017 12:35 PM
>> From: m...@protonmail.ch
>> To: Sanoj Unnikrishnan <sunni...@redhat.com>
>> Gluster Users <gluster-users@gluster.org>
>>
>> Hi Sanoj,
>>
>> I copied over the quota.conf file from the affected volume (node 1) and
>> opened it up with a hex editor but can not recognize anything really except
>> for the first few header/version bytes. I have attached it within this mail
>> (compressed with bzip2) as requested.
>>
>> Should I recreate them manually? there where around 10 of them. Or is
>> there a hope of recovering these quotas?
>>
>> Regards,
>> M.
>>
>>
>>
>>  Original Message 
>> Subject: Re: [Gluster-users] Quotas not working after adding arbiter
>> brick to replica 2
>> Local Time: August 2, 2017 1:06 PM
>> UTC Time: August 2, 2017 11:06 AM
>> From: sunni...@redhat.com
>> To: mabi <m...@protonmail.ch>
>> Gluster Users <gluster-users@gluster.org>
>>
>> Mabi,
>>
>> We have fixed a couple of issues in the quota list path.
>> Could you also please attach the quota.conf file
>> (/var/lib/glusterd/vols/patchy/quota.conf)
>> (Ideally, the first few bytes would be ascii characters followed by 17
>> bytes per directory on which quota limit is set)
>> Regards,
>> Sanoj
>>
>> On Tue, Aug 1, 2017 at 1:36 PM, mabi <m...@protonmail.ch> wrote:
>>
>>> I also just noticed quite a few of the following warning messages in the
>>> quotad.log log file:
>>>
>>> [2017-08-01 07:59:27.834202] W [MSGID: 108027]
>>> [afr-common.c:2496:afr_discover_done] 0-myvolume-replicate-0: no read
>>> subvols for (null)
>>>
>>>
>>>
>>>
>>>  Original Message 
>>> Subject: [Gluster-users] Quotas not working after adding arbiter brick
>>> to replica 2
>>> Local Time: August 1, 2017 8:49 AM
>>> UTC Time: August 1, 2017 6:49 AM
>>> From: m...@protonmail.ch
>>> To: Gluster Users <gluster-users@gluster.org>
>>>
>>> Hello,
>>>

Re: [Gluster-users] Quotas not working after adding arbiter brick to replica 2

[Sanoj Unnikrishnan]

Hi mabi,

This is a likely issue where the last gfid entry in the quota.conf file is
stale (because the directory was deleted with quota limit on it being
removed)
(https://review.gluster.org/#/c/16507/)

To fix the issue, we need to remove the last entry (last 17 bytes/ 16bytes
based on quota version) in the file.
Please use the below work around for the same until next upgrade.
you only need to change $vol to the name of volume.

===
vol=
qconf=/var/lib/glusterd/vols/$vol/quota.conf
qconf_bk="$qconf".bk
cp $qconf $qconf_bk

grep "GlusterFS Quota conf | version: v1.2"
/var/lib/glusterd/vols/v5/quota.conf
if [ $? -eq 0 ];
then
entry_size=17;
else
entry_size=16;
fi

size=`ls -l $qconf | awk '{print $5}'`
(( size_new = size - entry_size ))
dd if=$qconf_bk of=$qconf bs=1 count=$size_new
gluster v quota v5 list


In the unlikely case that there are multiple stale entries in the end of
file you may have to run it multiple times
to fix the issue (each time one stale entry at the end is removed)


On Thu, Aug 3, 2017 at 1:17 PM, mabi <m...@protonmail.ch> wrote:

> I tried to re-create manually my quotas but not even that works now.
> Running the "limit-usage" command as showed below returns success:
>
> $ sudo gluster volume quota myvolume limit-usage /userdirectory 50GB
> volume quota : success
>
>
>
> but when I list the quotas using "list" nothing appears.
>
> What can I do to fix that issue with the quotas?
>
>  Original Message 
> Subject: Re: [Gluster-users] Quotas not working after adding arbiter brick
> to replica 2
> Local Time: August 2, 2017 2:35 PM
> UTC Time: August 2, 2017 12:35 PM
> From: m...@protonmail.ch
> To: Sanoj Unnikrishnan <sunni...@redhat.com>
> Gluster Users <gluster-users@gluster.org>
>
> Hi Sanoj,
>
> I copied over the quota.conf file from the affected volume (node 1) and
> opened it up with a hex editor but can not recognize anything really except
> for the first few header/version bytes. I have attached it within this mail
> (compressed with bzip2) as requested.
>
> Should I recreate them manually? there where around 10 of them. Or is
> there a hope of recovering these quotas?
>
> Regards,
> M.
>
>
>
>  Original Message 
> Subject: Re: [Gluster-users] Quotas not working after adding arbiter brick
> to replica 2
> Local Time: August 2, 2017 1:06 PM
> UTC Time: August 2, 2017 11:06 AM
> From: sunni...@redhat.com
> To: mabi <m...@protonmail.ch>
> Gluster Users <gluster-users@gluster.org>
>
> Mabi,
>
> We have fixed a couple of issues in the quota list path.
> Could you also please attach the quota.conf file
> (/var/lib/glusterd/vols/patchy/quota.conf)
> (Ideally, the first few bytes would be ascii characters followed by 17
> bytes per directory on which quota limit is set)
> Regards,
> Sanoj
>
> On Tue, Aug 1, 2017 at 1:36 PM, mabi <m...@protonmail.ch> wrote:
>
>> I also just noticed quite a few of the following warning messages in the
>> quotad.log log file:
>>
>> [2017-08-01 07:59:27.834202] W [MSGID: 108027]
>> [afr-common.c:2496:afr_discover_done] 0-myvolume-replicate-0: no read
>> subvols for (null)
>>
>>
>>
>>
>>  Original Message 
>> Subject: [Gluster-users] Quotas not working after adding arbiter brick to
>> replica 2
>> Local Time: August 1, 2017 8:49 AM
>> UTC Time: August 1, 2017 6:49 AM
>> From: m...@protonmail.ch
>> To: Gluster Users <gluster-users@gluster.org>
>>
>> Hello,
>>
>> As you might have read in my previous post on the mailing list I have
>> added an arbiter node to my GlusterFS 3.8.11 replica 2 volume. After some
>> healing issues and help of Ravi that could get fixed but now I just noticed
>> that my quotas are all gone.
>>
>> When I run the following command:
>>
>> glusterfs volume quota myvolume list
>>
>> There is no output...
>>
>> In the /var/log/glusterfs/quotad.log I can see the following two lines
>> when running the list command:
>>
>> [2017-08-01 06:46:04.451765] W [dict.c:581:dict_unref]
>> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/features/quotad.so(+0x1f3d)
>> [0x7fe868e21f3d] -->/usr/lib/x86_64-linux-gnu/g
>> lusterfs/3.8.11/xlator/features/quotad.so(+0x2d82) [0x7fe868e22d82]
>> -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0)
>> [0x7fe86f5c2b10] ) 0-dict: dict is NULL [Invalid argument]
>> [2017-08-01 06:46:04.459154] W [dict.c:581:dict_unref]
>> (-->/usr/lib/x86_64-linux-gnu/gluster

Re: [Gluster-users] Quotas not working after adding arbiter brick to replica 2

[Sanoj Unnikrishnan]

Mabi,

We have fixed a couple of issues in the quota list path.
Could you also please attach the quota.conf file (/var/lib/glusterd/vols/
patchy/quota.conf)
(Ideally, the first few bytes would be ascii characters followed by 17
bytes per directory on which quota limit is set)
Regards,
Sanoj


On Tue, Aug 1, 2017 at 1:36 PM, mabi  wrote:

> I also just noticed quite a few of the following warning messages in the
> quotad.log log file:
>
> [2017-08-01 07:59:27.834202] W [MSGID: 108027] 
> [afr-common.c:2496:afr_discover_done]
> 0-myvolume-replicate-0: no read subvols for (null)
>
>
>
>
>  Original Message 
> Subject: [Gluster-users] Quotas not working after adding arbiter brick to
> replica 2
> Local Time: August 1, 2017 8:49 AM
> UTC Time: August 1, 2017 6:49 AM
> From: m...@protonmail.ch
> To: Gluster Users 
>
> Hello,
>
> As you might have read in my previous post on the mailing list I have
> added an arbiter node to my GlusterFS 3.8.11 replica 2 volume. After some
> healing issues and help of Ravi that could get fixed but now I just noticed
> that my quotas are all gone.
>
> When I run the following command:
>
> glusterfs volume quota myvolume list
>
> There is no output...
>
> In the /var/log/glusterfs/quotad.log I can see the following two lines
> when running the list command:
>
> [2017-08-01 06:46:04.451765] W [dict.c:581:dict_unref]
> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/features/quotad.so(+0x1f3d)
> [0x7fe868e21f3d] 
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/features/quotad.so(+0x2d82)
> [0x7fe868e22d82] -->/usr/lib/x86_64-linux-gnu/
> libglusterfs.so.0(dict_unref+0xc0) [0x7fe86f5c2b10] ) 0-dict: dict is
> NULL [Invalid argument]
> [2017-08-01 06:46:04.459154] W [dict.c:581:dict_unref]
> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/features/quotad.so(+0x1f3d)
> [0x7fe868e21f3d] 
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/features/quotad.so(+0x2d82)
> [0x7fe868e22d82] -->/usr/lib/x86_64-linux-gnu/
> libglusterfs.so.0(dict_unref+0xc0) [0x7fe86f5c2b10] ) 0-dict: dict is
> NULL [Invalid argument]
>
> In case you need this info, I have added by arbiter node to the replica 2
> by using this command:
>
> gluster volume add-brick myvolume replica 3 arbiter 1
> arbiternode.domain.tld:/srv/glusterfs/myvolume/brick
>
>
>
> How can I get my quotas back working as before? I had defined around 20
> quotas on different directories of that volume.
>
> Regards,
> Mabi
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] gfid and volume-id extended attributes lost

[Sanoj Unnikrishnan]

Please use the systemtap script(
https://paste.fedoraproject.org/paste/EGDa0ErwX0LV3y-gBYpfNA) to check
which process is invoking remove xattr calls.
It prints the pid, tid and arguments of all removexattr calls.
I have checked for these fops at the protocol/client and posix translators.

To run the script ..
1) install systemtap and dependencies.
2) install glusterfs-debuginfo
3) change the path of the translator in the systemtap script to appropriate
values for your system
(change "/usr/lib64/glusterfs/3.12dev/xlator/protocol/client.so" and
"/usr/lib64/glusterfs/3.12dev/xlator/storage/posix.so")
4) run the script as follows
#stap -v fop_trace.stp

The o/p would look like these .. additionally arguments will also be dumped
if glusterfs-debuginfo is also installed (i had not done it here.)
pid-958: 0 glusterfsd(3893):->posix_setxattr
pid-958:47 glusterfsd(3893):<-posix_setxattr
pid-966: 0 glusterfsd(5033):->posix_setxattr
pid-966:57 glusterfsd(5033):<-posix_setxattr
pid-1423: 0 glusterfs(1431):->client_setxattr
pid-1423:37 glusterfs(1431):<-client_setxattr
pid-1423: 0 glusterfs(1431):->client_setxattr
pid-1423:41 glusterfs(1431):<-client_setxattr

Regards,
Sanoj



On Mon, Jul 10, 2017 at 2:56 PM, Sanoj Unnikrishnan <sunni...@redhat.com>
wrote:

> @ pranith , yes . we can get the pid on all removexattr call and also
> print the backtrace of the glusterfsd process when trigerring removing
> xattr.
> I will write the script and reply back.
>
> On Sat, Jul 8, 2017 at 7:06 AM, Pranith Kumar Karampuri <
> pkara...@redhat.com> wrote:
>
>> Ram,
>>As per the code, self-heal was the only candidate which *can* do
>> it. Could you check logs of self-heal daemon and the mount to check if
>> there are any metadata heals on root?
>>
>>
>> +Sanoj
>>
>> Sanoj,
>>Is there any systemtap script we can use to detect which process
>> is removing these xattrs?
>>
>> On Sat, Jul 8, 2017 at 2:58 AM, Ankireddypalle Reddy <
>> are...@commvault.com> wrote:
>>
>>> We lost the attributes on all the bricks on servers glusterfs2 and
>>> glusterfs3 again.
>>>
>>>
>>>
>>> [root@glusterfs2 Log_Files]# gluster volume info
>>>
>>>
>>>
>>> Volume Name: StoragePool
>>>
>>> Type: Distributed-Disperse
>>>
>>> Volume ID: 149e976f-4e21-451c-bf0f-f5691208531f
>>>
>>> Status: Started
>>>
>>> Number of Bricks: 20 x (2 + 1) = 60
>>>
>>> Transport-type: tcp
>>>
>>> Bricks:
>>>
>>> Brick1: glusterfs1sds:/ws/disk1/ws_brick
>>>
>>> Brick2: glusterfs2sds:/ws/disk1/ws_brick
>>>
>>> Brick3: glusterfs3sds:/ws/disk1/ws_brick
>>>
>>> Brick4: glusterfs1sds:/ws/disk2/ws_brick
>>>
>>> Brick5: glusterfs2sds:/ws/disk2/ws_brick
>>>
>>> Brick6: glusterfs3sds:/ws/disk2/ws_brick
>>>
>>> Brick7: glusterfs1sds:/ws/disk3/ws_brick
>>>
>>> Brick8: glusterfs2sds:/ws/disk3/ws_brick
>>>
>>> Brick9: glusterfs3sds:/ws/disk3/ws_brick
>>>
>>> Brick10: glusterfs1sds:/ws/disk4/ws_brick
>>>
>>> Brick11: glusterfs2sds:/ws/disk4/ws_brick
>>>
>>> Brick12: glusterfs3sds:/ws/disk4/ws_brick
>>>
>>> Brick13: glusterfs1sds:/ws/disk5/ws_brick
>>>
>>> Brick14: glusterfs2sds:/ws/disk5/ws_brick
>>>
>>> Brick15: glusterfs3sds:/ws/disk5/ws_brick
>>>
>>> Brick16: glusterfs1sds:/ws/disk6/ws_brick
>>>
>>> Brick17: glusterfs2sds:/ws/disk6/ws_brick
>>>
>>> Brick18: glusterfs3sds:/ws/disk6/ws_brick
>>>
>>> Brick19: glusterfs1sds:/ws/disk7/ws_brick
>>>
>>> Brick20: glusterfs2sds:/ws/disk7/ws_brick
>>>
>>> Brick21: glusterfs3sds:/ws/disk7/ws_brick
>>>
>>> Brick22: glusterfs1sds:/ws/disk8/ws_brick
>>>
>>> Brick23: glusterfs2sds:/ws/disk8/ws_brick
>>>
>>> Brick24: glusterfs3sds:/ws/disk8/ws_brick
>>>
>>> Brick25: glusterfs4sds.commvault.com:/ws/disk1/ws_brick
>>>
>>> Brick26: glusterfs5sds.commvault.com:/ws/disk1/ws_brick
>>>
>>> Brick27: glusterfs6sds.commvault.com:/ws/disk1/ws_brick
>>>
>>> Brick28: glusterfs4sds.commvault.com:/ws/disk10/ws_brick
>>>
>>> Brick29: glusterfs5sds.commvault.com:/ws/disk10/ws_brick
>>>
>>> Brick30: glusterfs6sds.commvault.com:/ws/disk10/ws_brick
>>>
>>> Brick31: glusterfs4sds.commvault.com:/ws/

Re: [Gluster-users] [Gluster-devel] gfid and volume-id extended attributes lost

[Sanoj Unnikrishnan]

@ pranith , yes . we can get the pid on all removexattr call and also print
the backtrace of the glusterfsd process when trigerring removing xattr.
I will write the script and reply back.

On Sat, Jul 8, 2017 at 7:06 AM, Pranith Kumar Karampuri  wrote:

> Ram,
>As per the code, self-heal was the only candidate which *can* do
> it. Could you check logs of self-heal daemon and the mount to check if
> there are any metadata heals on root?
>
>
> +Sanoj
>
> Sanoj,
>Is there any systemtap script we can use to detect which process is
> removing these xattrs?
>
> On Sat, Jul 8, 2017 at 2:58 AM, Ankireddypalle Reddy  > wrote:
>
>> We lost the attributes on all the bricks on servers glusterfs2 and
>> glusterfs3 again.
>>
>>
>>
>> [root@glusterfs2 Log_Files]# gluster volume info
>>
>>
>>
>> Volume Name: StoragePool
>>
>> Type: Distributed-Disperse
>>
>> Volume ID: 149e976f-4e21-451c-bf0f-f5691208531f
>>
>> Status: Started
>>
>> Number of Bricks: 20 x (2 + 1) = 60
>>
>> Transport-type: tcp
>>
>> Bricks:
>>
>> Brick1: glusterfs1sds:/ws/disk1/ws_brick
>>
>> Brick2: glusterfs2sds:/ws/disk1/ws_brick
>>
>> Brick3: glusterfs3sds:/ws/disk1/ws_brick
>>
>> Brick4: glusterfs1sds:/ws/disk2/ws_brick
>>
>> Brick5: glusterfs2sds:/ws/disk2/ws_brick
>>
>> Brick6: glusterfs3sds:/ws/disk2/ws_brick
>>
>> Brick7: glusterfs1sds:/ws/disk3/ws_brick
>>
>> Brick8: glusterfs2sds:/ws/disk3/ws_brick
>>
>> Brick9: glusterfs3sds:/ws/disk3/ws_brick
>>
>> Brick10: glusterfs1sds:/ws/disk4/ws_brick
>>
>> Brick11: glusterfs2sds:/ws/disk4/ws_brick
>>
>> Brick12: glusterfs3sds:/ws/disk4/ws_brick
>>
>> Brick13: glusterfs1sds:/ws/disk5/ws_brick
>>
>> Brick14: glusterfs2sds:/ws/disk5/ws_brick
>>
>> Brick15: glusterfs3sds:/ws/disk5/ws_brick
>>
>> Brick16: glusterfs1sds:/ws/disk6/ws_brick
>>
>> Brick17: glusterfs2sds:/ws/disk6/ws_brick
>>
>> Brick18: glusterfs3sds:/ws/disk6/ws_brick
>>
>> Brick19: glusterfs1sds:/ws/disk7/ws_brick
>>
>> Brick20: glusterfs2sds:/ws/disk7/ws_brick
>>
>> Brick21: glusterfs3sds:/ws/disk7/ws_brick
>>
>> Brick22: glusterfs1sds:/ws/disk8/ws_brick
>>
>> Brick23: glusterfs2sds:/ws/disk8/ws_brick
>>
>> Brick24: glusterfs3sds:/ws/disk8/ws_brick
>>
>> Brick25: glusterfs4sds.commvault.com:/ws/disk1/ws_brick
>>
>> Brick26: glusterfs5sds.commvault.com:/ws/disk1/ws_brick
>>
>> Brick27: glusterfs6sds.commvault.com:/ws/disk1/ws_brick
>>
>> Brick28: glusterfs4sds.commvault.com:/ws/disk10/ws_brick
>>
>> Brick29: glusterfs5sds.commvault.com:/ws/disk10/ws_brick
>>
>> Brick30: glusterfs6sds.commvault.com:/ws/disk10/ws_brick
>>
>> Brick31: glusterfs4sds.commvault.com:/ws/disk11/ws_brick
>>
>> Brick32: glusterfs5sds.commvault.com:/ws/disk11/ws_brick
>>
>> Brick33: glusterfs6sds.commvault.com:/ws/disk11/ws_brick
>>
>> Brick34: glusterfs4sds.commvault.com:/ws/disk12/ws_brick
>>
>> Brick35: glusterfs5sds.commvault.com:/ws/disk12/ws_brick
>>
>> Brick36: glusterfs6sds.commvault.com:/ws/disk12/ws_brick
>>
>> Brick37: glusterfs4sds.commvault.com:/ws/disk2/ws_brick
>>
>> Brick38: glusterfs5sds.commvault.com:/ws/disk2/ws_brick
>>
>> Brick39: glusterfs6sds.commvault.com:/ws/disk2/ws_brick
>>
>> Brick40: glusterfs4sds.commvault.com:/ws/disk3/ws_brick
>>
>> Brick41: glusterfs5sds.commvault.com:/ws/disk3/ws_brick
>>
>> Brick42: glusterfs6sds.commvault.com:/ws/disk3/ws_brick
>>
>> Brick43: glusterfs4sds.commvault.com:/ws/disk4/ws_brick
>>
>> Brick44: glusterfs5sds.commvault.com:/ws/disk4/ws_brick
>>
>> Brick45: glusterfs6sds.commvault.com:/ws/disk4/ws_brick
>>
>> Brick46: glusterfs4sds.commvault.com:/ws/disk5/ws_brick
>>
>> Brick47: glusterfs5sds.commvault.com:/ws/disk5/ws_brick
>>
>> Brick48: glusterfs6sds.commvault.com:/ws/disk5/ws_brick
>>
>> Brick49: glusterfs4sds.commvault.com:/ws/disk6/ws_brick
>>
>> Brick50: glusterfs5sds.commvault.com:/ws/disk6/ws_brick
>>
>> Brick51: glusterfs6sds.commvault.com:/ws/disk6/ws_brick
>>
>> Brick52: glusterfs4sds.commvault.com:/ws/disk7/ws_brick
>>
>> Brick53: glusterfs5sds.commvault.com:/ws/disk7/ws_brick
>>
>> Brick54: glusterfs6sds.commvault.com:/ws/disk7/ws_brick
>>
>> Brick55: glusterfs4sds.commvault.com:/ws/disk8/ws_brick
>>
>> Brick56: glusterfs5sds.commvault.com:/ws/disk8/ws_brick
>>
>> Brick57: glusterfs6sds.commvault.com:/ws/disk8/ws_brick
>>
>> Brick58: glusterfs4sds.commvault.com:/ws/disk9/ws_brick
>>
>> Brick59: glusterfs5sds.commvault.com:/ws/disk9/ws_brick
>>
>> Brick60: glusterfs6sds.commvault.com:/ws/disk9/ws_brick
>>
>> Options Reconfigured:
>>
>> performance.readdir-ahead: on
>>
>> diagnostics.client-log-level: INFO
>>
>> auth.allow: glusterfs1sds,glusterfs2sds,glusterfs3sds,glusterfs4sds.comm
>> vault.com,glusterfs5sds.commvault.com,glusterfs6sds.commvault.com
>>
>>
>>
>> Thanks and Regards,
>>
>> Ram
>>
>> *From:* Pranith Kumar Karampuri [mailto:pkara...@redhat.com]
>> *Sent:* Friday, July 07, 2017 12:15 PM
>>
>> *To:* Ankireddypalle Reddy
>> *Cc:* Gluster Devel (gluster-de...@gluster.org);
>> gluster-users@gluster.org
>> *Subject:* Re: 

Re: [Gluster-users] Quota limits gone after upgrading to 3.8

[Sanoj Unnikrishnan]

Hi Mabi,

Note that limits are still configured and working.
re-adding the limits will not help here (unless you are willing to disable
and re-enable quota first).
The reason is if a gfid exists in quota.conf (because a limit was earlier
set on it), it does not  need change when limit changes.
The quota.conf file only keep track of which gfid have limit set. The
original value of the limits are set in xattr on filesystem

Another work around without mauallly touching quota.conf is,
> Create a new dummy directory anywhere in the FS. add a limit in this
directory.

After this you should be able to see the listing.
If you remove this dummy directory or limit on it, you will once again be
exposed to same issue.

Regards,
Sanoj

On Tue, May 9, 2017 at 10:59 PM, mabi  wrote:

> Hi Sanoj,
>
> Thanks for pointing me at this bug, I was not aware about it.
>
> As this is a production GlusterFS cluster I would rather not mess with the
> quota.conf file as you suggested. Instead I will simply re-add all my
> quotas by running the following command again:
>
> gluster volume quota myvolume limit-usage /directory1 100GB
>
> Can you confirm me that this is safe to run again?
>
>
>
> As soon as I have a minute I will complete your survey about quotas.
>
> Best,
> M.
>
>  Original Message 
> Subject: Re: [Gluster-users] Quota limits gone after upgrading to 3.8
> Local Time: May 9, 2017 6:50 AM
> UTC Time: May 9, 2017 4:50 AM
> From: sunni...@redhat.com
> To: mabi 
> Gluster Users 
>
> Hi mabi,
>
> This bug was fixed recently, https://bugzilla.redhat.com/sh
> ow_bug.cgi?id=1414346. It would be available in 3.11 release. I will plan
> to back port same to earlier releases.
>
> Your quota limits are still set and honored, It is only the listing that
> has gone wrong. Using list with command with single path should display the
> limit on that path. The printing of list gets messed up when the last gfid
> in the quota.conf file is not present in the FS (due to an rmdir without a
> remove limit)
>
> You could use the following workaround to get rid of the issue.
>  => Remove exactly the last 17 bytes of " /var/lib/glusterd/vols/ e>/quota.conf"
>   Note: keep a backup of quota.conf for safety
> If this does not solve the issue, please revert back with
> 1) quota.conf file
> 2) output of list command (when executed along with path)
> 3) getfattr -d -m . -e hex  | grep limit
> It would be great to have your feedback for quota on this thread (
> http://lists.gluster.org/pipermail/gluster-users/2017-April/030676.html)
>
> Thanks & Regards,
> Sanoj
>
>
> On Mon, May 8, 2017 at 7:58 PM, mabi  wrote:
>
>> Hello,
>>
>> I upgraded last week my 2 nodes replica GlusterFS cluster from 3.7.20 to
>> 3.8.11 and on one of the volumes I use the quota feature of GlusterFS.
>> Unfortunately, I just noticed by using the usual command "gluster volume
>> quota myvolume list" that all my quotas on that volume are gone. I had
>> around 10 different quotas set on different directories.
>>
>> Does anyone have an idea where the quotas have vanished? are they gone
>> for always and do I need to re-set them all?
>>
>> Regards,
>> M.
>>
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Quota limits gone after upgrading to 3.8

[Sanoj Unnikrishnan]

Hi mabi,

This bug was fixed recently, https://bugzilla.redhat.com/sh
ow_bug.cgi?id=1414346. It would be available in 3.11 release. I will plan
to back port same to earlier releases.

Your quota limits are still set and honored, It is only the listing that
has gone wrong. Using list with command with single path should display the
limit on that path. The printing of list gets messed up when the last gfid
in the quota.conf file is not present in the FS (due to an rmdir without a
remove limit)

You could use the following workaround to get rid of the issue.
 => Remove exactly the last 17 bytes of " /var/lib/glusterd/vols/<
volname>/quota.conf"
  Note: keep a backup of quota.conf for safety

If this does not solve the issue, please revert back with
1) quota.conf file
2) output of list command (when executed along with path)
3) getfattr -d -m . -e hex  | grep limit

It would be great to have your feedback for quota on this thread (
http://lists.gluster.org/pipermail/gluster-users/2017-April/030676.html)

Thanks & Regards,
Sanoj


On Mon, May 8, 2017 at 7:58 PM, mabi  wrote:

> Hello,
>
> I upgraded last week my 2 nodes replica GlusterFS cluster from 3.7.20 to
> 3.8.11 and on one of the volumes I use the quota feature of GlusterFS.
> Unfortunately, I just noticed by using the usual command "gluster volume
> quota myvolume list" that all my quotas on that volume are gone. I had
> around 10 different quotas set on different directories.
>
> Does anyone have an idea where the quotas have vanished? are they gone for
> always and do I need to re-set them all?
>
> Regards,
> M.
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Revisiting Quota functionality in GlusterFS

[Sanoj Unnikrishnan]

Hi All,

Considering we are coming out with major release plan, we would like to
revisit the QUOTA feature to decide its path forward.
I have been working on quota feature for a couple of months now and have
come across various issues from performance, usability and correctness
perspective.
We do have some initial thoughts on how these can be solved. But, to ensure
we work on the most important things first, we would like to get a pulse of
this from the users of quota feature.

Below is a questionnaire for the same. In order to put the questions in
perspective, I have provided rationale, external links to alternative
design thoughts.
The focus of this mail thread though is to get a user pulse than being a
design review. Please comment on design in the github issue itself.
We can bring the discussion to gluster-devel as they gain traction. We
would like the design discussion to be driven by the generated user
feedback.

1) On how many directories do u generally have quota limits configured.
How often do we see quota limits added/removed/changed. [numbers would
help here than qualitative answers]
Any use case with large number of quota limits (> 100 on a single
volume say)?
Is a filesystem Crawl acceptable each time a new quota limit is set?
(crawl may take time equivalent to du command, this would essentially
introduce a delay for a limit to take effect after it is set)

Rationale:
  Currently, we account the usage of all directories. A performance issue
with this approach is we need to recursively account the usage of a
file/directory on all its ancestors along the path.
  If we consider few directories have limits configured, we could explore
alternatives where accounting information is maintained only in directories
that have limits set [RFC1].

2) How strict accounting is generally acceptable. Is it acceptable if there
is an overshoot by 100MB say?
What are the general value of limits configured. Does anybody set
limits in the order of MBs ?

Rationale:
Relaxing the accounting could be another way to gain in performance. We
can batch / cache xattr updates. [RFC2]

3) Does directory quota suit your needs? Would you prefer if it works like
XFS directory quota?
What are the back-end Filesystem you expect to use quota with?
Is it acceptable to take the route of leveraging backend FS quota with
support for limited FS (or just xfs)?

Rationale:
Behavior of directory quota in GlusterFS largely differs from the XFS
way. both has its pros and cons.
GlusterFS will allow you to set a limit on /home and /home/user1. So if
you write to /home/user1/file1 the limit for both its ancestors are checked
and honored.
An admin who has to give storage to 50 users can configure /home to 1
TB and each user home to 25GB say (Hoping that not all users would
simultaneously use up their 25 GB). This may not make sense for a cloud
storage but it does make sense in a university lab kind of
setting.
The same cannot be done with XFS beacuse XFS directory quota relies on
project id. Only one project id is associated with a directory so only 1
limit can be honored at any time.
XFS directory quota has its advantages though. Please find details in
[RFC3]

4) Do you use quota with large number of bricks in a volume? (How many?)
Do you have quota with large number of volumes hosted on a node? (How
many?)
Have you seen any performance issues in such setting?

Rationale:
   We have a single quotad process (aggregator of accounting info) on each
nodes in the Trusted storage pool.
   All the bricks (even from different volumes) hosted on a node share the
quotad.
   Another issue in this design is large number of bricks within a volume
will increase IPC latency during aggregation.
   One way of mitigating this is by changing quotad and quota layer to work
on a lease based approach [RFC4]

5) If you set directory level quota, do you expect to have hard link across
the directories? or want to support rename() across directories?

Rationale:
Supporting these operations consistently does add complexities in
design.
XFS itself doesn't support these two operations when quota is set on it.

6) Do u use inode-quota feature?
Any user looking for the user/group quota feature?

7) Are you using current quota feature?
  If yes, are you happy?
  if yes, and not happy? what are the things you would like to see
  improve?

RFC1 - Alternative Accounting method in marker (
https://github.com/gluster/glusterfs/issues/182)
RFC2 - Batched updates (https://github.com/gluster/glusterfs/issues/183)
RFC3 - XFS based Quota (https://github.com/gluster/glusterfs/issues/184)
RFC4 - Lease based quotad (https://github.com/gluster/glusterfs/issues/181)

Note: These RFC are not interdependent.


Thanks and Regards,
Sanoj
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] crazy quota numbers

[Sanoj Unnikrishnan]

Hi lejeczek,

We have seen this issue before.
https://bugzilla.redhat.com/show_bug.cgi?id=1421933
The problem here is that the accounting xattr in the backend becomes
F...
This happens likely due to an underflow (value goes below 0) on some
operation like rm/unlink/truncate/mv. We would be adding defensive checks
to prevent this, but that will not solve the accounting issue.

This is a race and we do not have a consistent reproducer for it yet. *Could
you describe the operations that were being done on the volume when u ran
into this?*

Due to the way quota works, if xattr of any directory goes wrong they get
bubbled up to all its ancestors. So, it would also help if we can identify
the lowest directory in the tree where the xattr has gone wrong and then
try to find the operations on that particular directory.
*Could you also share the xattr values on all bricks: find . | xargs
getfattr -d -m. -e hex *
​
To recover from this you could do a quota disable and enable.

Thanks and Regards,
Sanoj
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [ovirt-users] gluster - Volume quota failed

[Sanoj Unnikrishnan]

That's correct Glusterfs manages its own quotas with xattrs. If you create
glusterfs over xfs and set quotas on both, then, both the quota will be
simultaneously active. Hence it would be advisable to only use gluster
quota commands to manage quota.

Regards,
Sanoj

On Sun, Nov 13, 2016 at 10:09 PM, knarra  wrote:

> [+gluster-users] list
>
> On 11/11/2016 04:09 PM, lejeczek wrote:
>
>>
>>
>> On 11/11/16 09:26, knarra wrote:
>>
>>> On 11/11/2016 01:07 PM, lejeczek wrote:
>>>


 On 11/11/16 05:57, knarra wrote:

> On 11/11/2016 03:20 AM, lejeczek wrote:
>
>> quota command failed : Volume quota failed. The cluster is operating
>> at version 30700. Quota command enable is unavailable in this version.
>>
>
> Hi,
>
> Could you please tell us what is the version of glusterfs you are
> using ? For 3.7.0 / 3.7.1 30700 op-version is applicable. If you have
> greater version you would need to bump up the op-version and check if 
> quota
> enable works.
>
> Hope this helps.
>
> Thanks
>
> kasturi.
>
> it is glusterfs-3.7.16-1.el7.x86_64 - so it's higher, correct? And if
 yes why gluster did set it 30700?
 What value should there be?

>>> Could you please set the op-version to 30712 and try enabling quota.
>>> Ideally, when a fresh install of glusterfs is done op version is set to
>>> correct value. When a upgrade of node is done then user has to go and bump
>>> up the op-version.
>>>
>>
>> hi, thanks,
>> like I mentioned earlier - no earlier setup, gluster was installed yes,
>> older versions but never was in use, not a single volume was set up.
>>
>> I'd have question still about quotas:  I've started googling but have to
>> failed - glusterfs does not translate local FS (xfs) quotas and presents
>> them to its clients, does it?
>> It is only glusterfs own quota functions that we have to manage quotas?
>>
>>
>>
>>> More info can be found here https://github.com/gluster/glu
>>> sterfs/blob/release-3.7/doc/release-notes/3.7.16.md
>>>


 thanks
 ___
 Users mailing list
 us...@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users

>>>
>>>
>>>
>>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users