Apologies - I copied and pasted the wrong ansible output: matthew@laptop:~/playbooks$ ansible -i hosts gluster-servers[0:6] -u matthewb --ask-pass -m shell -b --become-method=sudo --ask-become-pass -a "getfattr --absolute-names -m . -d -e hex /mnt/raid6-storage/storage/data/projects/MEOPAR | egrep '^trusted.glusterfs.quota.size'" SSH password: SUDO password[defaults to SSH password]: gluster02 | SUCCESS | rc=0 >> trusted.glusterfs.quota.size=0x0000011ecfa56c00000000000005cd6d000000000006d478 trusted.glusterfs.quota.size.1=0x0000010ad4a452000000000000012a0300000000000150fa
gluster05 | SUCCESS | rc=0 >> trusted.glusterfs.quota.size=0x00000033b8e93800000000000004cde9000000000006b1a4 trusted.glusterfs.quota.size.1=0x0000010dca277c00000000000001297d0000000000015005 gluster04 | SUCCESS | rc=0 >> trusted.glusterfs.quota.size=0xffffff396f3ec000000000000004d7eb0000000000068c62 trusted.glusterfs.quota.size.1=0x00000106e6724800000000000001138f0000000000012fb2 gluster01 | SUCCESS | rc=0 >> trusted.glusterfs.quota.size=0x0000003d4d4348000000000000057616000000000006afd2 trusted.glusterfs.quota.size.1=0x00000133fe211e00000000000005d161000000000006cfd4 gluster03 | SUCCESS | rc=0 >> trusted.glusterfs.quota.size=0xfffffd02acabf000000000000003599000000000000643e2 trusted.glusterfs.quota.size.1=0x00000114e20f5e0000000000000113b30000000000012fb2 gluster06 | SUCCESS | rc=0 >> trusted.glusterfs.quota.size=0xffffff0c98de440000000000000536e40000000000068cf2 trusted.glusterfs.quota.size.1=0x0000013532664e00000000000005e73f000000000006cfd4 gluster07 | SUCCESS | rc=0 >> trusted.glusterfs.quota.size=0x000001108e51140000000000000327c6000000000006bf6d Thanks, -Matthew On Fri, Sep 1, 2017 at 3:22 PM, Matthew B <[email protected]> wrote: > Thanks Sanoj, > > Now the brick is showing the correct xattrs: > > [root@gluster07 ~]# getfattr --absolute-names -m . -d -e hex > /mnt/raid6-storage/storage/data/projects/MEOPAR > # file: /mnt/raid6-storage/storage/data/projects/MEOPAR > security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 > trusted.gfid=0x7209b677f4b94d82a3820733620e6929 > trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime=0x599f228800088654 > trusted.glusterfs.dht=0x0000000100000000b6db6d41db6db6ee*trusted.glusterfs.quota.d5a5ecda-7511-4bbb-9b4c-4fcc84e3e1da.contri=0x000001108e51140000000000000327c6000000000006bf6d* > trusted.glusterfs.quota.dirty=0x3000 > trusted.glusterfs.quota.limit-set=0x0000088000000000ffffffffffffffff*trusted.glusterfs.quota.size=0x000001108e51140000000000000327c6000000000006bf6d* > > > However, the quota listing still shows the old (incorrect) value: > > > [root@gluster07 ~]# gluster volume quota storage list | egrep "MEOPAR " > /data/projects/MEOPAR 8.5TB 80%(6.8TB) *16384.0PB* > 10.6TB No No > > > I've checked on each of the bricks and they look fine now - is there any > way to reflect the new value in the quota itself? > > matthew@laptop:~/playbooks$ ansible -i hosts gluster-servers[0:6] -u matthewb > --ask-pass -m shell -b --become-method=sudo --ask-become-pass -a "getfattr > --absolute-names -m . -d -e hex > /mnt/raid6-storage/storage/data/projects/comp_support | egrep > '^trusted.glusterfs.quota.size\=' | sed 's/trusted.glusterfs.quota.size\=//' > | cut -c 1-18 | xargs printf '%d\n'" > SSH password: > SUDO password[defaults to SSH password]: > gluster05 | SUCCESS | rc=0 >> > 567293059584 > > gluster04 | SUCCESS | rc=0 >> > 510784812032 > > gluster03 | SUCCESS | rc=0 >> > 939742334464 > > gluster01 | SUCCESS | rc=0 >> > 98688324096 > > gluster02 | SUCCESS | rc=0 >> > 61449348096 > > gluster07 | SUCCESS | rc=0 >> > 29252869632 > > gluster06 | SUCCESS | rc=0 >> > 31899410944 > > > Thanks, > -Matthew > > On Fri, Sep 1, 2017 at 4:33 AM, Sanoj Unnikrishnan <[email protected]> > wrote: > >> Hi Mathew, >> >> The other option is to explicitly remove the size and contri xattr at the >> brick path and then do a stat from the mount point. >> >> #setfattr -x trusted.glusterfs.quota.000000 >> 00-0000-0000-0000-000000000001.contri.1 <brick path /dir> >> #setfattr -x trusted.glusterfs.quota.size.1 <brick path / dir> >> #stat <mount path /dir> >> >> Stat would heal the size and the contri xattr and the dirty xattr would >> heal only on the next operation on the directory. >> >> After this you could set dirty bit and do a stat again. >> >> setxattr -n trusted.glusterfs.quota.dirty -v 0x3100 <brick path / dir> >> >> stat <mount path /dir> >> >> >> >> Regards, >> Sanoj >> >> On Thu, Aug 31, 2017 at 9:12 PM, Matthew B <[email protected] >> om> wrote: >> >>> Hi Raghavendra, >>> >>> I didn't get a chance to implement your suggestions, however it looks >>> like the dirty bit is no longer set - so presumably the quota should have >>> been updated, however the quota.size attribute is still incorrect though >>> slightly different than before. Any other suggestions? >>> >>> [root@gluster07 ~]# getfattr --absolute-names -m . -d -e hex >>> /mnt/raid6-storage/storage/data/projects/MEOPAR >>> # file: /mnt/raid6-storage/storage/data/projects/MEOPAR >>> security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6 >>> c6162656c65645f743a733000 >>> trusted.gfid=0x7209b677f4b94d82a3820733620e6929 >>> trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime >>> =0x599f228800088654 >>> trusted.glusterfs.dht=0x0000000100000000b6db6d41db6db6ee >>> trusted.glusterfs.quota.d5a5ecda-7511-4bbb-9b4c-4fcc84e3e1da >>> .contri=0xfffffa3d7c28f60000000000000a9d0a000000000005fd2f >>> trusted.glusterfs.quota.dirty=0x3000 >>> trusted.glusterfs.quota.limit-set=0x0000088000000000ffffffffffffffff >>> trusted.glusterfs.quota.size=0xfffffa3d7c28f60000000000000a9 >>> d0a000000000005fd2f >>> >>> Thanks, >>> -Matthew >>> >>> On Mon, Aug 28, 2017 at 8:05 PM, Raghavendra Gowdappa < >>> [email protected]> wrote: >>> >>>> >>>> >>>> ----- Original Message ----- >>>> > From: "Matthew B" <[email protected]> >>>> > To: "Sanoj Unnikrishnan" <[email protected]> >>>> > Cc: "Raghavendra Gowdappa" <[email protected]>, "Gluster Devel" < >>>> [email protected]> >>>> > Sent: Monday, August 28, 2017 9:33:25 PM >>>> > Subject: Re: [Gluster-devel] Quota Used Value Incorrect - Fix now or >>>> after upgrade >>>> > >>>> > Hi Sanoj, >>>> > >>>> > Thank you for the information - I have applied the changes you >>>> specified >>>> > above - but I haven't seen any changes in the xattrs on the directory >>>> after >>>> > about 15 minutes: >>>> >>>> I think stat is served from cache - either gluster's md-cache or kernel >>>> attribute cache. For healing to happen we need to force a lookup (which we >>>> had hoped would be issued as part of stat cmd) and this lookup has to reach >>>> marker xlator loaded on bricks. To make sure a lookup on the directory >>>> reaches marker we need to: >>>> >>>> 1. Turn off kernel attribute and entry cache (using --entrytimeout=0 >>>> and --attribute-timeout=0 as options to glusterfs while mounting) >>>> 2. Turn off md-cache using gluster cli (gluster volume set >>>> performance.md-cache <volname> off) >>>> 3. Turn off readdirplus in the entire stack [1] >>>> >>>> Once the above steps are done I guess doing a stat results in a lookup >>>> on the directory witnessed by marker. Once the issue is fixed you can undo >>>> the above three steps so that performance is not affected in your setup. >>>> >>>> [1] http://nongnu.13855.n7.nabble.com/Turning-off-readdirp-in-th >>>> e-entire-stack-on-fuse-mount-td220297.html >>>> >>>> > >>>> > [root@gluster07 ~]# setfattr -n trusted.glusterfs.quota.dirty -v >>>> 0x3100 >>>> > /mnt/raid6-storage/storage/data/projects/MEOPAR/ >>>> > >>>> > [root@gluster07 ~]# stat /mnt/raid6-storage/storage/dat >>>> a/projects/MEOPAR >>>> > >>>> > [root@gluster07 ~]# getfattr --absolute-names -m . -d -e hex >>>> > /mnt/raid6-storage/storage/data/projects/MEOPAR >>>> > # file: /mnt/raid6-storage/storage/data/projects/MEOPAR >>>> > security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6 >>>> c6162656c65645f743a733000 >>>> > trusted.gfid=0x7209b677f4b94d82a3820733620e6929 >>>> > trusted.glusterfs.6f95525a-94d7-4174-bac4-e1a18fe010a2.xtime >>>> =0x599f228800088654 >>>> > trusted.glusterfs.dht=0x0000000100000000b6db6d41db6db6ee >>>> > trusted.glusterfs.quota.d5a5ecda-7511-4bbb-9b4c-4fcc84e3e1da >>>> .contri=0xfffffa3d7c1ba60000000000000a9ccb000000000005fd2f >>>> > trusted.glusterfs.quota.dirty=0x3100 >>>> > trusted.glusterfs.quota.limit-set=0x0000088000000000ffffffffffffffff >>>> > trusted.glusterfs.quota.size=0xfffffa3d7c1ba60000000000000a9 >>>> ccb000000000005fd2f >>>> > >>>> > [root@gluster07 ~]# gluster volume status storage >>>> > Status of volume: storage >>>> > Gluster process TCP Port RDMA Port >>>> Online Pid >>>> > ------------------------------------------------------------ >>>> ------------------ >>>> > Brick 10.0.231.50:/mnt/raid6-storage/storag >>>> > e 49159 0 Y >>>> > 2160 >>>> > Brick 10.0.231.51:/mnt/raid6-storage/storag >>>> > e 49153 0 Y >>>> > 16037 >>>> > Brick 10.0.231.52:/mnt/raid6-storage/storag >>>> > e 49159 0 Y >>>> > 2298 >>>> > Brick 10.0.231.53:/mnt/raid6-storage/storag >>>> > e 49154 0 Y >>>> > 9038 >>>> > Brick 10.0.231.54:/mnt/raid6-storage/storag >>>> > e 49153 0 Y >>>> > 32284 >>>> > Brick 10.0.231.55:/mnt/raid6-storage/storag >>>> > e 49153 0 Y >>>> > 14840 >>>> > Brick 10.0.231.56:/mnt/raid6-storage/storag >>>> > e 49152 0 Y >>>> > 29389 >>>> > NFS Server on localhost 2049 0 Y >>>> > 29421 >>>> > Quota Daemon on localhost N/A N/A Y >>>> > 29438 >>>> > NFS Server on 10.0.231.51 2049 0 Y >>>> > 18249 >>>> > Quota Daemon on 10.0.231.51 N/A N/A Y >>>> > 18260 >>>> > NFS Server on 10.0.231.55 2049 0 Y >>>> > 24128 >>>> > Quota Daemon on 10.0.231.55 N/A N/A Y >>>> > 24147 >>>> > NFS Server on 10.0.231.54 2049 0 Y >>>> > 9397 >>>> > Quota Daemon on 10.0.231.54 N/A N/A Y >>>> > 9406 >>>> > NFS Server on 10.0.231.53 2049 0 Y >>>> > 18387 >>>> > Quota Daemon on 10.0.231.53 N/A N/A Y >>>> > 18397 >>>> > NFS Server on 10.0.231.52 2049 0 Y >>>> > 2230 >>>> > Quota Daemon on 10.0.231.52 N/A N/A Y >>>> > 2262 >>>> > NFS Server on 10.0.231.50 2049 0 Y >>>> > 2113 >>>> > Quota Daemon on 10.0.231.50 N/A N/A Y >>>> > 2154 >>>> > >>>> > Task Status of Volume storage >>>> > ------------------------------------------------------------ >>>> ------------------ >>>> > There are no active volume tasks >>>> > >>>> > [root@gluster07 ~]# gluster volume quota storage list | egrep >>>> "MEOPAR " >>>> > /data/projects/MEOPAR 8.5TB 80%(6.8TB) >>>> 16384.0PB >>>> > 17.4TB No No >>>> > >>>> > >>>> > >>>> > >>>> > Looking at the quota daemon on gluster07: >>>> > >>>> > [root@gluster07 ~]# ps -f -p 29438 >>>> > UID PID PPID C STIME TTY TIME CMD >>>> > root 29438 1 0 Jun19 ? 04:43:31 /usr/sbin/glusterfs -s >>>> > localhost --volfile-id gluster/quotad -p >>>> > /var/lib/glusterd/quotad/run/quotad.pid -l >>>> /var/log/glusterfs/quotad.log >>>> > >>>> > I can see some errors on the log - not sure if those are related: >>>> > >>>> > [root@gluster07 ~]# tail /var/log/glusterfs/quotad.log >>>> > [2017-08-28 15:36:17.990909] W [dict.c:592:dict_unref] >>>> > (-->/usr/lib64/glusterfs/3.7.13/xlator/features/quotad.so(qd >>>> _lookup_cbk+0x35e) >>>> > [0x7f79fb09253e] >>>> > -->/usr/lib64/glusterfs/3.7.13/xlator/features/quotad.so(quo >>>> tad_aggregator_getlimit_cbk+0xb3) >>>> > [0x7f79fb093333] -->/lib64/libglusterfs.so.0(dict_unref+0x99) >>>> > [0x7f7a090299e9] ) 0-dict: dict is NULL [Invalid argument] >>>> > [2017-08-28 15:36:17.991389] W [dict.c:592:dict_unref] >>>> > (-->/usr/lib64/glusterfs/3.7.13/xlator/features/quotad.so(qd >>>> _lookup_cbk+0x35e) >>>> > [0x7f79fb09253e] >>>> > -->/usr/lib64/glusterfs/3.7.13/xlator/features/quotad.so(quo >>>> tad_aggregator_getlimit_cbk+0xb3) >>>> > [0x7f79fb093333] -->/lib64/libglusterfs.so.0(dict_unref+0x99) >>>> > [0x7f7a090299e9] ) 0-dict: dict is NULL [Invalid argument] >>>> > [2017-08-28 15:36:17.992656] W [dict.c:592:dict_unref] >>>> > (-->/usr/lib64/glusterfs/3.7.13/xlator/features/quotad.so(qd >>>> _lookup_cbk+0x35e) >>>> > [0x7f79fb09253e] >>>> > -->/usr/lib64/glusterfs/3.7.13/xlator/features/quotad.so(quo >>>> tad_aggregator_getlimit_cbk+0xb3) >>>> > [0x7f79fb093333] -->/lib64/libglusterfs.so.0(dict_unref+0x99) >>>> > [0x7f7a090299e9] ) 0-dict: dict is NULL [Invalid argument] >>>> > [2017-08-28 15:36:17.993235] W [dict.c:592:dict_unref] >>>> > (-->/usr/lib64/glusterfs/3.7.13/xlator/features/quotad.so(qd >>>> _lookup_cbk+0x35e) >>>> > [0x7f79fb09253e] >>>> > -->/usr/lib64/glusterfs/3.7.13/xlator/features/quotad.so(quo >>>> tad_aggregator_getlimit_cbk+0xb3) >>>> > [0x7f79fb093333] -->/lib64/libglusterfs.so.0(dict_unref+0x99) >>>> > [0x7f7a090299e9] ) 0-dict: dict is NULL [Invalid argument] >>>> > [2017-08-28 15:45:51.024756] W [dict.c:592:dict_unref] >>>> > (-->/usr/lib64/glusterfs/3.7.13/xlator/features/quotad.so(qd >>>> _lookup_cbk+0x35e) >>>> > [0x7f79fb09253e] >>>> > -->/usr/lib64/glusterfs/3.7.13/xlator/features/quotad.so(quo >>>> tad_aggregator_getlimit_cbk+0xb3) >>>> > [0x7f79fb093333] -->/lib64/libglusterfs.so.0(dict_unref+0x99) >>>> > [0x7f7a090299e9] ) 0-dict: dict is NULL [Invalid argument] >>>> > [2017-08-28 15:45:51.027871] W [dict.c:592:dict_unref] >>>> > (-->/usr/lib64/glusterfs/3.7.13/xlator/features/quotad.so(qd >>>> _lookup_cbk+0x35e) >>>> > [0x7f79fb09253e] >>>> > -->/usr/lib64/glusterfs/3.7.13/xlator/features/quotad.so(quo >>>> tad_aggregator_getlimit_cbk+0xb3) >>>> > [0x7f79fb093333] -->/lib64/libglusterfs.so.0(dict_unref+0x99) >>>> > [0x7f7a090299e9] ) 0-dict: dict is NULL [Invalid argument] >>>> > [2017-08-28 15:45:51.030843] W [dict.c:592:dict_unref] >>>> > (-->/usr/lib64/glusterfs/3.7.13/xlator/features/quotad.so(qd >>>> _lookup_cbk+0x35e) >>>> > [0x7f79fb09253e] >>>> > -->/usr/lib64/glusterfs/3.7.13/xlator/features/quotad.so(quo >>>> tad_aggregator_getlimit_cbk+0xb3) >>>> > [0x7f79fb093333] -->/lib64/libglusterfs.so.0(dict_unref+0x99) >>>> > [0x7f7a090299e9] ) 0-dict: dict is NULL [Invalid argument] >>>> > [2017-08-28 15:45:51.031324] W [dict.c:592:dict_unref] >>>> > (-->/usr/lib64/glusterfs/3.7.13/xlator/features/quotad.so(qd >>>> _lookup_cbk+0x35e) >>>> > [0x7f79fb09253e] >>>> > -->/usr/lib64/glusterfs/3.7.13/xlator/features/quotad.so(quo >>>> tad_aggregator_getlimit_cbk+0xb3) >>>> > [0x7f79fb093333] -->/lib64/libglusterfs.so.0(dict_unref+0x99) >>>> > [0x7f7a090299e9] ) 0-dict: dict is NULL [Invalid argument] >>>> > [2017-08-28 15:45:51.032791] W [dict.c:592:dict_unref] >>>> > (-->/usr/lib64/glusterfs/3.7.13/xlator/features/quotad.so(qd >>>> _lookup_cbk+0x35e) >>>> > [0x7f79fb09253e] >>>> > -->/usr/lib64/glusterfs/3.7.13/xlator/features/quotad.so(quo >>>> tad_aggregator_getlimit_cbk+0xb3) >>>> > [0x7f79fb093333] -->/lib64/libglusterfs.so.0(dict_unref+0x99) >>>> > [0x7f7a090299e9] ) 0-dict: dict is NULL [Invalid argument] >>>> > [2017-08-28 15:45:51.033295] W [dict.c:592:dict_unref] >>>> > (-->/usr/lib64/glusterfs/3.7.13/xlator/features/quotad.so(qd >>>> _lookup_cbk+0x35e) >>>> > [0x7f79fb09253e] >>>> > -->/usr/lib64/glusterfs/3.7.13/xlator/features/quotad.so(quo >>>> tad_aggregator_getlimit_cbk+0xb3) >>>> > [0x7f79fb093333] -->/lib64/libglusterfs.so.0(dict_unref+0x99) >>>> > [0x7f7a090299e9] ) 0-dict: dict is NULL [Invalid argument] >>>> > >>>> > How should I proceed? >>>> > >>>> > Thanks, >>>> > -Matthew >>>> > >>>> > On Mon, Aug 28, 2017 at 3:13 AM, Sanoj Unnikrishnan < >>>> [email protected]> >>>> > wrote: >>>> > >>>> > > Hi Mathew, >>>> > > >>>> > > If you are sure that "/mnt/raid6-storage/storage/da >>>> ta/projects/MEOPAR/" >>>> > > is the only directory with wrong accounting and its immediate sub >>>> > > directories have correct xattr values, Setting the dirty xattr and >>>> doing a >>>> > > stat after that should resolve the issue. >>>> > > >>>> > > 1) setxattr -n trusted.glusterfs.quota.dirty -v 0x3100 >>>> > > /mnt/raid6-storage/storage/data/projects/MEOPAR/ >>>> > > >>>> > > 2) stat /mnt/raid6-storage/storage/data/projects/MEOPAR/ >>>> > > >>>> > > Could you please share what kind of operations that happens on this >>>> > > directory, to help RCA the issue. >>>> > > >>>> > > If you think this can be true elsewhere in filesystem as well,use >>>> the >>>> > > following script to identify the same. >>>> > > >>>> > > 1) https://github.com/gluster/glusterfs/blob/master/extras/ >>>> > > quota/xattr_analysis.py >>>> > > 2) https://github.com/gluster/glusterfs/blob/master/extras/ >>>> > > quota/log_accounting.sh >>>> > > >>>> > > Regards, >>>> > > Sanoj >>>> > > >>>> > > >>>> > > >>>> > > >>>> > > On Mon, Aug 28, 2017 at 12:39 PM, Raghavendra Gowdappa < >>>> > > [email protected]> wrote: >>>> > > >>>> > >> +sanoj >>>> > >> >>>> > >> ----- Original Message ----- >>>> > >> > From: "Matthew B" <[email protected]> >>>> > >> > To: [email protected] >>>> > >> > Sent: Saturday, August 26, 2017 12:45:19 AM >>>> > >> > Subject: [Gluster-devel] Quota Used Value Incorrect - Fix now or >>>> after >>>> > >> upgrade >>>> > >> > >>>> > >> > Hello, >>>> > >> > >>>> > >> > I need some advice on fixing an issue with quota on my gluster >>>> volume. >>>> > >> It's >>>> > >> > running version 3.7, distributed volume, with 7 nodes. >>>> > >> > >>>> > >> > # gluster --version >>>> > >> > glusterfs 3.7.13 built on Jul 8 2016 15:26:18 >>>> > >> > Repository revision: git:// git.gluster.com/glusterfs.git >>>> > >> > Copyright (c) 2006-2011 Gluster Inc. < http://www.gluster.com > >>>> > >> > GlusterFS comes with ABSOLUTELY NO WARRANTY. >>>> > >> > You may redistribute copies of GlusterFS under the terms of the >>>> GNU >>>> > >> General >>>> > >> > Public License. >>>> > >> > >>>> > >> > # gluster volume info storage >>>> > >> > >>>> > >> > Volume Name: storage >>>> > >> > Type: Distribute >>>> > >> > Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2 >>>> > >> > Status: Started >>>> > >> > Number of Bricks: 7 >>>> > >> > Transport-type: tcp >>>> > >> > Bricks: >>>> > >> > Brick1: 10.0.231.50:/mnt/raid6-storage/storage >>>> > >> > Brick2: 10.0.231.51:/mnt/raid6-storage/storage >>>> > >> > Brick3: 10.0.231.52:/mnt/raid6-storage/storage >>>> > >> > Brick4: 10.0.231.53:/mnt/raid6-storage/storage >>>> > >> > Brick5: 10.0.231.54:/mnt/raid6-storage/storage >>>> > >> > Brick6: 10.0.231.55:/mnt/raid6-storage/storage >>>> > >> > Brick7: 10.0.231.56:/mnt/raid6-storage/storage >>>> > >> > Options Reconfigured: >>>> > >> > changelog.changelog: on >>>> > >> > geo-replication.ignore-pid-check: on >>>> > >> > geo-replication.indexing: on >>>> > >> > nfs.disable: no >>>> > >> > performance.readdir-ahead: on >>>> > >> > features.quota: on >>>> > >> > features.inode-quota: on >>>> > >> > features.quota-deem-statfs: on >>>> > >> > features.read-only: off >>>> > >> > >>>> > >> > # df -h /storage/ >>>> > >> > Filesystem Size Used Avail Use% Mounted on >>>> > >> > 10.0.231.50:/storage 255T 172T 83T 68% /storage >>>> > >> > >>>> > >> > >>>> > >> > I am planning to upgrade to 3.10 (or 3.12 when it's available) >>>> but I >>>> > >> have a >>>> > >> > number of quotas configured, and one of them (below) has a very >>>> wrong >>>> > >> "Used" >>>> > >> > value: >>>> > >> > >>>> > >> > # gluster volume quota storage list | egrep "MEOPAR " >>>> > >> > /data/projects/MEOPAR 8.5TB 80%(6.8TB) 16384.0PB 17.4TB No No >>>> > >> > >>>> > >> > >>>> > >> > I have confirmed the bad value appears in one of the bricks >>>> current >>>> > >> xattrs, >>>> > >> > and it looks like the issue has been encountered previously on >>>> bricks >>>> > >> 04, >>>> > >> > 03, and 06: (gluster07 does not have a >>>> trusted.glusterfs.quota.size.1 >>>> > >> as it >>>> > >> > was recently added) >>>> > >> > >>>> > >> > $ ansible -i hosts gluster-servers[0:6] -u <user> --ask-pass -m >>>> shell -b >>>> > >> > --become-method=sudo --ask-become-pass -a "getfattr >>>> --absolute-names -m >>>> > >> . -d >>>> > >> > -e hex /mnt/raid6-storage/storage/data/projects/MEOPAR | egrep >>>> > >> > '^trusted.glusterfs.quota.size'" >>>> > >> > SSH password: >>>> > >> > SUDO password[defaults to SSH password]: >>>> > >> > >>>> > >> > gluster02 | SUCCESS | rc=0 >> >>>> > >> > trusted.glusterfs.quota.size=0x0000011ecfa56c00000000000005c >>>> > >> d6d000000000006d478 >>>> > >> > trusted.glusterfs.quota.size.1=0x0000010ad4a4520000000000000 >>>> > >> 12a0300000000000150fa >>>> > >> > >>>> > >> > gluster05 | SUCCESS | rc=0 >> >>>> > >> > trusted.glusterfs.quota.size=0x00000033b8e92200000000000004c >>>> > >> de8000000000006b1a4 >>>> > >> > trusted.glusterfs.quota.size.1=0x0000010dca277c0000000000000 >>>> > >> 1297d0000000000015005 >>>> > >> > >>>> > >> > gluster01 | SUCCESS | rc=0 >> >>>> > >> > trusted.glusterfs.quota.size=0x0000003d4d4348000000000000057 >>>> > >> 616000000000006afd2 >>>> > >> > trusted.glusterfs.quota.size.1=0x00000133fe211e0000000000000 >>>> > >> 5d161000000000006cfd4 >>>> > >> > >>>> > >> > gluster04 | SUCCESS | rc=0 >> >>>> > >> > trusted.glusterfs.quota.size=0xffffff396f3e9400000000000004d >>>> > >> 7ea0000000000068c62 >>>> > >> > trusted.glusterfs.quota.size.1=0x00000106e672480000000000000 >>>> > >> 1138f0000000000012fb2 >>>> > >> > >>>> > >> > gluster03 | SUCCESS | rc=0 >> >>>> > >> > trusted.glusterfs.quota.size=0xfffffd02acabf0000000000000035 >>>> > >> 99000000000000643e2 >>>> > >> > trusted.glusterfs.quota.size.1=0x00000114e20f5e0000000000000 >>>> > >> 113b30000000000012fb2 >>>> > >> > >>>> > >> > gluster06 | SUCCESS | rc=0 >> >>>> > >> > trusted.glusterfs.quota.size=0xffffff0c98de44000000000000053 >>>> > >> 6e40000000000068cf2 >>>> > >> > trusted.glusterfs.quota.size.1=0x0000013532664e0000000000000 >>>> > >> 5e73f000000000006cfd4 >>>> > >> > >>>> > >> > gluster07 | SUCCESS | rc=0 >> >>>> > >> > trusted.glusterfs.quota.size=0xfffffa3d7c1ba60000000000000a9 >>>> > >> ccb000000000005fd2f >>>> > >> > >>>> > >> > And reviewing the subdirectories of that folder on the impacted >>>> server >>>> > >> you >>>> > >> > can see that none of the direct children have such incorrect >>>> values: >>>> > >> > >>>> > >> > [root@gluster07 ~]# getfattr --absolute-names -m . -d -e hex >>>> > >> > /mnt/raid6-storage/storage/data/projects/MEOPAR/* >>>> > >> > # file: /mnt/raid6-storage/storage/data/projects/MEOPAR/<dir1 > >>>> > >> > ... >>>> > >> > trusted.glusterfs.quota.7209b677-f4b9-4d82-a382-0733620e6929 >>>> > >> .contri=0x000000fb6841820000000000000074730000000000000dae >>>> > >> > trusted.glusterfs.quota.dirty=0x3000 >>>> > >> > trusted.glusterfs.quota.size=0x000000fb684182000000000000007 >>>> > >> 4730000000000000dae >>>> > >> > >>>> > >> > # file: /mnt/raid6-storage/storage/data/projects/MEOPAR/<dir2 > >>>> > >> > ... >>>> > >> > trusted.glusterfs.quota.7209b677-f4b9-4d82-a382-0733620e6929 >>>> > >> .contri=0x0000000416d5f4000000000000000baa0000000000000441 >>>> > >> > trusted.glusterfs.quota.dirty=0x3000 >>>> > >> > trusted.glusterfs.quota.limit-set=0x0000010000000000ffffffff >>>> ffffffff >>>> > >> > trusted.glusterfs.quota.size=0x0000000416d5f4000000000000000 >>>> > >> baa0000000000000441 >>>> > >> > >>>> > >> > # file: /mnt/raid6-storage/storage/data/projects/MEOPAR/<dir3> >>>> > >> > ... >>>> > >> > trusted.glusterfs.quota.7209b677-f4b9-4d82-a382-0733620e6929 >>>> > >> .contri=0x000000110f2c4e00000000000002a76a000000000006ad7d >>>> > >> > trusted.glusterfs.quota.dirty=0x3000 >>>> > >> > trusted.glusterfs.quota.limit-set=0x0000020000000000ffffffff >>>> ffffffff >>>> > >> > trusted.glusterfs.quota.size=0x000000110f2c4e00000000000002a >>>> > >> 76a000000000006ad7d >>>> > >> > >>>> > >> > >>>> > >> > Can I fix this on the current version of gluster (3.7) on just >>>> the one >>>> > >> brick >>>> > >> > before I upgrade? Or would I be better off upgrading to 3.10 and >>>> trying >>>> > >> to >>>> > >> > fix it then? >>>> > >> > >>>> > >> > I have reviewed information here: >>>> > >> > >>>> > >> > http://lists.gluster.org/pipermail/gluster-devel/2016-Februa >>>> > >> ry/048282.html >>>> > >> > http://lists.gluster.org/pipermail/gluster-users.old/2016- >>>> > >> September/028365.html >>>> > >> > >>>> > >> > It seems like since I am on Gluster 3.7 I can disable quotas and >>>> > >> re-enable >>>> > >> > and everything will get recalculated and increment the index on >>>> the >>>> > >> > quota.size xattr. But with the size of the volume that will take >>>> a very >>>> > >> long >>>> > >> > time. >>>> > >> > >>>> > >> > Could I simply mark the impacted directly as dirty on gluster07? >>>> Or >>>> > >> update >>>> > >> > the xattr directly as the sum of the size of dir1, 2, and 3? >>>> > >> > >>>> > >> > Thanks, >>>> > >> > -Matthew >>>> > >> > >>>> > >> > _______________________________________________ >>>> > >> > Gluster-devel mailing list >>>> > >> > [email protected] >>>> > >> > http://lists.gluster.org/mailman/listinfo/gluster-devel >>>> > >> >>>> > > >>>> > > >>>> > >>>> >>> >>> >> >
_______________________________________________ Gluster-devel mailing list [email protected] http://lists.gluster.org/mailman/listinfo/gluster-devel
