Re: [ceph-users] Defining quota in CephFS - quota is ignored

[Gregory Farnum]

On Wed, Jul 26, 2017 at 1:03 PM Patrick Donnelly 
wrote:

> On Wed, Jul 26, 2017 at 2:26 AM,   wrote:
> > Hello!
> >
> > Based on the documentation for defining quotas in CephFS for any
> directory (http://docs.ceph.com/docs/master/cephfs/quota/), I defined a
> quota for attribute max_bytes:
> > ld4257:~ # getfattr -n ceph.quota.max_bytes /mnt/ceph-fuse/MTY/
> > getfattr: Removing leading '/' from absolute path names
> > # file: mnt/ceph-fuse/MTY/
> > ceph.quota.max_bytes="1"
> >
> > To validate if the quota is working, I write a 128MB file in
> /mnt/ceph-fuse/MTY:
> > ld4257:~ # dd if=/dev/zero of=/mnt/ceph-fuse/MTY/128MBfile bs=64M count=2
> > 2+0 records in
> > 2+0 records out
> > 134217728 bytes (134 MB, 128 MiB) copied, 0.351206 s, 382 MB/s
> >
> > This file is created correctly, and the utilization statistcs confirm it:
> > ld4257:~ # rados df
> > pool name KB  objects   clones degraded
> unfound   rdrd KB   wrwr KB
> > hdb-backup131072   3200
>   08843251 88572586
> > hdb-backup_metadata27920   2700
>   0  301   168115 645955386
> > rbd0000
>   00000
> > templates  0000
>   00000
> >   total used 9528188   59
> >   total avail   811829446772
> >   total space   811838974960
> >
> >
> > Question:
> > Why can I create a file with size 128MB after defining a quota of 100MB?
>
> I don't have a cluster to check this on now but perhaps because a
> sparse file (you wrote all zeros) does not consume its entire file
> size in the quota (only what it uses).  Retry with /dev/urandom.
>
> (And the usual disclaimer: quotas only work with libcephfs/ceph-fuse.
> The kernel client does not support quotas.)



Also, with older clients you needed to explicitly turn it on with the
config option. I think that includes Jewel.

>
>
> --
> Patrick Donnelly
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Defining quota in CephFS - quota is ignored

[Patrick Donnelly]

On Wed, Jul 26, 2017 at 2:26 AM,   wrote:
> Hello!
>
> Based on the documentation for defining quotas in CephFS for any directory 
> (http://docs.ceph.com/docs/master/cephfs/quota/), I defined a quota for 
> attribute max_bytes:
> ld4257:~ # getfattr -n ceph.quota.max_bytes /mnt/ceph-fuse/MTY/
> getfattr: Removing leading '/' from absolute path names
> # file: mnt/ceph-fuse/MTY/
> ceph.quota.max_bytes="1"
>
> To validate if the quota is working, I write a 128MB file in 
> /mnt/ceph-fuse/MTY:
> ld4257:~ # dd if=/dev/zero of=/mnt/ceph-fuse/MTY/128MBfile bs=64M count=2
> 2+0 records in
> 2+0 records out
> 134217728 bytes (134 MB, 128 MiB) copied, 0.351206 s, 382 MB/s
>
> This file is created correctly, and the utilization statistcs confirm it:
> ld4257:~ # rados df
> pool name KB  objects   clones degraded  
> unfound   rdrd KB   wrwr KB
> hdb-backup131072   3200   
>  08843251 88572586
> hdb-backup_metadata27920   2700   
>  0  301   168115 645955386
> rbd0000   
>  00000
> templates  0000   
>  00000
>   total used 9528188   59
>   total avail   811829446772
>   total space   811838974960
>
>
> Question:
> Why can I create a file with size 128MB after defining a quota of 100MB?

I don't have a cluster to check this on now but perhaps because a
sparse file (you wrote all zeros) does not consume its entire file
size in the quota (only what it uses).  Retry with /dev/urandom.

(And the usual disclaimer: quotas only work with libcephfs/ceph-fuse.
The kernel client does not support quotas.)

-- 
Patrick Donnelly
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Defining quota in CephFS - quota is ignored

[Wido den Hollander]

> Op 26 juli 2017 om 13:43 schreef c.mo...@web.de:
> 
> 
> 26. Juli 2017 11:29, "Wido den Hollander"  schrieb:
> 
> >> Op 26 juli 2017 om 11:26 schreef c.mo...@web.de:
> >> 
> >> Hello!
> >> 
> >> Based on the documentation for defining quotas in CephFS for any directory
> >> (http://docs.ceph.com/docs/master/cephfs/quota), I defined a quota for 
> >> attribute max_bytes:
> >> ld4257:~ # getfattr -n ceph.quota.max_bytes /mnt/ceph-fuse/MTY/
> >> getfattr: Removing leading '/' from absolute path names
> >> # file: mnt/ceph-fuse/MTY/
> >> ceph.quota.max_bytes="1"
> >> 
> >> To validate if the quota is working, I write a 128MB file in 
> >> /mnt/ceph-fuse/MTY:
> >> ld4257:~ # dd if=/dev/zero of=/mnt/ceph-fuse/MTY/128MBfile bs=64M count=2
> >> 2+0 records in
> >> 2+0 records out
> >> 134217728 bytes (134 MB, 128 MiB) copied, 0.351206 s, 382 MB/s
> >> 
> >> This file is created correctly, and the utilization statistcs confirm it:
> >> ld4257:~ # rados df
> >> pool name KB objects clones degraded unfound rd rd KB wr wr KB
> >> hdb-backup 131072 32 0 0 0 8 8 43251 88572586
> >> hdb-backup_metadata 27920 27 0 0 0 301 168115 6459 55386
> >> rbd 0 0 0 0 0 0 0 0 0
> >> templates 0 0 0 0 0 0 0 0 0
> >> total used 9528188 59
> >> total avail 811829446772
> >> total space 811838974960
> >> 
> >> Question:
> >> Why can I create a file with size 128MB after defining a quota of 100MB?
> > 
> > What kernel version does the client use? Quotas rely on client support.
> > 
> > Also, quotas are lazy and can take a bit of time before they start to block 
> > writes.
> > 
> > Wido
> > 
> >> THX
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> ld4257:~ # uname -r
> 4.4.59-92.24-default
> 

Hmm, thinking about it, I don't think the kernel supports CephFS quotas yet.

Somebody else might be able to confirm, but afaik it's not supported.

Wido

> In the meantime I have created 4 files of 210MB space allocation in total:
> ld4257:~ # ll -h /mnt/ceph-fuse/MTY/
> total 210M
> -rw-r--r-- 1 root root 100M Jul 26 13:40 100MBfile_from_ld4257
> -rw-r--r-- 1 root root  10M Jul 26 11:54 10MBfile_from_ld4257
> -rw-r--r-- 1 root root  50M Jul 26 11:55 50MBfile_from_ld2398
> -rw-r--r-- 1 root root  50M Jul 26 11:56 50MBfile_from_ld4257
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Defining quota in CephFS - quota is ignored

[c . monty]

26. Juli 2017 11:29, "Wido den Hollander"  schrieb:

>> Op 26 juli 2017 om 11:26 schreef c.mo...@web.de:
>> 
>> Hello!
>> 
>> Based on the documentation for defining quotas in CephFS for any directory
>> (http://docs.ceph.com/docs/master/cephfs/quota), I defined a quota for 
>> attribute max_bytes:
>> ld4257:~ # getfattr -n ceph.quota.max_bytes /mnt/ceph-fuse/MTY/
>> getfattr: Removing leading '/' from absolute path names
>> # file: mnt/ceph-fuse/MTY/
>> ceph.quota.max_bytes="1"
>> 
>> To validate if the quota is working, I write a 128MB file in 
>> /mnt/ceph-fuse/MTY:
>> ld4257:~ # dd if=/dev/zero of=/mnt/ceph-fuse/MTY/128MBfile bs=64M count=2
>> 2+0 records in
>> 2+0 records out
>> 134217728 bytes (134 MB, 128 MiB) copied, 0.351206 s, 382 MB/s
>> 
>> This file is created correctly, and the utilization statistcs confirm it:
>> ld4257:~ # rados df
>> pool name KB objects clones degraded unfound rd rd KB wr wr KB
>> hdb-backup 131072 32 0 0 0 8 8 43251 88572586
>> hdb-backup_metadata 27920 27 0 0 0 301 168115 6459 55386
>> rbd 0 0 0 0 0 0 0 0 0
>> templates 0 0 0 0 0 0 0 0 0
>> total used 9528188 59
>> total avail 811829446772
>> total space 811838974960
>> 
>> Question:
>> Why can I create a file with size 128MB after defining a quota of 100MB?
> 
> What kernel version does the client use? Quotas rely on client support.
> 
> Also, quotas are lazy and can take a bit of time before they start to block 
> writes.
> 
> Wido
> 
>> THX
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


ld4257:~ # uname -r
4.4.59-92.24-default

In the meantime I have created 4 files of 210MB space allocation in total:
ld4257:~ # ll -h /mnt/ceph-fuse/MTY/
total 210M
-rw-r--r-- 1 root root 100M Jul 26 13:40 100MBfile_from_ld4257
-rw-r--r-- 1 root root  10M Jul 26 11:54 10MBfile_from_ld4257
-rw-r--r-- 1 root root  50M Jul 26 11:55 50MBfile_from_ld2398
-rw-r--r-- 1 root root  50M Jul 26 11:56 50MBfile_from_ld4257
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Defining quota in CephFS - quota is ignored

[Wido den Hollander]

> Op 26 juli 2017 om 11:26 schreef c.mo...@web.de:
> 
> 
> Hello!
> 
> Based on the documentation for defining quotas in CephFS for any directory 
> (http://docs.ceph.com/docs/master/cephfs/quota/), I defined a quota for 
> attribute max_bytes:
> ld4257:~ # getfattr -n ceph.quota.max_bytes /mnt/ceph-fuse/MTY/
> getfattr: Removing leading '/' from absolute path names
> # file: mnt/ceph-fuse/MTY/
> ceph.quota.max_bytes="1"
> 
> To validate if the quota is working, I write a 128MB file in 
> /mnt/ceph-fuse/MTY:
> ld4257:~ # dd if=/dev/zero of=/mnt/ceph-fuse/MTY/128MBfile bs=64M count=2
> 2+0 records in
> 2+0 records out
> 134217728 bytes (134 MB, 128 MiB) copied, 0.351206 s, 382 MB/s
> 
> This file is created correctly, and the utilization statistcs confirm it:
> ld4257:~ # rados df
> pool name KB  objects   clones degraded  
> unfound   rdrd KB   wrwr KB
> hdb-backup131072   3200   
>  08843251 88572586
> hdb-backup_metadata27920   2700   
>  0  301   168115 645955386
> rbd0000   
>  00000
> templates  0000   
>  00000
>   total used 9528188   59
>   total avail   811829446772
>   total space   811838974960
> 
> 
> Question:
> Why can I create a file with size 128MB after defining a quota of 100MB?
> 

What kernel version does the client use? Quotas rely on client support.

Also, quotas are lazy and can take a bit of time before they start to block 
writes.

Wido

> 
> THX
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com