Hi Again...
Once more, my environment:
- ceph/cephfs in 10.2.2.
- All infrastructure is in the same version (rados cluster, mons,
mds and cephfs clients).
- We mount cephfs using ceph-fuse.
I want to set up quotas to limit users from filling the filesystem and
proactively avoid a situation where I have several simultaneous full or
near full osds. However, I am not understanding how the reporting of
space works once quotas are in place. My cephfs cluster provides a
~100TB of space (~ 300 TB of raw space since I have a replication of
3x). Check the following two cases:
1./ In clients where the full filesystem hierarchy is available:
- I have the following quota:
# getfattr -n ceph.quota.max_bytes /coepp/cephfs
getfattr: Removing leading '/' from absolute path names
# file: coepp/cephfs
ceph.quota.max_bytes="88000000000000"
- I am mounting the client as
# ceph-fuse --id mount_user -k
/etc/ceph/ceph.client.mount_user.keyring -m <MON IP>:6789
*--client-quota* --fuse_default_permissions=0
--client_acl_type=posix_acl -r /cephfs /coepp/cephfs/
- The results of two consecutive 'df' commands, executed after
the mount operation, is the following. You can conclude that in
the first command, the reported values are computed with respect
to the quota but then fallback to the default when no quotas is
in place.
# puppet agent -t; df -h ; df -h
(...)
ceph-fuse 81T 51T 30T 64% /coepp/cephfs
(...)
ceph-fuse 306T 153T 154T 50% /coepp/cephfs
2./ On another type of clients where I only mount a subdirectory of the
filesystem (/coepp/cephfs/borg instead of coepp/cephfs):
- I have the following quota:
# getfattr -n ceph.quota.max_bytes /coepp/cephfs/borg
getfattr: Removing leading '/' from absolute path name
# file: coepp/cephfs/borg
ceph.quota.max_bytes="10000000000000"
- I mount the filesystem as:
# ceph-fuse --id mount_user -k
/etc/ceph/ceph.client.mount_user.keyring -m <MON IP>:6789
--client-quota --fuse_default_permissions=0
--client_acl_type=posix_acl -r /cephfs/borg /coepp/cephfs/borg
- The reported space is
# puppet agent -t; df -h ; df -h
(...)
ceph-fuse 9.1T 5.7T 3.5T 62% /coepp/cephfs/borg
(...)
ceph-fuse 81T 51T 30T 64% /coepp/cephfs/borg
3./ Both clients are behaving in the same way where they start by
reporting according to the implemented quota
51T of used space with respect to 81TB in total (case 1)
5.7T of used space with respect to 9.1T in total (case 2)
and then fallback to the values enforced in the previous level of the
hierarchy.
153T of used space with respect to 306T in total (case 1)
51T of used space with respect to 81TB in total (case 2)
Am i doing something wrong here?
Cheers
Goncalo
--
Goncalo Borges
Research Computing
ARC Centre of Excellence for Particle Physics at the Terascale
School of Physics A28 | University of Sydney, NSW 2006
T: +61 2 93511937
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
