Dear CephFS gurus...
I am testing CephFS and, up to now, I am pretty happy how it is
performing. Thank you for this nice piece of software.
However, I do have a couple of doubts. Just for reference:
- The OS / kernel version I am using in my client is:
# cat /etc/redhat-release
Scientific Linux release 6.6 (Carbon)
# uname -a
Linux xxx.my.domain 3.10.84-1.el6.elrepo.x86_64 #1 SMP Sat Jul
11 11:33:48 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
- Both my client and the ceph storage cluster are using ceph and
cephfs 0.94.2
# rpm -qa | grep ceph | sort
ceph-0.94.2-0.el6.x86_64
ceph-common-0.94.2-0.el6.x86_64
ceph-fuse-0.94.2-0.el6.x86_64
libcephfs1-0.94.2-0.el6.x86_64
python-cephfs-0.94.2-0.el6.x86_64
- My client and servers are in the same data center, served by a
10GbE connection.
My questions regards the space usage reported by CephFS. I am seeing
problems with the reported used space. I am not really sure what
triggers an update or if the update really happens. Here is a use case
I've experienced. Please note that I do have a couple of TB of test data
in the filesystem which represents about 15% of my full system, and I
wonder if in an empty system you will see such problems.
*1) Check the occupation in a given directory*
# getfattr -d -m ceph.*
/cephfs/objectsize4M_stripeunit512K_stripecount8/
getfattr: Removing leading '/' from absolute path names
# file: cephfs/objectsize4M_stripeunit512K_stripecount8/
ceph.dir.entries="65"
ceph.dir.files="65"
ceph.dir.layout="stripe_unit=524288 stripe_count=8
object_size=4194304 pool=cephfs_dt"
*ceph.dir.rbytes="549755813888"*
ceph.dir.rctime="1438568092.09360459179"
ceph.dir.rentries="66"
ceph.dir.rfiles="65"
ceph.dir.rsubdirs="1"
ceph.dir.subdirs="0"
*2) Create a 2 char file ("b" and "\n") in the directory. Those 2 chars
should occupy 2 bytes*
# echo "b" > /cephfs/objectsize4M_stripeunit512K_stripecount8/b.txt
# getfattr -d -m ceph.*
/cephfs/objectsize4M_stripeunit512K_stripecount8/
getfattr: Removing leading '/' from absolute path names
# file: cephfs/objectsize4M_stripeunit512K_stripecount8/
ceph.dir.entries="66"
ceph.dir.files="66"
ceph.dir.layout="stripe_unit=524288 stripe_count=8
object_size=4194304 pool=cephfs_dt"
*ceph.dir.rbytes="549755813890"*
ceph.dir.rctime="1438568742.09305192950"
ceph.dir.rentries="67"
ceph.dir.rfiles="66"
ceph.dir.rsubdirs="1"
ceph.dir.subdirs="0"
*3) Delete the b.txt file._
_ --> Please note that the space is not recovered!!!*
# rm /cephfs/objectsize4M_stripeunit512K_stripecount8/b.txt
rm: remove regular file
`/cephfs/objectsize4M_stripeunit512K_stripecount8/b.txt'? y
# getfattr -d -m ceph.*
/cephfs/objectsize4M_stripeunit512K_stripecount8/
getfattr: Removing leading '/' from absolute path names
# file: cephfs/objectsize4M_stripeunit512K_stripecount8/
ceph.dir.entries="65"
ceph.dir.files="65"
ceph.dir.layout="stripe_unit=524288 stripe_count=8
object_size=4194304 pool=cephfs_dt"
*ceph.dir.rbytes="549755813890"*
ceph.dir.rctime="1438568933.09123552306"
ceph.dir.rentries="66"
ceph.dir.rfiles="65"
ceph.dir.rsubdirs="1"
ceph.dir.subdirs="0"
*4) I've created a new file which I called bb.txt with much more chars
inside.
--> *_*Please note that space reported by cephfs is still the same
as in step 3) even after multiple updates to the new bb.txt file*_
# i=0; while [ $i -lt 1 ]; do echo "1" >>
/cephfs/objectsize4M_stripeunit512K_stripecount8/bb.txt; i=`expr $i
+ 1` ; done
# getfattr -d -m ceph.*
/cephfs/objectsize4M_stripeunit512K_stripecount8/
getfattr: Removing leading '/' from absolute path names
# file: cephfs/objectsize4M_stripeunit512K_stripecount8/
ceph.dir.entries="66"
ceph.dir.files="66"
ceph.dir.layout="stripe_unit=524288 stripe_count=8
object_size=4194304 pool=cephfs_dt"
*ceph.dir.rbytes="549755813890"*
ceph.dir.rctime="1438569099.09854895368"
ceph.dir.rentries="67"
ceph.dir.rfiles="66"
(...)
# i=0; while [ $i -lt 20480 ]; do echo "1" >>
/cephfs/objectsize4M_stripeunit512K_stripecount8/bb.txt; i=`expr $i
+ 1` ; done
# ls -l /cephfs/objectsize4M_stripeunit512K_stripecount8/bb.txt
-rw-r--r-- 1 root root 49156 Aug 3 02:36
/cephfs/objectsize4M_stripeunit512K_stripecount8/bb.txt
# getfattr -d -m ceph.*
/cephfs/objectsize4M_stripeunit512K_stripecount8/getfattr: Removing
leading '/' from absolute path names
# file: cephfs/objectsize4M_stripeunit512K_stripecount8/
ceph.dir.entries="66"
ceph.dir.files="66"
ceph.dir.layout="stripe_unit=524288 stripe_count=8
object_size=4194304 pool=cephfs_dt"
*ceph.dir.rbytes="549755813890"*
ceph.dir.rctime="1438569099.09854895368"
ceph.dir.rentries="67"
ceph.dir.rfiles="66"
ceph.dir.rsubdirs="1"
ceph.dir.subdirs="0"
*
**5) I've created a new file which I called c.txt.**
******--> **_Please note that, after a while, the filesystem is finally
updated as the sum of both bb.txt and c.txt_*
# i=0; while [ $i -lt 20480 ]; do echo "1" >>
/cephfs/objectsize4M_stripeunit512K_stripecount8/c.txt; i=`expr $i +
1` ; done
# ls -l /cephfs/objectsize4M_stripeunit512K_stripecount8/c.txt
-rw-r--r-- 1 root root 40960 Aug 3 02:38
/cephfs/objectsize4M_stripeunit512K_stripecount8/c.txt
# getfattr -d -m ceph.*
/cephfs/objectsize4M_stripeunit512K_stripecount8/getfattr: Removing
leading '/' from absolute path names
# file: cephfs/objectsize4M_stripeunit512K_stripecount8/
ceph.dir.entries="67"
ceph.dir.files="67"
ceph.dir.layout="stripe_unit=524288 stripe_count=8
object_size=4194304 pool=cephfs_dt"
ceph.dir.rbytes="549755863046"
ceph.dir.rctime="1438569478.09995047310"
ceph.dir.rentries="68"
ceph.dir.rfiles="67"
ceph.dir.rsubdirs="1"
ceph.dir.subdirs="0"
ceph.dir.rsubdirs="1"
ceph.dir.subdirs="0"
549755863046-549755813890=90116=(49156+40960)
# ls -l /cephfs/objectsize4M_stripeunit512K_stripecount8/bb.txt
-rw-r--r-- 1 root root 49156 Aug 3 02:36
/cephfs/objectsize4M_stripeunit512K_stripecount8/bb.txt
# ls -l /cephfs/objectsize4M_stripeunit512K_stripecount8/c.txt
-rw-r--r-- 1 root root 40960 Aug 3 02:38
/cephfs/objectsize4M_stripeunit512K_stripecount8/c.txt
Suggestions? Bug? Comment?
Cheers
Goncalo
--
Goncalo Borges
Research Computing
ARC Centre of Excellence for Particle Physics at the Terascale
School of Physics A28 | University of Sydney, NSW 2006
T: +61 2 93511937
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
