I setup a cephfs file system with a cache tier over an erasure coded tier as an
experiment:
ceph osd erasure-code-profile set raid6 k=4 m=2
ceph osd pool create cephfs-metadata 512 512
ceph osd pool set cephfs-metadata size 3
ceph osd pool create cache-data 2048 2048
ceph osd pool create cephfs-data 256 256 erasure raid6 default_erasure
ceph osd tier add cephfs-data cache-data
ceph osd tier cache-mode cache-data writeback
ceph osd tier set-overlay cephfs-data cache-data
ceph osd pool set cache-data hit_set_type bloom
ceph osd pool set cache-data target_max_bytes 1099511627776
The file system was created from the cephfs-metadata and cephfs-data pools
After adding a lot of data to this and waiting for the pools to idle down and
stabilize I removing the file system content with rm. I am seeing very strange
behavior, the file system remove was quick, and then it started removing the
data from the pools. However it appears to be reading the data from the erasure
coded pool and creating empty content in the cache pool.
At its peak capacity the system looked like this:
NAME ID CATEGORY USED %USED MAX AVAIL
OBJECTS DIRTY READ WRITE
cache-data 21 - 791G 5.25 6755G
256302 140k 22969 2138k
cephfs-data 22 - 4156G 27.54 4503G
1064086 1039k 51271 1046k
2 hours later it looked like this:
NAME ID CATEGORY USED %USED MAX AVAIL
OBJECTS DIRTY READ WRITE
cache-data 21 - 326G 2.17 7576G
702142 559k 22969 2689k
cephfs-data 22 - 3964G 26.27 5051G
1014842 991k 476k 1143k
The object count in the erasure coded pool has gone down a little, the count in
the cache pool has gone up a lot, there has been a lot of read activity in the
erasure coded pool and write activity into both pools. The used count in the
cache pool is also going down. It looks like the cache pool is gaining 9
objects for each one removed from the erasure code pool. Looking at the actual
files being created by the OSD for this, they are empty.
What is going on here? It looks like this will take a day or so to complete at
this rate of progress.
The ceph version here is the master branch from a couple of days ago.
Thanks
Steve Lord
----------------------------------------------------------------------
The information contained in this transmission may be confidential. Any
disclosure, copying, or further distribution of confidential information is not
permitted unless such privilege is explicitly granted in writing by Quantum.
Quantum reserves the right to have electronic communications, including email
and attachments, sent across its networks filtered through anti virus and spam
software programs and retain such messages in order to comply with applicable
data security and retention requirements. Quantum is not responsible for the
proper and complete transmission of the substance of this communication or for
any delay in its receipt.
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
