hi, folks! I'm testing cache tier for erasure coded pool and with
RBD image on it. And now I'm facing a problem with full cache pool
and object are not evicted automatically, Only if I run manually
rados -p cache cache-flush-evict-all*
client side is:
superuser@share:~$ uname -a
*Linux share 3.16-2-amd64 #1 SMP Debian 3.16.3-2 (2014-09-20) x86_64
GNU/Linux
ceph node all are debian wheezy
*superuser~$ dpkg -l | grep ceph
*ii ceph 0.87-1~bpo70+1 amd64 distributed
storage and file system
ii ceph-common 0.87-1~bpo70+1 amd64 common
utilities to mount and interact with a ceph storage cluster
ii ceph-fs-common 0.87-1~bpo70+1 amd64
common utilities to mount and interact with a ceph file system
ii ceph-mds 0.87-1~bpo70+1 amd64 metadata
server for the ceph distributed file system
ii libcephfs1 0.87-1~bpo70+1 amd64 Ceph
distributed file system client library
ii libcurl3-gnutls:amd64 7.29.0-1~bpo70+1.ceph
amd64 easy-to-use client-side URL transfer library (GnuTLS
flavour)
ii python-ceph 0.87-1~bpo70+1 amd64 Python
libraries for the Ceph distributed filesystem
*
*There are all steps to reproduce (excepting creation of pools):
*superuser@admin:~$ ceph osd pool get ec_backup-storage
erasure_code_profile**
*erasure_code_profile: default
*superuser@admin:~$ ceph osd erasure-code-profile get default*
directory=/usr/lib/ceph/erasure-code
k=2
m=1
plugin=jerasure
technique=reed_sol_van*
*********** ADMIN NODE OPERATIONS *************
**
*superuser@admin:~$ ceph df*
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
242T 224T 6092G 2.46
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
ec_backup-storage 4 0 0 147T
0
*cache* 5 0 0 185G 0
*block-devices* 6 1948G 0.79 75638G 498771
*superuser@admin:~$ rados df*
pool name category KB objects clones degraded
unfound rd rd KB wr wr KB
block-devices - 2042805201 498771 0 0
0 67127 259320535 2070571 2403248346
*cache* - 0 0 0
0 0 60496 247235411 966553 499544074
*ec_backup-storage* - 0 0 0
0 0 156988 537227276 400355 819838985
total used 6388431372 498771
total avail 240559782780
total space 260163775608
****** 'cache' pool is replicated pool, 'ec_backup-storage' -
Erasure Encoded pool ******
****** running simple script for enabling cache tiering:*
****** There is my simple script for enabling cache tier:*
*superuser@admin:~$ ./enable_cache_tier.sh cache ec_backup-storage*
pool 'cache' is now (or already was) a tier of 'ec_backup-storage'
set cache-mode for pool 'cache' to writeback
overlay for 'ec_backup-storage' is now (or already was) 'cache'
set pool 5 hit_set_type to bloom
set pool 5 cache_target_dirty_ratio to 0.4
set pool 5 cache_target_full_ratio to 0.7
set pool 5 cache_min_flush_age to 10
set pool 5 cache_min_evict_age to 10
****** **Displaying**some cache pool parameters:*
*superuser@admin:~$ for param in cache_target_dirty_ratio
cache_target_full_ratio target_max_bytes target_max_objects
cache_min_flush_age *cache_min_evict_age; do ceph osd pool get
cache $param; done
cache_target_dirty_ratio: 0.4
*cache_target_full_ratio: 0.7*
target_max_bytes: 0
target_max_objects: 0
cache_min_flush_age: 10
cache_min_evict_age: 10
************ END ADMIN NODE OPERATIONS *************
************ CEPH CLIENT OPERATIONS *************
*superuser@share:~$ rbd create -p ec_backup-storage ec_image.img
--size 500000 --image-format 2*
*superuser@share:~$ rbd -p ec_backup-storage ls*
ec_image.img
*superuser@share:~$ sudo rbd map -p ec_backup-storage ec_image.img*
/dev/rbd0
*superuser@share:~$ rbd showmapped *
id pool image snap device
0 ec_backup-storage ec_image.img - /dev/rbd0
*superuser@share:~$ sudo parted /dev/rbd0 p*
Error: /dev/rbd0: unrecognised disk label
Model: Unknown (unknown)
Disk /dev/rbd0: 524GB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:
***** now start filling the rbd disk with 400Gb of 'zeros' *****
*superuser@share:~$ sudo dd if=/dev/zero of=/dev/rbd0 bs=4M
count=100000 oflag=direct*
^C39901+0 записей получено *; There I'm pressed CTRL+C because
already getting warnings in cpeh -s (see below)*
39901+0 записей отправлено
скопировано 167356923904 байта (167 GB), 3387,59 c, 49,4 MB/c
*
*********** END CEPH CLIENT OPERATIONS ************
***
*********** ADMIN NODE OPERATIONS ************
**
***** Now check ceph df: 145G of 172G 'cache' pool it is about 90%
is occupied already!*
*superuser@admin:~$ ceph df*
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
242T 223T 6555G 2.64
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
*ec_backup-storage 4 0 0 147T 0 *
* cache 5 155G 0.06 12333M 39906 *
block-devices 6 1948G 0.79 75642G
498771
****** ec_backup-storage - tier pool for cold data still empty????*
****** but in the same time:*
*superuser@admin:~$* *ceph health detail*
HEALTH_WARN 3 near full osd(s)
osd.45 is near full at 88%
osd.95 is near full at 87%
osd.100 is near full at 86%
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
