Hi, I have a cephfs cluster as follows:
1 15x HDD data pool (primary cephfs data pool) 1 2x SSD data pool (linked to a specific dir via xattrs) 1 2x SSD metadata pool 1 2x SSD cache tier pool the cache tier pool consists in 2 host, with one SSD OSD on each host, with size=2 replicated by host. Last night the disks went 100% full and the cluster went down. I know I made a mistake and set target_max_objects and target_max_bytes to 0 in the cache pool, but isn't ceph supposed to stop writing to an OSD when it reaches it's full_ratio (default 0.95) ? And what about the cache_target_full_ratio in the cache tier pool? Here is the cluster: ~# ceph fs ls name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ssd ] * the Metadata and the SSD data pools use the same 2 OSDs (one cephfs directory is linked to the SSD data pool via xattrs) ~# ceph -v ceph version 10.2.9-4-gbeaec39 (beaec397f00491079cd74f7b9e3e10660859e26b) ~# ceph osd pool ls detail pool 1 'cephfs_data' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 136 lfor 115 flags hashpspool crash_replay_interval 45 tiers 3 read_tier 3 write_tier 3 stripe_width 0 pool 2 'cephfs_metadata' replicated size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 128 pgp_num 128 last_change 617 flags hashpspool stripe_width 0 pool 3 'cephfs_cache' replicated size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 128 pgp_num 128 last_change 1493 lfor 115 flags hashpspool,incomplete_clones tier_of 1 cache_mode writeback target_bytes 343597383680 hit_set bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 0s x0 decay_rate 0 search_last_n 0 stripe_width 0 pool 12 'cephfs_data_ssd' replicated size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 128 pgp_num 128 last_change 653 flags hashpspool stripe_width 0 ~# ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -8 0.17598 root default-ssd -9 0.09299 host bhs1-mail03-fe01-data 17 0.09299 osd.17 up 1.00000 1.00000 -10 0.08299 host bhs1-mail03-fe02-data 18 0.08299 osd.18 up 1.00000 1.00000 -7 0.86319 root cache-ssd -5 0.43159 host bhs1-mail03-fe01 15 0.43159 osd.15 up 1.00000 1.00000 -6 0.43159 host bhs1-mail03-fe02 16 0.43159 osd.16 up 1.00000 1.00000 -1 79.95895 root default -2 26.65298 host bhs1-mail03-ds01 0 5.33060 osd.0 up 1.00000 1.00000 1 5.33060 osd.1 up 1.00000 1.00000 2 5.33060 osd.2 up 1.00000 1.00000 3 5.33060 osd.3 up 1.00000 1.00000 4 5.33060 osd.4 up 1.00000 1.00000 -3 26.65298 host bhs1-mail03-ds02 5 5.33060 osd.5 up 1.00000 1.00000 6 5.33060 osd.6 up 1.00000 1.00000 7 5.33060 osd.7 up 1.00000 1.00000 8 5.33060 osd.8 up 1.00000 1.00000 9 5.33060 osd.9 up 1.00000 1.00000 -4 26.65298 host bhs1-mail03-ds03 10 5.33060 osd.10 up 1.00000 1.00000 12 5.33060 osd.12 up 1.00000 1.00000 13 5.33060 osd.13 up 1.00000 1.00000 14 5.33060 osd.14 up 1.00000 1.00000 19 5.33060 osd.19 up 1.00000 1.00000 ~# ceph osd crush rule dump [ { "rule_id": 0, "rule_name": "replicated_ruleset", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "replicated_ruleset_ssd", "ruleset": 1, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -7, "item_name": "cache-ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "replicated-data-ssd", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -8, "item_name": "default-ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ] ~# ceph osd pool get cephfs_cache all size: 2 min_size: 1 crash_replay_interval: 0 pg_num: 128 pgp_num: 128 crush_ruleset: 1 hashpspool: true nodelete: false nopgchange: false nosizechange: false write_fadvise_dontneed: false noscrub: false nodeep-scrub: false hit_set_type: bloom hit_set_period: 0 hit_set_count: 0 hit_set_fpp: 0.05 use_gmt_hitset: 1 auid: 0 target_max_objects: 0 target_max_bytes: 343597383680 cache_target_dirty_ratio: 0.5 cache_target_dirty_high_ratio: 0.7 cache_target_full_ratio: 0.8 cache_min_flush_age: 3600 cache_min_evict_age: 3600 min_read_recency_for_promote: 0 min_write_recency_for_promote: 0 fast_read: 0 hit_set_grade_decay_rate: 0 hit_set_search_last_n: 0 Regards, Webert Lima DevOps Engineer at MAV Tecnologia *Belo Horizonte - Brasil*
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com