TL;DR bobtail Ceph cluster unable to finish rebalance after drive failure, usage increasing even with no clients connected.....
I've been running a test bobtail cluster for a couple of months and it's been working great. Last week I had a drive die and rebalance; durring that time another OSD crashed. All was still well, however as the second osd had just crashed I restarted made sure that it re-entered properly and rebalancing continued and then I went to bed. Waking up in the morning I found 2 OSDs were 100% full and two more were almost full. To get out of the situation I decreased the replication size from 3 to 2, and then also carefully (I believe carefully enough) remove some PGs in order to start things up again. I got things going again and things appeared to be rebalancing correctly; however it got to the point were it stopped at 1420 PGs active+clean and the rest were stuck backfilling. Looking at the PG dump, all of the PGs that were having issues were on osd.1. So I stopped it, verified things were continuing to rebalance after it was down/out and then formated osd.1's disk and put it back in. Since then I've not been able to get the cluster back to HEALTHY, due to a combination of OSDs dying while recovering (not due to disk failure, just crashes) as well as the used space in the cluster increasing abnormally. Right now I have all the clients disconnected and just the cluster rebalancing and the usage is increasing to the point where I have 12TB used when I have only < 3TB in cephfs and 2TB in a single RBD image (replication size 2). I've since shutdown the cluster so I don't fill it up. My crushmap is the default, here is the usual suspects. I'm happy to provide additional information. pg dump: http://pastebin.com/LUyu6Z09 ceph osd tree: osd.8 is the failed drive (I will be replacing tonight), weight on osd.1 and osd.6 was done via reweight-by-utilization # id weight type name up/down reweight -1 19.5 root default -3 19.5 rack unknownrack -2 19.5 host ceph-test 0 1.5 osd.0 up 1 1 1.5 osd.1 up 0.6027 2 1.5 osd.2 up 1 3 1.5 osd.3 up 1 4 1.5 osd.4 up 1 5 2 osd.5 up 1 6 2 osd.6 up 0.6676 7 2 osd.7 up 1 8 2 osd.8 down 0 9 2 osd.9 up 1 10 2 osd.10 up 1 ceph -s: health HEALTH_WARN 24 pgs backfill; 85 pgs backfill_toofull; 29 pgs backfilling; 40 pgs degraded; 1 pgs recovery_wait; 121 pgs stuck unclean; recovery 109306/2091318 degraded (5.227%); recovering 3 o/s, 43344KB/s; 2 near full osd(s); noout flag(s) set monmap e2: 1 mons at {a=10.200.200.21:6789/0}, election epoch 1, quorum 0 a osdmap e16251: 11 osds: 10 up, 10 in pgmap v3145187: 1536 pgs: 1414 active+clean, 6 active+remapped+wait_backfill, 10 active+remapped+wait_backfill+backfill_toofull, 4 active+degraded+wait_backfill+backfill_toofull, 22 active+remapped+backfilling, 42 active+remapped+backfill_toofull, 7 active+degraded+backfilling, 17 active+degraded+backfill_toofull, 1 active+recovery_wait+remapped, 4 active+degraded+remapped+wait_backfill+backfill_toofull, 8 active+degraded+remapped+backfill_toofull, 1 active+clean+scrubbing+deep; 31607 GB data, 12251 GB used, 4042 GB / 16293 GB avail; 109306/2091318 degraded (5.227%); recovering 3 o/s, 43344KB/s mdsmap e3363: 1/1/1 up {0=a=up:active} rep size: pool 0 'data' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 384 pgp_num 384 last_change 897 owner 0 crash_replay_interval 45 pool 1 'metadata' rep size 2 crush_ruleset 1 object_hash rjenkins pg_num 384 pgp_num 384 last_change 13364 owner 0 pool 2 'rbd' rep size 2 crush_ruleset 2 object_hash rjenkins pg_num 384 pgp_num 384 last_change 13208 owner 0 pool 4 'media_video' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 384 pgp_num 384 last_change 890 owner 0 ceph.conf: [global] auth cluster required = cephx auth service required = cephx auth client required = cephx osd pool default size = 3 osd pool default min size = 1 osd pool default pg num = 366 osd pool default pgp num = 366 [osd] osd journal size = 1000 journal_aio = true #osd recovery max active = 10 osd mkfs type = xfs osd mkfs options xfs = -f -i size=2048 osd mount options xfs = inode64,noatime [mon.a] host = ceph01 mon addr = 10.200.200.21:6789 [osd.0] # 1.5 TB SATA host = ceph01 devs = /dev/sdc weight = 1.5 [osd.1] # 1.5 TB SATA host = ceph01 devs = /dev/sdd weight = 1.5 [osd.2] # 1.5 TB SATA host = ceph01 devs = /dev/sdg weight = 1.5 [osd.3] # 1.5 TB SATA host = ceph01 devs = /dev/sdj weight = 1.5 [osd.4] # 1.5 TB SATA host = ceph01 devs = /dev/sdk weight = 1.5 [osd.5] # 2 TB SAS host = ceph01 devs = /dev/sdf weight = 2 [osd.6] # 2 TB SAS host = ceph01 devs = /dev/sdh weight = 2 [osd.7] # 2 TB SAS host = ceph01 devs = /dev/sda weight = 2 [osd.8] # 2 TB SAS host = ceph01 devs = /dev/sdb weight = 2 [osd.9] # 2 TB SAS host = ceph01 devs = /dev/sdi weight = 2 [osd.10] # 2 TB SAS host = ceph01 devs = /dev/sde weight = 2 [mds.a] host = ceph01
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
