Attempting to expand our small ceph cluster currently.
Have 8 nodes, 3 mons, and went from a single 8TB disk per node to 2x 8TB disks
per node, and the rebalancing process is excruciatingly slow.
Originally at 576 PGs before expansion, and wanted to allow rebalance to finish
before expanding the PG count for the single pool, and the replication size.
I have stopped scrubs for the time being, as well as set client and recovery io
to equal parts so that client io is not burying the recovery io. Also have
increased the number of recovery threads per osd.
> [osd]
> osd_recovery_threads = 5
> filestore_max_sync_interval = 30
> osd_client_op_priority = 32
> osd_recovery_op_priority = 32
Also, this is 10G networking we are working with and recovery io typically
hovers between 0-35 MB’s but typically very bursty.
Disks are 8TB 7.2k SAS disks behind an LSI 3108 controller, configured as
individual RAID0 VD’s, with pdcache disabled, but BBU backed write back caching
enabled at the controller level.
Have thought about increasing the ‘osd_max_backfills’ as well as
‘osd_recovery_max_active’, and possibly ‘osd_recovery_max_chunk’ to attempt to
speed it up, but will hopefully get some insight from the community here.
ceph -s about 4 days in:
> health HEALTH_WARN
> 255 pgs backfill_wait
> 4 pgs backfilling
> 385 pgs degraded
> 129 pgs recovery_wait
> 388 pgs stuck unclean
> 274 pgs undersized
> recovery 165319973/681597074 objects degraded (24.255%)
> recovery 298607229/681597074 objects misplaced (43.810%)
> noscrub,nodeep-scrub,sortbitwise flag(s) set
> monmap e1: 3 mons at
> {core=10.0.1.249:6789/0,db=10.0.1.251:6789/0,dev=10.0.1.250:6789/0}
> election epoch 190, quorum 0,1,2 core,dev,db
> osdmap e4226: 16 osds: 16 up, 16 in; 303 remapped pgs
> flags noscrub,nodeep-scrub,sortbitwise
> pgmap v1583742: 576 pgs, 2 pools, 6426 GB data, 292 Mobjects
> 15301 GB used, 101 TB / 116 TB avail
> 165319973/681597074 objects degraded (24.255%)
> 298607229/681597074 objects misplaced (43.810%)
> 249 active+undersized+degraded+remapped+wait_backfill
> 188 active+clean
> 85 active+recovery_wait+degraded
> 22 active+recovery_wait+degraded+remapped
> 22 active+recovery_wait+undersized+degraded+remapped
> 3 active+remapped+wait_backfill
> 3 active+undersized+degraded+remapped+backfilling
> 3 active+degraded+remapped+wait_backfill
> 1 active+degraded+remapped+backfilling
> recovery io 9361 kB/s, 415 objects/s
> client io 597 kB/s rd, 62 op/s rd, 0 op/s wr
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
