Dear everyone, Update, on Friday we decided to apply blunt force to the problem (this still being a prototype cluster for the time being) and started rebooting nodes, one by one, again. After the second rebooted node had come back online the misplaced percentage jumped by about 10 percent but we finally began to see 'ceph -s' report recovery traffic - and when I checked this morning all PGs were back to normal.
Just in case this is helps anyone figure out what did happen, so that we can avoid this / fix it faster in the future: * CRUSH: we do group our hosts into buckets corresponding to the server rooms they are located in (of which there are three). Apart from that, nothing non-default; * EC pools - yes, we have got some of these. See below. * pool and CRUSH-rule details: pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 545 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 50.00 pool 2 'cephfs.fs1.meta' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 548 lfor 0/0/153 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 3.02 pool 3 'cephfs.fs1.data' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 38714 lfor 0/0/201 flags hashpspool,selfmanaged_snaps,bulk stripe_width 0 application cephfs read_balance_score 1.08 pool 4 '.nfs' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 546 lfor 0/0/233 flags hashpspool stripe_width 0 application nfs read_balance_score 2.99 pool 5 'cephfs.fs1.repl1' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 38714 lfor 0/0/264 flags hashpspool,selfmanaged_snaps,bulk stripe_width 0 application cephfs read_balance_score 1.13 pool 8 '.rgw.root' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20219 lfor 0/0/569 flags hashpspool stripe_width 0 application rgw read_balance_score 4.52 pool 9 'default.rgw.log' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20221 lfor 0/0/571 flags hashpspool stripe_width 0 application rgw read_balance_score 2.99 pool 10 'default.rgw.control' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20226 lfor 0/0/571 flags hashpspool stripe_width 0 application rgw read_balance_score 4.49 pool 11 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20231 lfor 0/0/573 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 2.99 pool 15 '.smb' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20233 lfor 0/0/6089 flags hashpspool stripe_width 0 application smb read_balance_score 4.50 pool 16 'cephfs.fs1.ec1' erasure profile isa_rsv_4p2_host_hdd size 6 min_size 5 crush_rule 4 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 38714 lfor 0/0/9461 flags hashpspool,ec_overwrites,selfmanaged_snaps,bulk stripe_width 16384 fast_read 1 application cephfs Where the relevant CRUSH rules are: * 4 - erasure, on hard drives only, host-level failure domain * 6 - replicated, on NVMe drives only, room-level failure domain -- MS _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
