Hi everybody, I am trying to understand why am I keep on getting remapped+wait_backfill pg statuses, when doing some cluster pg shuffling. Sometimes it happens just by doing small reweight-by-utilization operation, and sometimes when I modify the crushmap (bigger movement of data).
Taking look at ceph health detail and investigating some of the pgs by using pg #ipg_id query, I can see that all of the "acting" pgs are healthy and of the same size. "Up" pgs do have a pg folder created, but dont have any data inside (empty head + TEMP). I dont have any (near)full pgs, and ceph pg debug unfound_objects_exist yields FALSE. Cluster is also 100% functional(but in WARN state), and I can see that if I write some data, acting pgs are all happily syncing between each other. Acting pgs Up pgs I can simply recover form this following this steps: - set noout, norecover, norebalance to avoid unnecessary data movement - stopping all actingbackfill pgs (active + up) at the same time - remove empty "up" pgs - start all the pgs again - unset noout, norecover, norebalance After that new "up" pgs are recreated in remapped+backfilling state, and marked as active+clean after some time. I have also tried to "kick the cluster in the head" with ceph pg send_pg_creates (as stated here https://www.mail-archive.com/[email protected]/msg12287.html), but I get: $ ceph pg send_pg_creates Error EINVAL: (22) Invalid argument BTW What is send_pg_creates really supposed to do? Does anyone have some hints is this occurring? Thank you, Ivan Jewel 12.2.2 size 3 min_size 2 #I havent been playing with tunables # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable chooseleaf_vary_r 1 tunable straw_calc_version 1 #standard ruleset rule replicated_ruleset { ruleset 0 type replicated min_size 1 max_size 10 step take dc step chooseleaf firstn 0 type host step emit }
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
