As mentioned in another thread I'm trying to remove several thousand buckets on a hammer cluster (0.94.10), but I'm running into a problem using --bypass-gc.
I usually see either this error: # radosgw-admin bucket rm --bucket=sg2pl598 --purge-objects --bypass-gc 2017-10-31 09:21:04.111599 7f45f5d108c0 0 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2017-10-31 09:21:04.121664 7f45f5d108c0 0 RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2017-10-31 09:21:04.126356 7f45f5d108c0 0 RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2017-10-31 09:21:04.130582 7f45f5d108c0 0 RGWObjManifest::operator++(): result: ofs=15728640 stripe_ofs=15728640 part_ofs=15728640 rule->part_size=15728640 2017-10-31 09:21:04.135791 7f45f5d108c0 0 RGWObjManifest::operator++(): result: ofs=19922944 stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2017-10-31 09:21:04.140240 7f45f5d108c0 0 RGWObjManifest::operator++(): result: ofs=24117248 stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2017-10-31 09:21:04.145792 7f45f5d108c0 0 RGWObjManifest::operator++(): result: ofs=28311552 stripe_ofs=28311552 part_ofs=15728640 rule->part_size=15728640 2017-10-31 09:21:04.149964 7f45f5d108c0 0 RGWObjManifest::operator++(): result: ofs=31457280 stripe_ofs=31457280 part_ofs=31457280 rule->part_size=15728640 2017-10-31 09:21:04.165820 7f45f5d108c0 0 RGWObjManifest::operator++(): result: ofs=35651584 stripe_ofs=35651584 part_ofs=31457280 rule->part_size=15728640 2017-10-31 09:21:04.171099 7f45f5d108c0 0 RGWObjManifest::operator++(): result: ofs=39845888 stripe_ofs=39845888 part_ofs=31457280 rule->part_size=15728640 2017-10-31 09:21:04.176765 7f45f5d108c0 0 RGWObjManifest::operator++(): result: ofs=44040192 stripe_ofs=44040192 part_ofs=31457280 rule->part_size=15728640 2017-10-31 09:21:04.183664 7f45f5d108c0 0 RGWObjManifest::operator++(): result: ofs=47185920 stripe_ofs=47185920 part_ofs=47185920 rule->part_size=83674 2017-10-31 09:21:04.188140 7f45f5d108c0 0 RGWObjManifest::operator++(): result: ofs=47269594 stripe_ofs=47269594 part_ofs=47269594 rule->part_size=83674 2017-10-31 09:21:05.034837 7f45f5d108c0 -1 ERROR: failed to get obj ref with ret=-22 2017-10-31 09:21:05.034846 7f45f5d108c0 -1 ERROR: delete obj aio failed with -22 or this error: # radosgw-admin bucket rm --bucket=sg2pl593 --purge-objects --bypass-gc 2017-10-31 09:24:09.082063 7fe7f4be68c0 0 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2017-10-31 09:24:09.090394 7fe7f4be68c0 0 RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2017-10-31 09:24:09.095172 7fe7f4be68c0 0 RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2017-10-31 09:24:09.099116 7fe7f4be68c0 0 RGWObjManifest::operator++(): result: ofs=15728640 stripe_ofs=15728640 part_ofs=15728640 rule->part_size=15728640 [...snip...] 2017-10-31 09:24:09.245171 7fe7f4be68c0 0 RGWObjManifest::operator++(): result: ofs=110100480 stripe_ofs=110100480 part_ofs=110100480 rule->part_size=15728640 2017-10-31 09:24:09.251659 7fe7f4be68c0 0 RGWObjManifest::operator++(): result: ofs=114294784 stripe_ofs=114294784 part_ofs=110100480 rule->part_size=15728640 2017-10-31 09:24:09.269739 7fe7f4be68c0 0 RGWObjManifest::operator++(): result: ofs=118489088 stripe_ofs=118489088 part_ofs=110100480 rule->part_size=15728640 2017-10-31 09:24:09.273871 7fe7f4be68c0 0 RGWObjManifest::operator++(): result: ofs=122683392 stripe_ofs=122683392 part_ofs=110100480 rule->part_size=15728640 2017-10-31 09:24:09.274968 7fe7f4be68c0 -1 ERROR: could not drain handles as aio completion returned with -2 Then successive runs continue failing at the same spot preventing further progress. I can then run it without --bypass-gc for a few seconds followed by running it with --bypass-gc, but usually it fails again after a few minutes. For example, here's another run on sg2pl593 after running it without --bypass-gc for a few seconds: # radosgw-admin bucket rm --bucket=sg2pl593 --purge-objects --bypass-gc 2017-10-31 09:28:03.704490 7efdb31d08c0 0 RGWObjManifest::operator++(): result: ofs=565628 stripe_ofs=565628 part_ofs=0 rule->part_size=0 2017-10-31 09:28:03.890675 7efdb31d08c0 0 RGWObjManifest::operator++(): result: ofs=1757663 stripe_ofs=1757663 part_ofs=0 rule->part_size=0 2017-10-31 09:28:04.144966 7efdb31d08c0 0 RGWObjManifest::operator++(): result: ofs=2723340 stripe_ofs=2723340 part_ofs=0 rule->part_size=0 2017-10-31 09:28:04.380761 7efdb31d08c0 -1 ERROR: could not drain handles as aio completion returned with -2 This cluster recently switched from a production cluster to a test cluster after a data migration, so I have the option to test new builds out easily. However, we'll probably only keep it around for another month or so before wiping the drives and adding it to the new production cluster. Thanks, Bryan _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
