[ceph-users] radosgw-admin bucket check lasts forever
Hi! A followup to my previous message http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-February/007392.html . Due to some maintenance we had in out network I restarted all the nodes and the gateway one by one. Now when I run radosgw-admin bucket check --fix --check-objects --bucket %bucket name% it runs for hours (23 hours already) on a ~10 GB bucket with ~1500 objects. I see some activity going on with "ceph -w", ~200 op/s and ~30mb/s of reads and nothing more. BTW, the same applies to radosgw-admin bucket list --bucket %bucket name% I can still use tools like s3 explorer and s3 api works on this bucket, except listing of one specific prefix though. Is there a way to trace what is going on? Is there any description on the bucket to pool mapping internals, so I can track the inconsistency myself? -- Best regards, Alexey Kuntsevich ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Bucket listing requests get stuck
Hi! I have a ver 0.94.5 debian-based cluster used mostly through rados. I tried to delete objects with the same prefix from one of the buckets (~1300 objects) using a python boto library. The process finished after several minutes without any errors, but now I can list only a subset (~20) of objects in this bucket and if I increase the number of objects to list even to 25 the request hangs for hours. I can still access objects directly in this bucket and list objects with any other prefix than the one that was used for deletion. I tried rebooting nodes and the gateway server, checked the radosgw logs (nothing except messages with 200 and 499 return codes) and doing random maintenance tasks that I was able to find in the documentation. Is there a way to fix the issue without moving the rest of the data into another bucket and dropping the old bucket with radosgw-admin? Are there any monitoring means that can show locks/data consistency issues for radosgw? Is it possible to trace where the request gets stuck? Is there any documentation on how radosgw stores its data inside ceph? Any help is appreciated! -- Best regards, Alexey Kuntsevich ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
