[ceph-users] Auto recovering after loosing all copies of a PG(s)

2016-08-16 Thread Iain Buclaw
f it can still be accessed is just not workable, I'm afraid. Thanks ahead of time for any suggestions. -- Iain Buclaw *(p < e ? p++ : p) = (c & 0x0f) + '0'; ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Auto recovering after loosing all copies of a PG(s)

2016-09-01 Thread Iain Buclaw
On 16 August 2016 at 17:13, Wido den Hollander <w...@42on.com> wrote: > >> Op 16 augustus 2016 om 15:59 schreef Iain Buclaw <ibuc...@gmail.com>: >> >> >> The desired behaviour for me would be for the client to get an instant >> "not found" re

Re: [ceph-users] Give up on backfill, remove slow OSD

2016-10-08 Thread Iain Buclaw
On 3 October 2016 at 07:30, Ronny Aasen <ronny+ceph-us...@aasen.cx> wrote: > On 22. sep. 2016 09:16, Iain Buclaw wrote: >> >> Hi, >> >> I currently have an OSD that has been backfilling data off it for a >> little over two days now, and it's gone from appro

[ceph-users] Stat speed for objects in ceph

2016-09-20 Thread Iain Buclaw
at all. And cache won't help either, there is no determinism in what's accessed, and given the size of the pool OS filesystem cache is useless anyway. Thanks, -- Iain Buclaw *(p < e ? p++ : p) = (c & 0x0f) + '0'; ___ ceph-users mailing list cep

Re: [ceph-users] Stat speed for objects in ceph

2016-09-21 Thread Iain Buclaw
ay with it, and I assume with another very large index and actively keeping track of all set destruction times. -- Iain Buclaw ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Stat speed for objects in ceph

2016-09-21 Thread Iain Buclaw
of random small file IO compared to FileStore. But this is only with around 1/25th of the data we are holding. So having an index pool is the only way to get faster lookup speeds? I don't think having one is really for my use case, with billions of objects being held, I don't think maintaining such an index would be any quicker than what rados_stat() is capable of achieving already. In any case, these clients maintain and validate the data that's stored, it would inherently assume that any index is wrong. -- Iain Buclaw ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Stat speed for objects in ceph

2016-09-21 Thread Iain Buclaw
? And it would do no harm prefetching that data before it's requested by the rados client. -- Iain Buclaw ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Stat speed for objects in ceph

2016-09-23 Thread Iain Buclaw
urrent running config shows objecter_inflight_ops = 10240 objecter_completion_locks_per_session = 32 filestore_expected_throughput_ops = 200 I wonder what effect might happen if I increase any of these. -- Iain Buclaw ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Give up on backfill, remove slow OSD

2016-09-22 Thread Iain Buclaw
be stored on it. Thanks -- Iain Buclaw *(p < e ? p++ : p) = (c & 0x0f) + '0'; ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Auto recovering after loosing all copies of a PG(s)

2016-09-20 Thread Iain Buclaw
On 1 September 2016 at 23:04, Wido den Hollander <w...@42on.com> wrote: > >> Op 1 september 2016 om 17:37 schreef Iain Buclaw <ibuc...@gmail.com>: >> >> >> On 16 August 2016 at 17:13, Wido den Hollander <w...@42on.com> wrote: >> > >&g

Re: [ceph-users] Down OSDs blocking read requests.

2016-11-18 Thread Iain Buclaw
On 18 November 2016 at 13:14, John Spray <jsp...@redhat.com> wrote: > On Fri, Nov 18, 2016 at 11:53 AM, Iain Buclaw <ibuc...@gmail.com> wrote: >> Hi, >> >> Follow up from the suggestion to use any of the following options: >> >> - c

[ceph-users] Down OSDs blocking read requests.

2016-11-18 Thread Iain Buclaw
to cause more false positives. But in reality, if an OSD operation can't serve in 150ms, then it's missed the train by over an hour. -- Iain Buclaw *(p < e ? p++ : p) = (c & 0x0f) + '0'; ___ ceph-users mailing list ceph-users@lists.ceph.c

Re: [ceph-users] RGW: Reshard index of non-master zones in multi-site

2019-02-05 Thread Iain Buclaw
On Tue, 5 Feb 2019 at 09:46, Iain Buclaw wrote: > > Hi, > > Following the update of one secondary site from 12.2.8 to 12.2.11, the > following warning have come up. > > HEALTH_WARN 1 large omap objects > LARGE_OMAP_OBJECTS 1 large omap objects > 1

[ceph-users] RGW: Reshard index of non-master zones in multi-site

2019-02-05 Thread Iain Buclaw
"index_pool": "" } }, "owner": "mybucket", "creation_time": "2018-03-27 19:05:22.776182Z", "linked": "true", "has_bucket_info": "false" } } Is this the reason why resharding hasn't propagated? -- Iain Buclaw *(p < e ? p++ : p) = (c & 0x0f) + '0'; ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW: Reshard index of non-master zones in multi-site

2019-02-06 Thread Iain Buclaw
On Tue, 5 Feb 2019 at 10:04, Iain Buclaw wrote: > > On Tue, 5 Feb 2019 at 09:46, Iain Buclaw wrote: > > > > Hi, > > > > Following the update of one secondary site from 12.2.8 to 12.2.11, the > > following warning have come up. > > > > HEALTH_WA

Re: [ceph-users] RGW: Reshard index of non-master zones in multi-site

2019-02-19 Thread Iain Buclaw
On Wed, 6 Feb 2019 at 09:28, Iain Buclaw wrote: > > On Tue, 5 Feb 2019 at 10:04, Iain Buclaw wrote: > > > > On Tue, 5 Feb 2019 at 09:46, Iain Buclaw wrote: > > > > > > Hi, > > > > > > Following the update of one secondary site from 12.2.

Re: [ceph-users] RGW: Reshard index of non-master zones in multi-site

2019-02-19 Thread Iain Buclaw
On Tue, 19 Feb 2019 at 09:59, Iain Buclaw wrote: > > On Wed, 6 Feb 2019 at 09:28, Iain Buclaw wrote: > > > > On Tue, 5 Feb 2019 at 10:04, Iain Buclaw wrote: > > > > > > On Tue, 5 Feb 2019 at 09:46, Iain Buclaw wrote: > > > > > > > &g

Re: [ceph-users] RGW: Reshard index of non-master zones in multi-site

2019-02-19 Thread Iain Buclaw
On Tue, 19 Feb 2019 at 10:05, Iain Buclaw wrote: > > On Tue, 19 Feb 2019 at 09:59, Iain Buclaw wrote: > > > > On Wed, 6 Feb 2019 at 09:28, Iain Buclaw wrote: > > > > > > On Tue, 5 Feb 2019 at 10:04, Iain Buclaw wrote: > > > > > &g

Re: [ceph-users] RGW: Reshard index of non-master zones in multi-site

2019-04-03 Thread Iain Buclaw
On Tue, 19 Feb 2019 at 10:11, Iain Buclaw wrote: > > On Tue, 19 Feb 2019 at 10:05, Iain Buclaw wrote: > > > > On Tue, 19 Feb 2019 at 09:59, Iain Buclaw wrote: > > > > > > On Wed, 6 Feb 2019 at 09:28, Iain Buclaw wrote: > > > > > &g

Re: [ceph-users] RGW: Reshard index of non-master zones in multi-site

2019-04-03 Thread Iain Buclaw
On Tue, 19 Feb 2019 at 10:11, Iain Buclaw wrote: > > > # ./radosgw-gc-bucket-indexes.sh master.rgw.buckets.index | wc -l > 7511 > > # ./radosgw-gc-bucket-indexes.sh secondary1.rgw.buckets.index | wc -l > 3509 > > # ./radosgw-gc-bucket-indexes.sh secondary2.rgw.b

Re: [ceph-users] ceph osd pg-upmap-items not working

2019-04-04 Thread Iain Buclaw
ize myplan ` param. > >From experimenting, in three different clusters, this is not quite right. I've found that the balancer is quite unable to optimize correctly if you have mixed sized OSDs, even if there's only one OSD that's bigger by 12 GBs. -- Iain Buclaw *(p < e ?

Re: [ceph-users] RGW: Reshard index of non-master zones in multi-site

2019-04-04 Thread Iain Buclaw
On Wed, 3 Apr 2019 at 09:41, Iain Buclaw wrote: > > On Tue, 19 Feb 2019 at 10:11, Iain Buclaw wrote: > > > > > > # ./radosgw-gc-bucket-indexes.sh master.rgw.buckets.index | wc -l > > 7511 > > > > # ./radosgw-gc-bucket-indexes.sh secondary1.rgw.buckets.

Re: [ceph-users] ceph osd pg-upmap-items not working

2019-04-08 Thread Iain Buclaw
s for your issues, please do create a tracker. > Upgraded to v13.2.x, and it's still the same. https://tracker.ceph.com/issues/39136 -- Iain Buclaw *(p < e ? p++ : p) = (c & 0x0f) + '0'; ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW: Reshard index of non-master zones in multi-site

2019-04-08 Thread Iain Buclaw
tools able to work with that? On the secondaries, it is no problem to proxy_pass all requests to the master whilst all rgw pools are destroyed and recreated. I would have though however that manually removing the known orphaned indexes be safe though, side-stepping the annoying job of having to force degrade the working

Re: [ceph-users] RGW: Reshard index of non-master zones in multi-site

2019-04-06 Thread Iain Buclaw
ets is fixed in my cluster, there's no chance of misidentifying anything. They're dead index objects, and they have been causing large omap problems, it looks like the corrective action is to hand delete these and restart radosgw, but no one else who has done investigations into this has confirmed o

Re: [ceph-users] RGW: Reshard index of non-master zones in multi-site

2019-04-17 Thread Iain Buclaw
On Mon, 8 Apr 2019 at 10:33, Iain Buclaw wrote: > > On Mon, 8 Apr 2019 at 05:01, Matt Benjamin wrote: > > > > Hi Christian, > > > > Dynamic bucket-index sharding for multi-site setups is being worked > > on, and will land in the N release cycle. > &g