On Wed, Mar 1, 2017 at 9:06 AM, Marius Vaitiekunas < [email protected]> wrote:
> > > On Mon, Feb 27, 2017 at 11:40 AM, Marius Vaitiekunas < > [email protected]> wrote: > >> >> >> On Mon, Feb 27, 2017 at 9:59 AM, Marius Vaitiekunas < >> [email protected]> wrote: >> >>> >>> >>> On Fri, Feb 24, 2017 at 6:35 PM, Yehuda Sadeh-Weinraub < >>> [email protected]> wrote: >>> >>>> On Fri, Feb 24, 2017 at 3:59 AM, Marius Vaitiekunas >>>> <[email protected]> wrote: >>>> > >>>> > >>>> > On Wed, Feb 22, 2017 at 8:33 PM, Yehuda Sadeh-Weinraub < >>>> [email protected]> >>>> > wrote: >>>> >> >>>> >> On Wed, Feb 22, 2017 at 6:19 AM, Marius Vaitiekunas >>>> >> <[email protected]> wrote: >>>> >> > Hi Cephers, >>>> >> > >>>> >> > We are testing rgw multisite solution between to DC. We have one >>>> >> > zonegroup >>>> >> > and to zones. At the moment all writes/deletes are done only to >>>> primary >>>> >> > zone. >>>> >> > >>>> >> > Sometimes not all the objects are replicated.. We've written >>>> prometheus >>>> >> > exporter to check replication status. It gives us each bucket >>>> object >>>> >> > count >>>> >> > from user perspective, because we have millions of objects and >>>> hundreds >>>> >> > of >>>> >> > buckets. We just want to be sure, that everything is replicated >>>> without >>>> >> > using ceph internals like rgw admin api for now. >>>> >> > >>>> >> > Is it possible to initiate full resync of only one rgw bucket from >>>> >> > master >>>> >> > zone? What are the options about resync when things go wrong and >>>> >> > replication >>>> >> > misses some objects? >>>> >> > >>>> >> > We run latest jewel 10.2.5. >>>> >> >>>> >> >>>> >> There's the 'radosgw-admin bucket sync init' command that you can run >>>> >> on the specific bucket on the target zone. This will reinitialize the >>>> >> sync state, so that when it starts syncing it will go through the >>>> >> whole full sync process. Note that it shouldn't actually copy data >>>> >> that already exists on the target. Also, in order to actually start >>>> >> the sync, you'll need to have some change that would trigger the sync >>>> >> on that bucket, e.g., create a new object there. >>>> >> >>>> >> Yehuda >>>> >> >>>> > >>>> > Hi, >>>> > >>>> > I've tried to resync a bucket, but it didn't manage to resync a >>>> missing >>>> > object. If I try to copy missing object by hand into secondary zone, >>>> i get >>>> > asked to overwrite existing object.. It looks like the object is >>>> replicated, >>>> > but is not in a bucket index. I've tried to check bucket index with >>>> --fix >>>> > and --check-objects flags, but nothing changes. What else should i >>>> try? >>>> > >>>> >>>> That's weird. Do you see anything when you run 'radosgw-admin bi list >>>> --bucket=<bucket>'? >>>> >>>> Yehuda >>>> >>> >>> 'radosgw-admin bi list --bucket=<bucket>' gives me an error: >>> 2017-02-27 08:55:30.861659 7f20c15779c0 0 error in read_id for id : >>> (2) No such file or directory >>> 2017-02-27 08:55:30.861991 7f20c15779c0 0 error in read_id for id : >>> (2) No such file or directory >>> ERROR: bi_list(): (5) Input/output error >>> >>> 'radosgw-admin bucket list --bucket=<bucket>' successfully list all the >>> files except missing ones. >>> >>> >>> >>> >> >> I've done some more investigation. These missing objects could be found >> in "rgw.buckets.data" pool, but bucket index is not aware about them. >> How does 'radosgw-admin bucket check -b <bucket> --fix --check-objects' >> works? >> I guess that it's not scanning "rgw.buckets.data" pool for "leaked" >> objects? These unreplicated objects looks for me the same like leaked ones >> :) >> >> >> > By the way in rgw logs I can find all the missing files with http 304 > return code. For example: > "GET /go84/WRWRDGROWKFKROTWKHXXIBHERRLHBK HTTP/1.1" 304 0 - - > > All the gateways in both sites are behind haproxies. Any ideas? > > > > > Sorry.. Don't pay attention to my last message. Wrong 'grep'. -- Marius Vaitiekūnas
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
