Hi All,
I believe we forgot to take the snapshot in the previous test. Here's the
output from current test where we took snapshot on Primary side but the
snapshot did not replicated to DR side?
VTIER1 is the Primary box with cluster ceph. Vtier2a is the DR box with cluster
name cephdr.
root@VTIER1:~# rbd ls -l nfs
NAME SIZE PARENT FMT PROT LOCK
dir_research 200TiB 2 excl
dir_research@dr_test 200TiB 2
test01 100MiB 2
root@VTIER1:~#
root@vtier2a:~# rbd ls -l nfs
NAME SIZE PARENT FMT PROT LOCK
dir_research 200TiB 2 excl
test01 100MiB 2 excl
root@vtier2a:~# rbd mirror pool status nfs --verbose --cluster=cephdr
health: OK
images: 2 total
2 replaying
dir_research:
global_id: 92f46320-d43d-48eb-8a09-b68a1945cc77
state: up+replaying
description: replaying, master_position=[object_number=597902, tag_tid=3,
entry_tid=705172054], mirror_position=[object_number=311129, tag_tid=3,
entry_tid=283416457], entries_behind_master=421755597
last_update: 2019-11-25 12:14:52
test01:
global_id: 06fbfe68-b7e4-4d3a-93b2-cd18c569f7f7
state: up+replaying
description: replaying, master_position=[object_number=3, tag_tid=1,
entry_tid=3], mirror_position=[object_number=3, tag_tid=1, entry_tid=3],
entries_behind_master=0
last_update: 2019-11-25 12:14:50
root@vtier2a:~# rbd-nbd --cluster=cephdr map nfs/dir_research@dr_test
2019-11-25 12:17:45.764091 7f8bd73c5dc0 -1 asok(0x55fd9a7202a0)
AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to
bind the UNIX domain socket to '/var/run/ceph/cephdr-client.admin.asok': (17)
File exists
Did we missed anything and why the snapshot didn't replicated to DR side?
Thanks,
-Vikas
-----Original Message-----
From: Jason Dillaman <[email protected]>
Sent: Thursday, November 21, 2019 10:24 AM
To: Vikas Rana <[email protected]>
Cc: dillaman <[email protected]>; ceph-users <[email protected]>
Subject: Re: [ceph-users] RBD Mirror DR Testing
On Thu, Nov 21, 2019 at 10:16 AM Vikas Rana <[email protected]> wrote:
>
> Thanks Jason.
> We are just mounting and verifying the directory structure and make sure it
> looks good.
>
> My understanding was, in 12.2.10, we can't mount the DR snapshot as the RBD
> image is non-primary. Is this wrong?
You have always been able to access non-primary images for read-only operations
(only writes are prevented):
$ rbd info test
rbd image 'test':
<... snip ...>
mirroring primary: false
$ rbd device --device-type nbd map test@1
/dev/nbd0
$ mount /dev/nbd0 /mnt/
mount: /mnt: WARNING: device write-protected, mounted read-only.
$ ll /mnt/
total 0
-rw-r--r--. 1 root root 0 Nov 21 10:20 hello.world
> Thanks,
> -Vikas
>
> -----Original Message-----
> From: Jason Dillaman <[email protected]>
> Sent: Thursday, November 21, 2019 9:58 AM
> To: Vikas Rana <[email protected]>
> Cc: ceph-users <[email protected]>
> Subject: Re: [ceph-users] RBD Mirror DR Testing
>
> On Thu, Nov 21, 2019 at 9:56 AM Jason Dillaman <[email protected]> wrote:
> >
> > On Thu, Nov 21, 2019 at 8:49 AM Vikas Rana <[email protected]> wrote:
> > >
> > > Thanks Jason for such a quick response. We are on 12.2.10.
> > >
> > > Checksuming a 200TB image will take a long time.
> >
> > How would mounting an RBD image and scanning the image be faster?
> > Are you only using a small percentage of the image?
>
> ... and of course, you can mount an RBD snapshot in read-only mode.
>
> > > To test the DR copy by mounting it, these are the steps I'm
> > > planning to follow 1. Demote the Prod copy and promote the DR copy
> > > 2. Do we have to recreate the rbd mirror relationship going from DR to
> > > primary?
> > > 3. Mount and validate the data
> > > 4. Demote the DR copy and promote the Prod copy 5. Revert the peer
> > > relationship if required?
> > >
> > > Did I do it right or miss anything?
> >
> > You cannot change the peers or you will lose the relationship. If
> > you insist on your course of action, you just need to be configured
> > for two-way mirroring and leave it that way.
> >
> > >
> > > Thanks,
> > > -Vikas
> > >
> > > -----Original Message-----
> > > From: Jason Dillaman <[email protected]>
> > > Sent: Thursday, November 21, 2019 8:33 AM
> > > To: Vikas Rana <[email protected]>
> > > Cc: ceph-users <[email protected]>
> > > Subject: Re: [ceph-users] RBD Mirror DR Testing
> > >
> > > On Thu, Nov 21, 2019 at 8:29 AM Vikas Rana <[email protected]> wrote:
> > > >
> > > > Hi all,
> > > >
> > > >
> > > >
> > > > We have a 200TB RBD image which we are replicating using RBD mirroring.
> > > >
> > > > We want to test the DR copy and make sure that we have a consistent
> > > > copy in case primary site is lost.
> > > >
> > > >
> > > >
> > > > We did it previously and promoted the DR copy which broken the DR copy
> > > > from primary and we have to resync the whole 200TB data.
> > > >
> > > >
> > > >
> > > > Is there any correct way of doing it so we don’t have to resync all
> > > > 200TB again?
> > >
> > > Yes, create a snapshot on the primary site and let it propagate to the
> > > non-primary site. Then you can compare checksums at the snapshot w/o
> > > having to worry about the data changing. Once you have finished, delete
> > > the snapshot on the primary site and it will propagate over to the
> > > non-primary site.
> > >
> > > >
> > > >
> > > > Can we demote current primary and then promote the DR copy and test and
> > > > then revert back? Will that require the complete 200TB sync?
> > > >
> > >
> > > It's only the forced-promotion that causes split-brain. If you gracefully
> > > demote from site A and promote site B, and then demote site B and promote
> > > site A, that will not require a sync. However, again, it's probably just
> > > easier to use a snapshot.
> > >
> > > >
> > > > Thanks in advance for your help and suggestions.
> > > >
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > -Vikas
> > > >
> > > > _______________________________________________
> > > > ceph-users mailing list
> > > > [email protected]
> > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > >
> > >
> > >
> > > --
> > > Jason
> > >
> > >
> >
> >
> > --
> > Jason
>
>
>
> --
> Jason
>
>
--
Jason
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
