Re: [ceph-users] pool/volume live migration
On Mon, Feb 11, 2019 at 4:53 AM Luis Periquito wrote: > > Hi Jason, > > that's been very helpful, but it got me thinking and looking. > > The pool name is both inside the libvirt.xml (and running KVM config) > and it's cached in the Nova database. For it to change would require a > detach/attach which may not be viable or easy, specially for boot > volumes. > > What about: > 1 - upgrade all binaries to Nautilus and ensure Ceph daemons are > restarted and all are running Nautilus > 2 - stop the OpenStack instances that are on that pool > 3 - rename "openstack" pool to "old.openstack" > 4 - create new pool "openstack" > 5 - "rbd migration" prepare all the RBD volumes in the "old.openstack" > 6 - start all instances again. This should ensure all instances are > running with the Nautilius binaries > 7 - run the "rbd migration execute" and "rbd migration commit" as > performance allows > > when all is finished delete the "old.openstack" pool. > > As we're running an EC+CT environment will this still apply? Should we > remove the CT before doing this or will Ceph be aware of the Cache > Tier and do it transparently? > > I will test all this a few times before doing it in any significant > environment. I will do my best to share the experience in the mailing > list after... > Is there anything we should be aware, and is there anything I could > report back to the community to test/experiment/debug the solution? > > and thanks for all your help. I think that should work. You won't be able to remove the cache tier against the old pool since directly storing RBD images on an EC pool is not possible. However, internally, Ceph should be using the unique pool id (and not the pool name), so the cache tier will stick to the original pool even after a rename. > On Fri, Feb 8, 2019 at 5:20 PM Jason Dillaman wrote: > > > > On Fri, Feb 8, 2019 at 11:43 AM Luis Periquito wrote: > > > > > > This is indeed for an OpenStack cloud - it didn't require any level of > > > performance (so was created on an EC pool) and now it does :( > > > > > > So the idea would be: > > > > 0 - upgrade OSDs and librbd clients to Nautilus > > > > > 1- create a new pool > > > > Are you using EC via cache tier over a replicated pool or an RBD image > > with an EC data pool? > > > > > 2- change cinder to use the new pool > > > > > > for each volume > > > 3- stop the usage of the volume (stop the instance?) > > > 4- "live migrate" the volume to the new pool > > > > Yes, execute the "rbd migration prepare" step here and manually update > > the Cinder database to point the instance to the new pool (if the pool > > name changed). I cannot remember if Nova caches the Cinder volume > > connector details, so you might also need to detach/re-attach the > > volume if that's the case (or tweak the Nova database entries as > > well). > > > > > 5- start up the instance again > > > > 6 - run "rbd migration execute" and "rbd migration commit" at your > > convenience. > > > > > > > > > > > Does that sound right? > > > > > > thanks, > > > > > > On Fri, Feb 8, 2019 at 4:25 PM Jason Dillaman wrote: > > > > > > > > Correction: at least for the initial version of live-migration, you > > > > need to temporarily stop clients that are using the image, execute > > > > "rbd migration prepare", and then restart the clients against the new > > > > destination image. The "prepare" step will fail if it detects that the > > > > source image is in-use. > > > > > > > > On Fri, Feb 8, 2019 at 9:00 AM Jason Dillaman > > > > wrote: > > > > > > > > > > Indeed, it is forthcoming in the Nautilus release. > > > > > > > > > > You would initiate a "rbd migration prepare > > > > > " to transparently link the dst-image-spec to the > > > > > src-image-spec. Any active Nautilus clients against the image will > > > > > then re-open the dst-image-spec for all IO operations. Read requests > > > > > that cannot be fulfilled by the new dst-image-spec will be forwarded > > > > > to the original src-image-spec (similar to how parent/child cloning > > > > > behaves). Write requests to the dst-image-spec will force a deep-copy > > > > > of all impacted src-image-spec backing data objects (including > > > > > snapshot history) to the associated dst-image-spec backing data > > > > > object. At any point a storage admin can run "rbd migration execute" > > > > > to deep-copy all src-image-spec data blocks to the dst-image-spec. > > > > > Once the migration is complete, you would just run "rbd migration > > > > > commit" to remove src-image-spec. > > > > > > > > > > Note: at some point prior to "rbd migration commit", you will need to > > > > > take minimal downtime to switch OpenStack volume registration from the > > > > > old image to the new image if you are changing pools. > > > > > > > > > > On Fri, Feb 8, 2019 at 5:33 AM Caspar Smit > > > > > wrote: > > > > > > > > > > > > Hi Luis, > > > > > > > > > > > > According to slide 21 of Sage's presentation at FOSDEM it is coming > > > > > >
Re: [ceph-users] pool/volume live migration
On Fri, Feb 8, 2019 at 11:43 AM Luis Periquito wrote: > > This is indeed for an OpenStack cloud - it didn't require any level of > performance (so was created on an EC pool) and now it does :( > > So the idea would be: 0 - upgrade OSDs and librbd clients to Nautilus > 1- create a new pool Are you using EC via cache tier over a replicated pool or an RBD image with an EC data pool? > 2- change cinder to use the new pool > > for each volume > 3- stop the usage of the volume (stop the instance?) > 4- "live migrate" the volume to the new pool Yes, execute the "rbd migration prepare" step here and manually update the Cinder database to point the instance to the new pool (if the pool name changed). I cannot remember if Nova caches the Cinder volume connector details, so you might also need to detach/re-attach the volume if that's the case (or tweak the Nova database entries as well). > 5- start up the instance again 6 - run "rbd migration execute" and "rbd migration commit" at your convenience. > > > Does that sound right? > > thanks, > > On Fri, Feb 8, 2019 at 4:25 PM Jason Dillaman wrote: > > > > Correction: at least for the initial version of live-migration, you > > need to temporarily stop clients that are using the image, execute > > "rbd migration prepare", and then restart the clients against the new > > destination image. The "prepare" step will fail if it detects that the > > source image is in-use. > > > > On Fri, Feb 8, 2019 at 9:00 AM Jason Dillaman wrote: > > > > > > Indeed, it is forthcoming in the Nautilus release. > > > > > > You would initiate a "rbd migration prepare > > > " to transparently link the dst-image-spec to the > > > src-image-spec. Any active Nautilus clients against the image will > > > then re-open the dst-image-spec for all IO operations. Read requests > > > that cannot be fulfilled by the new dst-image-spec will be forwarded > > > to the original src-image-spec (similar to how parent/child cloning > > > behaves). Write requests to the dst-image-spec will force a deep-copy > > > of all impacted src-image-spec backing data objects (including > > > snapshot history) to the associated dst-image-spec backing data > > > object. At any point a storage admin can run "rbd migration execute" > > > to deep-copy all src-image-spec data blocks to the dst-image-spec. > > > Once the migration is complete, you would just run "rbd migration > > > commit" to remove src-image-spec. > > > > > > Note: at some point prior to "rbd migration commit", you will need to > > > take minimal downtime to switch OpenStack volume registration from the > > > old image to the new image if you are changing pools. > > > > > > On Fri, Feb 8, 2019 at 5:33 AM Caspar Smit wrote: > > > > > > > > Hi Luis, > > > > > > > > According to slide 21 of Sage's presentation at FOSDEM it is coming in > > > > Nautilus: > > > > > > > > https://fosdem.org/2019/schedule/event/ceph_project_status_update/attachments/slides/3251/export/events/attachments/ceph_project_status_update/slides/3251/ceph_new_in_nautilus.pdf > > > > > > > > Kind regards, > > > > Caspar > > > > > > > > Op vr 8 feb. 2019 om 11:24 schreef Luis Periquito : > > > >> > > > >> Hi, > > > >> > > > >> a recurring topic is live migration and pool type change (moving from > > > >> EC to replicated or vice versa). > > > >> > > > >> When I went to the OpenStack open infrastructure (aka summit) Sage > > > >> mentioned about support of live migration of volumes (and as a result > > > >> of pools) in Nautilus. Is this still the case and is expected to have > > > >> live migration working by then? > > > >> > > > >> thanks, > > > >> ___ > > > >> ceph-users mailing list > > > >> ceph-users@lists.ceph.com > > > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > ___ > > > > ceph-users mailing list > > > > ceph-users@lists.ceph.com > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > > > -- > > > Jason > > > > > > > > -- > > Jason > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Jason ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] pool/volume live migration
This is indeed for an OpenStack cloud - it didn't require any level of performance (so was created on an EC pool) and now it does :( So the idea would be: 1- create a new pool 2- change cinder to use the new pool for each volume 3- stop the usage of the volume (stop the instance?) 4- "live migrate" the volume to the new pool 5- start up the instance again Does that sound right? thanks, On Fri, Feb 8, 2019 at 4:25 PM Jason Dillaman wrote: > > Correction: at least for the initial version of live-migration, you > need to temporarily stop clients that are using the image, execute > "rbd migration prepare", and then restart the clients against the new > destination image. The "prepare" step will fail if it detects that the > source image is in-use. > > On Fri, Feb 8, 2019 at 9:00 AM Jason Dillaman wrote: > > > > Indeed, it is forthcoming in the Nautilus release. > > > > You would initiate a "rbd migration prepare > > " to transparently link the dst-image-spec to the > > src-image-spec. Any active Nautilus clients against the image will > > then re-open the dst-image-spec for all IO operations. Read requests > > that cannot be fulfilled by the new dst-image-spec will be forwarded > > to the original src-image-spec (similar to how parent/child cloning > > behaves). Write requests to the dst-image-spec will force a deep-copy > > of all impacted src-image-spec backing data objects (including > > snapshot history) to the associated dst-image-spec backing data > > object. At any point a storage admin can run "rbd migration execute" > > to deep-copy all src-image-spec data blocks to the dst-image-spec. > > Once the migration is complete, you would just run "rbd migration > > commit" to remove src-image-spec. > > > > Note: at some point prior to "rbd migration commit", you will need to > > take minimal downtime to switch OpenStack volume registration from the > > old image to the new image if you are changing pools. > > > > On Fri, Feb 8, 2019 at 5:33 AM Caspar Smit wrote: > > > > > > Hi Luis, > > > > > > According to slide 21 of Sage's presentation at FOSDEM it is coming in > > > Nautilus: > > > > > > https://fosdem.org/2019/schedule/event/ceph_project_status_update/attachments/slides/3251/export/events/attachments/ceph_project_status_update/slides/3251/ceph_new_in_nautilus.pdf > > > > > > Kind regards, > > > Caspar > > > > > > Op vr 8 feb. 2019 om 11:24 schreef Luis Periquito : > > >> > > >> Hi, > > >> > > >> a recurring topic is live migration and pool type change (moving from > > >> EC to replicated or vice versa). > > >> > > >> When I went to the OpenStack open infrastructure (aka summit) Sage > > >> mentioned about support of live migration of volumes (and as a result > > >> of pools) in Nautilus. Is this still the case and is expected to have > > >> live migration working by then? > > >> > > >> thanks, > > >> ___ > > >> ceph-users mailing list > > >> ceph-users@lists.ceph.com > > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > ___ > > > ceph-users mailing list > > > ceph-users@lists.ceph.com > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > -- > > Jason > > > > -- > Jason > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] pool/volume live migration
Correction: at least for the initial version of live-migration, you need to temporarily stop clients that are using the image, execute "rbd migration prepare", and then restart the clients against the new destination image. The "prepare" step will fail if it detects that the source image is in-use. On Fri, Feb 8, 2019 at 9:00 AM Jason Dillaman wrote: > > Indeed, it is forthcoming in the Nautilus release. > > You would initiate a "rbd migration prepare > " to transparently link the dst-image-spec to the > src-image-spec. Any active Nautilus clients against the image will > then re-open the dst-image-spec for all IO operations. Read requests > that cannot be fulfilled by the new dst-image-spec will be forwarded > to the original src-image-spec (similar to how parent/child cloning > behaves). Write requests to the dst-image-spec will force a deep-copy > of all impacted src-image-spec backing data objects (including > snapshot history) to the associated dst-image-spec backing data > object. At any point a storage admin can run "rbd migration execute" > to deep-copy all src-image-spec data blocks to the dst-image-spec. > Once the migration is complete, you would just run "rbd migration > commit" to remove src-image-spec. > > Note: at some point prior to "rbd migration commit", you will need to > take minimal downtime to switch OpenStack volume registration from the > old image to the new image if you are changing pools. > > On Fri, Feb 8, 2019 at 5:33 AM Caspar Smit wrote: > > > > Hi Luis, > > > > According to slide 21 of Sage's presentation at FOSDEM it is coming in > > Nautilus: > > > > https://fosdem.org/2019/schedule/event/ceph_project_status_update/attachments/slides/3251/export/events/attachments/ceph_project_status_update/slides/3251/ceph_new_in_nautilus.pdf > > > > Kind regards, > > Caspar > > > > Op vr 8 feb. 2019 om 11:24 schreef Luis Periquito : > >> > >> Hi, > >> > >> a recurring topic is live migration and pool type change (moving from > >> EC to replicated or vice versa). > >> > >> When I went to the OpenStack open infrastructure (aka summit) Sage > >> mentioned about support of live migration of volumes (and as a result > >> of pools) in Nautilus. Is this still the case and is expected to have > >> live migration working by then? > >> > >> thanks, > >> ___ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > Jason -- Jason ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] pool/volume live migration
Indeed, it is forthcoming in the Nautilus release. You would initiate a "rbd migration prepare " to transparently link the dst-image-spec to the src-image-spec. Any active Nautilus clients against the image will then re-open the dst-image-spec for all IO operations. Read requests that cannot be fulfilled by the new dst-image-spec will be forwarded to the original src-image-spec (similar to how parent/child cloning behaves). Write requests to the dst-image-spec will force a deep-copy of all impacted src-image-spec backing data objects (including snapshot history) to the associated dst-image-spec backing data object. At any point a storage admin can run "rbd migration execute" to deep-copy all src-image-spec data blocks to the dst-image-spec. Once the migration is complete, you would just run "rbd migration commit" to remove src-image-spec. Note: at some point prior to "rbd migration commit", you will need to take minimal downtime to switch OpenStack volume registration from the old image to the new image if you are changing pools. On Fri, Feb 8, 2019 at 5:33 AM Caspar Smit wrote: > > Hi Luis, > > According to slide 21 of Sage's presentation at FOSDEM it is coming in > Nautilus: > > https://fosdem.org/2019/schedule/event/ceph_project_status_update/attachments/slides/3251/export/events/attachments/ceph_project_status_update/slides/3251/ceph_new_in_nautilus.pdf > > Kind regards, > Caspar > > Op vr 8 feb. 2019 om 11:24 schreef Luis Periquito : >> >> Hi, >> >> a recurring topic is live migration and pool type change (moving from >> EC to replicated or vice versa). >> >> When I went to the OpenStack open infrastructure (aka summit) Sage >> mentioned about support of live migration of volumes (and as a result >> of pools) in Nautilus. Is this still the case and is expected to have >> live migration working by then? >> >> thanks, >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Jason ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] pool/volume live migration
Hi Luis, According to slide 21 of Sage's presentation at FOSDEM it is coming in Nautilus: https://fosdem.org/2019/schedule/event/ceph_project_status_update/attachments/slides/3251/export/events/attachments/ceph_project_status_update/slides/3251/ceph_new_in_nautilus.pdf Kind regards, Caspar Op vr 8 feb. 2019 om 11:24 schreef Luis Periquito : > Hi, > > a recurring topic is live migration and pool type change (moving from > EC to replicated or vice versa). > > When I went to the OpenStack open infrastructure (aka summit) Sage > mentioned about support of live migration of volumes (and as a result > of pools) in Nautilus. Is this still the case and is expected to have > live migration working by then? > > thanks, > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
