Re: [openstack-dev] FFE Request: Make RBD Usable for Ephemeral Storage
Mike Perez wrote: Currently in Havana development, RBD as ephemeral storage has serious stability and performance issues that makes the Ceph cluster a bottleneck for using an image as a source. [...] This comes up a bit late, and the current RC bugs curves[1] really do not encourage me to add more distraction for core reviewers. The only way I could be fine with this would be for the performance issue to actually be considered a bug (being so slow you can't really use it without the fix), *and* the review being very advanced and consensual that the distraction is minimal. Could you quantify the performance issue, and address Zhi Yan Liu's comments ? -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] FFE Request: Make RBD Usable for Ephemeral Storage
On Wed, Sep 18, 2013 at 8:14 AM, Thierry Carrez thie...@openstack.orgwrote: Mike Perez wrote: Currently in Havana development, RBD as ephemeral storage has serious stability and performance issues that makes the Ceph cluster a bottleneck for using an image as a source. [...] This comes up a bit late, and the current RC bugs curves[1] really do not encourage me to add more distraction for core reviewers. The only way I could be fine with this would be for the performance issue to actually be considered a bug (being so slow you can't really use it without the fix), *and* the review being very advanced and consensual that the distraction is minimal. Could you quantify the performance issue, and address Zhi Yan Liu's comments ? -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev I have to say that this seems EXTREMELY late to be raising as an issue now. I also have to say that I don't see that this would be that critical as it's something that's never been raised up until this point. If it were a reported issue that we just never got around to addressing that might be different. As Thierry pointed out the bug trajectory is not quite what we want yet anyway, so reworking a feature that works but just doesn't work as efficiently as it could/should doesn't seem like it meets the requirements for an FFE at all. All of these things combined with how late the request is it seems to me like it's pretty difficult to consider this for Havana. Thanks, John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] FFE Request: Make RBD Usable for Ephemeral Storage
On 09/18/2013 10:34 AM, John Griffith wrote: On Wed, Sep 18, 2013 at 8:14 AM, Thierry Carrez thie...@openstack.org mailto:thie...@openstack.org wrote: Mike Perez wrote: Currently in Havana development, RBD as ephemeral storage has serious stability and performance issues that makes the Ceph cluster a bottleneck for using an image as a source. [...] This comes up a bit late, and the current RC bugs curves[1] really do not encourage me to add more distraction for core reviewers. The only way I could be fine with this would be for the performance issue to actually be considered a bug (being so slow you can't really use it without the fix), *and* the review being very advanced and consensual that the distraction is minimal. Could you quantify the performance issue, and address Zhi Yan Liu's comments ? -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev I have to say that this seems EXTREMELY late to be raising as an issue now. I also have to say that I don't see that this would be that critical as it's something that's never been raised up until this point. If it were a reported issue that we just never got around to addressing that might be different. As Thierry pointed out the bug trajectory is not quite what we want yet anyway, so reworking a feature that works but just doesn't work as efficiently as it could/should doesn't seem like it meets the requirements for an FFE at all. All of these things combined with how late the request is it seems to me like it's pretty difficult to consider this for Havana. Agreed with all the points here. We're adding bugs to the RC1 list for Nova faster than fixing them right now, so I don't think we can afford exceptions unless we really have to. I hate it though, because I definitely see how this could be very useful. It's just bad timing. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] FFE Request: Make RBD Usable for Ephemeral Storage
On 09/18/2013 07:14 AM, Thierry Carrez wrote: Mike Perez wrote: Currently in Havana development, RBD as ephemeral storage has serious stability and performance issues that makes the Ceph cluster a bottleneck for using an image as a source. [...] This comes up a bit late, and the current RC bugs curves[1] really do not encourage me to add more distraction for core reviewers. The only way I could be fine with this would be for the performance issue to actually be considered a bug (being so slow you can't really use it without the fix), *and* the review being very advanced and consensual that the distraction is minimal. Could you quantify the performance issue, and address Zhi Yan Liu's comments ? In terms of performance, consider an image that's already cached on the hypervisor host. Using the libvirt_images_type=rbd, the locally cached copy is imported via the 'rbd import' command. This is a very slow operation, which ended up taking upwards of 15 minutes in my tests. Using libvirt_images_type=qcow2, this takes roughly 0.2 seconds. If the image was not already cached locally, it would have to be downloaded from glance, which would take the same amount of time regardless of libvirt_images_type or glance backend. The proposed patch avoids downloading from glance and uploading to rbd, and results in similar times to qcow2 without requiring the image to be cached locally. Josh ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] FFE Request: Make RBD Usable for Ephemeral Storage
On Sep 18, 2013, at 4:50 PM, Josh Durgin josh.dur...@inktank.com wrote: On 09/18/2013 07:14 AM, Thierry Carrez wrote: Mike Perez wrote: Currently in Havana development, RBD as ephemeral storage has serious stability and performance issues that makes the Ceph cluster a bottleneck for using an image as a source. [...] This comes up a bit late, and the current RC bugs curves[1] really do not encourage me to add more distraction for core reviewers. The only way I could be fine with this would be for the performance issue to actually be considered a bug (being so slow you can't really use it without the fix), *and* the review being very advanced and consensual that the distraction is minimal. Could you quantify the performance issue, and address Zhi Yan Liu's comments ? In terms of performance, consider an image that's already cached on the hypervisor host. Using the libvirt_images_type=rbd, the locally cached copy is imported via the 'rbd import' command. This is a very slow operation, which ended up taking upwards of 15 minutes in my tests. Using libvirt_images_type=qcow2, this takes roughly 0.2 seconds. If the image was not already cached locally, it would have to be downloaded from glance, which would take the same amount of time regardless of libvirt_images_type or glance backend. The proposed patch avoids downloading from glance and uploading to rbd, and results in similar times to qcow2 without requiring the image to be cached locally. Josh Its too bad that the timing is so bad because this seems like a really useful feature. It might be worth it merge a patch into icehouse ASAP and see if you can get the distros on board with backporting the patch. Vish ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] FFE Request: Make RBD Usable for Ephemeral Storage
Folks, Currently in Havana development, RBD as ephemeral storage has serious stability and performance issues that makes the Ceph cluster a bottleneck for using an image as a source. Nova has to currently communicate with the external service Glance, which has to talk to the separate Ceph storage backend to fetch path information. The entire image is then downloaded to local disk, and then imported from local disk to RBD. This leaves a stability concern, especially with large images for the instance to be successfully created, due to unnecessary data pulling and pushing for solutions like RBD. Due to the fact we have to do a import from local disk to RBD, this can make performance even slower than a normal backend filesystem since the import is single threaded. This can be eliminated by instead having Nova's RBD image backend utility communicate directly with the Ceph backend to do a copy-on-write of the image. Not only does this greatly improve stability, but performance is drastically improved by not having to do a full copy of the image. A lot of the code to make this happen came from the RBD Cinder driver which has been stable and merged for quite a while. Bug: https://code.launchpad.net/bugs/1226351 Patch: https://review.openstack.org/#/c/46879/1 Thanks, Mike Perez ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] FFE Request: Make RBD Usable for Ephemeral Storage
On Wed, Sep 18, 2013 at 6:16 AM, Mike Perez thin...@gmail.com wrote: Folks, Currently in Havana development, RBD as ephemeral storage has serious stability and performance issues that makes the Ceph cluster a bottleneck for using an image as a source. Nova has to currently communicate with the external service Glance, which has to talk to the separate Ceph storage backend to fetch path information. The entire image is then downloaded to local disk, and then imported from local disk to RBD. This leaves a stability concern, especially with large images for the instance to be successfully created, due to unnecessary data pulling and pushing for solutions like RBD. Due to the fact we have to do a import from local disk to RBD, this can make performance even slower than a normal backend filesystem since the import is single threaded. This can be eliminated by instead having Nova's RBD image backend utility communicate directly with the Ceph backend to do a copy-on-write of the image. Not only does this greatly improve stability, but performance is drastically improved by not having to do a full copy of the image. A lot of the code to make this happen came from the RBD Cinder driver which has been stable and merged for quite a while. Bug: https://code.launchpad.net/bugs/1226351 Patch: https://review.openstack.org/#/c/46879/1 Thanks, Mike Perez ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Hi Mike Perez, folks, I absolutely agree use zero-copy approach to prepare template image is a good idea, such as CoW. But after check your patch I have some concerns on the currently implementation. Actually I had prepared some dedicated BPs [1][2] and a patch [3] to cover such requirements and problems around zero-copy (aka your 'direct_fetch') image preparing, it been implemented as a framework and allow other people realize such plug-ins for a particular image storage backend/location. So I'd very like to invite you (and Josh Durgin) to take a look on them, I believe (and welcome) your stuff within #46879 around RBD image handling can be implemented as a RBDImangeHandler plug-ins under my framework. I consider above implementation is better, since framework code within #33409 can handle most common logic such as plug-ins loading, image handler selecting base on image location, image multiple location supporting and etc.. And each particular image handler can just implement such special methods easily and don't need to rebuild the existed (and tested) part. Of cause, with the production of new handlers we probably need add more interfaces and pass more context data to the structure / ImageHandler base class as needed, we can talk this in irc. [1] https://blueprints.launchpad.net/nova/+spec/image-multiple-location [2] https://blueprints.launchpad.net/nova/+spec/effective-template-base-image-preparing [3] https://review.openstack.org/#/c/33409/ thanks, zhiyan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
