** Description changed: *********** SRU TEMPLATE AT THE BOTTOM ************** When Cinder volume service creates a volume from an image, it will download an image from Glance to a cinder-volume unit (to be precise, when CoW is not available by the backend between Glance and Cinder). The image fetch speed is not super fast as of today: 2023-05-17 02:52:28.275 158391 DEBUG cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image fetch details: dest /var/lib/cinder/conversion/image_fetch_741953c3-08f7-43b6-b423-2be938ac8d02_bc1hvgd7cinder@cinder-ceph, size 1907.35 MB, duration 27.80 sec fetch /usr/lib/python3/dist-packages/cinder/image/image_utils.py:637 2023-05-17 02:52:28.275 158391 INFO cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image download 1907.35 MB at 68.61 MB/s -> ~ 549 Mbps By running a profiler, it looks like serious_integrity_iter (glanceclient/common/utils.py) is consuming a lot of CPU cycles on the client side (in this case the cinder-volume unit). Image caching can overcome this kind of challenges but this report is for when we need to download images (new images, updated images, etc.). https://docs.openstack.org/cinder/latest/admin/image-volume-cache.html =============== SRU DESCRIPTION =============== [Impact] This issue causes images to be download much slower than they should, causing VM creation to potentially time out if the image is too big and takes too long. The fix proposed greatly improves performance and prevents timeouts in most cases where it would timeout without the fix. [Test case] + Steps for python-glanceclient: + ------------------------------ + 1. Deploy OpenStack with Cinder and Glance 2. Upload a resonable large image, like jammy ~650mb is enough. 3. Create a volume from the image openstack volume create --size 3 --image jammy v1 4. Check the logs for messages of time to fetch/download the image: ... cinder.image.image_utils ... Image fetch details: dest /var/lib/cinder/conversion/image_fetch_ed429779-0d11-45cb-8ee5-fee7c2003d86_rubea933juju-056b6e-jy3-0@LVM- default, size 645.16 MB, duration 3.59 sec fetch ... ... cinder.image.image_utils ... Image download 645.16 MB at 179.58 MB/s 5. Repeat steps (3) and (4) 3 times just to make sure it is consistent and to have an average (avoid underlying infra fluctuations) 6. Install fix and restart all cinder services (such as apache2, cinder- scheduler and cinder-volume) 7. Repeat steps (3), (4) and (5), and make sure the new duration and download rate is at least about 50% faster. + Steps for python-openstacksdk: + ------------------------------ + + 1. Deploy OpenStack with Glance + + 2. Upload a resonable large image, like jammy ~650mb is enough. + + 3. Install python3-openstackclient in your client machine + + sudo apt install python3-openstackclient + + 4. Download the image (repeat about 5 times to have an average) + + time openstack image save jammy --file /dev/null + + 5. Install fixed package (python3-openstacksdk) + + 6. Download the image again (repeat about 5 times to have an average). + This time there should be a significant decrease in time + [Where problems could occur] If there are issues with the fix, it could impact the ability to download images at all, therefore impacting the ability to create volumes from images, and create VMs. Reverting the package is enough to undo the changes and restore previous functionality. [Other Info] Both python-glanceclient and openstacksdk fixes listed below are necessary. https://review.opendev.org/c/openstack/openstacksdk/+/883461 https://review.opendev.org/c/openstack/python-glanceclient/+/924060
** Attachment added: "lp2020139_focal_yoga_verification.txt" https://bugs.launchpad.net/ubuntu/jammy/+source/python-openstacksdk/+bug/2020139/+attachment/5886141/+files/lp2020139_focal_yoga_verification.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2020139 Title: [SRU] Image fetch speed from Glance to Cinder volume service can be slow and capped by client CPU To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2020139/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
