[Bug 1934849] Re: s3 backend takes time exponentially
This bug was fixed in the package python-glance-store - 2.0.0-0ubuntu2~cloud0 --- python-glance-store (2.0.0-0ubuntu2~cloud0) bionic-ussuri; urgency=medium . * New update for the Ubuntu Cloud Archive. . python-glance-store (2.0.0-0ubuntu2) focal; urgency=medium . [ Corey Bryant ] * d/gbp.conf: Create stable/ussuri branch. . [ Chris MacNaughton ] * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev. * d/p/s3-optimize-write-chunksize.patch: Cherry-picked from upstream to fix S3 write performance (LP: #1934849). ** Changed in: cloud-archive/ussuri Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
This bug was fixed in the package python-glance-store - 2.3.0-0ubuntu1~cloud1 --- python-glance-store (2.3.0-0ubuntu1~cloud1) focal-victoria; urgency=medium . [ Chris MacNaughton ] * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev. . [ Corey Bryant ] * d/gbp.conf: Create stable/victoria branch. * d/p/s3-optimize-write-chunksize.patch: Cherry-picked from upstream to fix S3 write performance (LP: #1934849). ** Changed in: cloud-archive/victoria Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
This bug was fixed in the package python-glance-store - 2.5.0-0ubuntu2~cloud0 --- python-glance-store (2.5.0-0ubuntu2~cloud0) focal-wallaby; urgency=medium . * New update for the Ubuntu Cloud Archive. . python-glance-store (2.5.0-0ubuntu2) hirsute; urgency=medium . * d/gbp.conf: Create stable/wallaby branch. * d/p/s3-optimize-write-chunksize.patch: Cherry-picked from upstream to fix S3 write performance (LP: #1934849). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
This bug was fixed in the package python-glance-store - 2.0.0-0ubuntu2 --- python-glance-store (2.0.0-0ubuntu2) focal; urgency=medium [ Corey Bryant ] * d/gbp.conf: Create stable/ussuri branch. [ Chris MacNaughton ] * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev. * d/p/s3-optimize-write-chunksize.patch: Cherry-picked from upstream to fix S3 write performance (LP: #1934849). -- Corey Bryant Mon, 19 Jul 2021 14:30:58 -0400 ** Changed in: python-glance-store (Ubuntu Focal) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
This bug was fixed in the package python-glance-store - 2.5.0-0ubuntu2 --- python-glance-store (2.5.0-0ubuntu2) hirsute; urgency=medium * d/gbp.conf: Create stable/wallaby branch. * d/p/s3-optimize-write-chunksize.patch: Cherry-picked from upstream to fix S3 write performance (LP: #1934849). -- Corey Bryant Mon, 19 Jul 2021 14:07:21 -0400 ** Changed in: python-glance-store (Ubuntu Hirsute) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
[focal-victoria] All of the uploads succeeded. And -proposed shortened time for the larger sizes. $ sudo apt-get install python3-glance-store/focal-proposed $ sudo systemctl restart glance-api $ apt-cache policy python3-glance-store python3-glance-store: Installed: 2.3.0-0ubuntu1~cloud1 Candidate: 2.3.0-0ubuntu1~cloud1 Version table: *** 2.3.0-0ubuntu1~cloud1 500 500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-proposed/victoria/main amd64 Packages 100 /var/lib/dpkg/status 2.3.0-0ubuntu1~cloud0 500 500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/victoria/main amd64 Packages 2.0.0-0ubuntu1 500 500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages sizecephswift s3 s3 -proposed 0MiB0m10.780s 0m2.762s0m3.819s0m3.235s 3MiB0m3.334s0m2.541s0m3.019s0m2.389s 5MiB0m2.731s0m2.244s0m2.687s0m2.417s 9MiB0m2.725s0m2.334s0m2.243s0m2.069s 10MiB 0m2.752s0m2.315s0m3.055s0m2.120s 128MiB 0m4.404s0m3.604s0m56.566s 0m4.728s 512MiB 0m8.411s0m6.021s13m45.246s 0m17.453s 1024MiB 0m14.707s 0m11.430s 54m22.978s 0m55.937s ** Tags removed: verification-needed verification-victoria-needed ** Tags added: verification-done verification-victoria-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
[bionic-ussuri] All of the uploads succeeded. And -proposed shortened time for the larger sizes. $ sudo apt-get install python3-glance-store/bionic-proposed $ sudo systemctl restart glance-api $ apt-cache policy python3-glance-store python3-glance-store: Installed: 2.0.0-0ubuntu2~cloud0 Candidate: 2.0.0-0ubuntu2~cloud0 Version table: *** 2.0.0-0ubuntu2~cloud0 500 500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/ussuri/main amd64 Packages 100 /var/lib/dpkg/status 2.0.0-0ubuntu1~cloud0 500 500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/ussuri/main amd64 Packages 0.23.0-0ubuntu1 500 500 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages sizecephswift s3 s3 -proposed 0MiB0m2.985s0m2.832s0m2.487s0m2.293s 3MiB0m2.761s0m2.175s0m1.970s0m2.367s 5MiB0m2.729s0m2.224s0m2.031s0m1.981s 9MiB0m2.783s0m2.265s0m2.168s0m2.087s 10MiB 0m2.824s0m2.256s0m2.122s0m2.037s 128MiB 0m4.703s0m2.966s0m47.824s 0m3.869s 512MiB 0m8.979s0m5.110s12m27.668s 0m16.269s 1024MiB 0m15.121s 0m15.742s 50m56.406s 0m48.282s ** Tags added: verification-ussuri-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
Just for the record, this is the current status with focal-victoria. No diff between -updates and -proposed. $ apt-cache policy python3-glance-store python3-glance-store: Installed: 2.3.0-0ubuntu1~cloud0 Candidate: 2.3.0-0ubuntu1~cloud0 Version table: *** 2.3.0-0ubuntu1~cloud0 500 500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/victoria/main amd64 Packages 500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-proposed/victoria/main amd64 Packages 100 /var/lib/dpkg/status 2.0.0-0ubuntu1 500 500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
@Corey, Somehow the binary package for cloud-archive:victoria-proposed is not published yet. Can you please double-check the build status of the package? I just don't know where to look. cloud1 in the source vs cloud0 in the binary. $ curl -s http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-proposed/victoria/main/source/Sources.gz | zcat | grep -A2 'Package: python-glance-store' Package: python-glance-store Binary: python-glance-store-doc, python3-glance-store Version: 2.3.0-0ubuntu1~cloud1 $ curl -s http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/focal-proposed/victoria/main/binary-amd64/Packages | grep -A7 'Package: python3-glance-store' Package: python3-glance-store Source: python-glance-store Priority: optional Section: python Installed-Size: 983 Maintainer: Ubuntu Developers Architecture: all Version: 2.3.0-0ubuntu1~cloud0 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
[focal-wallaby] All of the uploads succeeded. And -proposed shortened time for the larger sizes. $ sudo apt-get install python3-glance-store/focal-proposed $ sudo systemctl restart glance-api $ apt-cache policy python3-glance-store python3-glance-store: Installed: 2.5.0-0ubuntu2~cloud0 Candidate: 2.5.0-0ubuntu2~cloud0 Version table: *** 2.5.0-0ubuntu2~cloud0 500 500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-proposed/wallaby/main amd64 Packages 100 /var/lib/dpkg/status 2.5.0-0ubuntu1~cloud0 500 500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/wallaby/main amd64 Packages 2.0.0-0ubuntu1 500 500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages sizecephswift s3 s3 -proposed 0MiB0m3.389s0m2.876s0m2.389s0m4.459s 3MiB0m2.788s0m2.278s0m2.076s0m3.138s 5MiB0m2.716s0m2.337s0m2.122s0m3.246s 9MiB0m3.506s0m2.396s0m2.274s0m2.782s 10MiB 0m2.859s0m2.366s0m2.324s0m2.829s 128MiB 0m4.514s0m3.306s0m55.145s 0m4.868s 512MiB 0m10.862s 0m9.692s13m31.848s 0m21.378s 1024MiB 0m21.965s 0m27.575s 54m27.784s 1m4.299s ** Tags removed: verification-wallaby-needed ** Tags added: verification-wallaby-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
[focal] All of the uploads succeeded. And -proposed shortened time for the larger sizes. $ sudo apt-get install python3-glance-store/focal-proposed $ sudo systemctl restart glance-api $ apt-cache policy python3-glance-store python3-glance-store: Installed: 2.0.0-0ubuntu2 Candidate: 2.0.0-0ubuntu2 Version table: *** 2.0.0-0ubuntu2 400 400 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages 100 /var/lib/dpkg/status 2.0.0-0ubuntu1 500 500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages sizecephswift s3 s3 -proposed 0MiB0m3.571s0m2.249s0m2.185s0m1.959s 3MiB0m2.724s0m2.285s0m1.986s0m2.118s 5MiB0m2.717s0m2.694s0m2.093s0m2.213s 9MiB0m2.749s0m2.342s0m2.357s0m2.107s 10MiB 0m3.415s0m2.342s0m2.289s0m2.139s 128MiB 0m4.644s0m3.109s0m54.343s 0m4.287s 512MiB 0m11.121s 0m6.246s13m35.948s 0m16.853s 1024MiB 0m16.292s 0m11.441s 54m24.339s 0m52.231s ** Tags removed: verification-needed-focal ** Tags added: verification-done-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
[hirsute] All of the uploads succeeded. And -proposed shortened time for the larger sizes. $ sudo apt-get install python3-glance-store/hirsute-proposed $ sudo systemctl restart glance-api $ apt-cache policy python3-glance-store python3-glance-store: Installed: 2.5.0-0ubuntu2 Candidate: 2.5.0-0ubuntu2 Version table: *** 2.5.0-0ubuntu2 400 400 http://archive.ubuntu.com/ubuntu hirsute-proposed/main amd64 Packages 100 /var/lib/dpkg/status 2.5.0-0ubuntu1 500 500 http://archive.ubuntu.com/ubuntu hirsute/main amd64 Packages sizecephswift s3 s3 -proposed 0MiB0m2.495s0m2.394s0m2.262s0m2.352s 3MiB0m2.721s0m2.270s0m2.144s0m2.430s 5MiB0m2.726s0m2.314s0m2.200s0m2.177s 9MiB0m2.737s0m2.377s0m2.664s0m2.238s 10MiB 0m2.772s0m2.350s0m2.756s0m2.160s 128MiB 0m4.261s0m3.381s0m57.001s 0m5.542s 512MiB 0m11.619s 0m11.734s 14m13.448s 0m24.012s 1024MiB 0m23.787s 0m20.756s 56m37.906s 1m3.716s ** Tags removed: verification-needed-hirsute ** Tags added: verification-done-hirsute -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
Hello Nobuto, or anyone else affected, Accepted python-glance-store into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-glance-store/2.5.0-0ubuntu2 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: python-glance-store (Ubuntu Hirsute) Status: Triaged => Fix Committed ** Tags added: verification-needed verification-needed-hirsute ** Changed in: python-glance-store (Ubuntu Focal) Status: Triaged => Fix Committed ** Tags added: verification-needed-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
My update in the bug description was somehow rolled back (by me in the
record), trying again.
** Description changed:
[Impact]
- [Test Case]
- I have a test Ceph cluster as an object storage with both Swift and S3
protocols enabled for Glance (Ussuri). When I use Swift backend with Glance, an
image upload completes quickly enough. But with S3 backend Glance, it takes
much more time to upload an image and it seems to rise exponentially.
+
+ Glance with S3 backend cannot accept image uploads in a realistic time
+ frame. For example, an 1GB image upload takes ~60 minutes although other
+ backends such as swift can complete it with 10 seconds.
+
+ [Test Plan]
+
+ 1. Deploy a partial OpenStack with multiple Glance backends including S3
+ (zaza test bundles can be used with "ceph" which will set up "rbd",
"swift", and "s3" backends -
https://opendev.org/openstack/charm-glance/src/branch/master/tests/tests.yaml)
+ 2. Upload multiple images with variety of sizes
+ 3. confirm the duration of uploading images are shorter in general after
applying the updated package
+ (expected duration of 1GB is from ~60 minutes to 1-3 minutes)
+
+ for backend in ceph swift s3; do
+ echo "[$backend]"
+ for i in {0,3,5,9,10,128,512,1024}; do
+ dd if=/dev/zero of=my-image.img bs=1M count=$i oflag=sync
+ echo "${i}MiB"
+ time glance image-create \
+ --store $backend \
+ --file my-image.img --name "my-image-${backend}-${i}MiB" \
+ --disk-format raw --container-format bare \
+ --progress
+ done
+ done
+
+ [Where problems could occur]
+
+ Since we bump WRITE_CHUNKSIZE from 64KiB to 5MiB, there might be a case where
image uploads fail if the size of the image is less than WRITE_CHUNKSIZE. Or
there might be an unexpected latency in the worst case scenario. We will try to
address the concerns by testing multiple images uploads with multiple sizes
including some corner cases as follows:
+ - 0 - zero
+ - 3MiB - less than the new WRITE_CHUNKSIZE(5MiB)
+ - 5MiB - exactly same as the new WRITE_CHUNKSIZE(5MiB)
+ - 9MiB - bigger than new WRITE_CHUNKSIZE(5MiB) but less than twice
+ - 10MiB - exactly twice as the new WRITE_CHUNKSIZE(5MiB)
+ - 128MiB, 512MiB, 1024MiB - some large images
+
+
+
+ I have a test Ceph cluster as an object storage with both Swift and S3
+ protocols enabled for Glance (Ussuri). When I use Swift backend with
+ Glance, an image upload completes quickly enough. But with S3 backend
+ Glance, it takes much more time to upload an image and it seems to rise
+ exponentially.
It's worth noting that when uploading an image with S3 backend, a single
core is consumed 100% by glance-api process.
-
- for backend in swift s3; do
- for i in {8,16,32,64,128,512}; do
- dd if=/dev/zero of=my-image.img bs=1M count=$i oflag=sync
- /usr/bin/time --format=%E glance image-create \
- --store $backend \
- --file my-image.img --name my-image \
- --disk-format raw --container-format bare \
- --progress
- done
- done
[swift]
8MB - 2.4s
16MB - 2.8s
32MB - 2.6s
64MB - 2.7s
128MB - 3.1s
...
512MB - 5.9s
[s3]
8MB - 2.2s
16MB - 2.9s
32MB - 5.5s
64MB - 16.3s
128MB - 54.9s
...
512MB - 14m26s
Btw, downloading of 512MB image with S3 backend can complete with less
than 10 seconds.
$ time openstack image save --file downloaded.img
917c5424-4350-4bc5-98ca-66d40e101843
real0m5.673s
$ du -h downloaded.img
512Mdownloaded.img
[/etc/glance/glance-api.conf]
enabled_backends = local:file, ceph:rbd, swift:swift, s3:s3
[swift]
auth_version = 3
auth_address = http://192.168.151.131:5000/v3
...
container = glance
large_object_size = 5120
large_object_chunk_size = 200
[s3]
s3_store_host = http://192.168.151.137:80/
...
s3_store_bucket = zaza-glance-s3-test
s3_store_large_object_size = 5120
s3_store_large_object_chunk_size = 200
+ ProblemType: Bug
+ DistroRelease: Ubuntu 20.04
ProblemType: BugDistroRelease: Ubuntu 20.04
Package: python3-glance-store 2.0.0-0ubuntu1
ProcVersionSignature: Ubuntu 5.4.0-77.86-generic 5.4.119
Uname: Linux 5.4.0-77-generic x86_64
- NonfreeKernelModules: bluetooth ecdh_generic ecc tcp_diag inet_diag
binfmt_misc veth zfs zunicode zlua zavl icp zcommon znvpair spl unix_diag
nft_masq nft_chain_nat bridge stp llc vhost_vsock
vmw_vsock_virtio_transport_common vhost vsock ebtable_filter ebtables
ip6table_raw ip6table_mangle ip6table_nat ip6table_filter ip6_tables
iptable_raw iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6
nf_defrag_ipv4 iptable_filter bpfilter nf_tables nfnetlink dm_multipath
scsi_dh_rdac scsi_dh_emc scsi_dh_alua kvm_amd ccp input_leds kvm joydev mac_hid
serio_raw qemu_fw_cfg sch_fq_codel ip_tables x_tables autofs4 btrfs
zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq
[Bug 1934849] Re: s3 backend takes time exponentially
** Description changed:
[Impact]
-
- Glance with S3 backend cannot accept image uploads in a realistic time
- frame. For example, an 1GB image upload takes ~60 minutes although other
- backends such as swift can complete it with 10 seconds.
-
- [Test Plan]
-
- 1. Deploy a partial OpenStack with multiple Glance backends including S3
- (zaza test bundles can be used with "ceph" which will set up "rbd",
"swift", and "s3" backends -
https://opendev.org/openstack/charm-glance/src/branch/master/tests/tests.yaml)
- 2. Upload multiple images with variety of sizes
- 3. confirm the duration of uploading images are shorter in general after
applying the updated package
- (expected duration of 1GB is from ~60 minutes to 1-3 minutes)
-
- for backend in ceph swift s3; do
- echo "[$backend]"
- for i in {0,3,5,9,10,128,512,1024}; do
- dd if=/dev/zero of=my-image.img bs=1M count=$i oflag=sync
- echo "${i}MiB"
- time glance image-create \
- --store $backend \
- --file my-image.img --name "my-image-${backend}-${i}MiB" \
- --disk-format raw --container-format bare \
- --progress
- done
- done
-
-
- [Where problems could occur]
-
- Since we bump WRITE_CHUNKSIZE from 64KiB to 5MiB, there might be a case where
image uploads fail if the size of the image is less than WRITE_CHUNKSIZE. Or
there might be an unexpected latency in the worst case scenario. We will try to
address the concerns by testing multiple images uploads with multiple sizes
including some corner cases as follows:
- - 0 - zero
- - 3MiB - less than the new WRITE_CHUNKSIZE(5MiB)
- - 5MiB - exactly same as the new WRITE_CHUNKSIZE(5MiB)
- - 9MiB - bigger than new WRITE_CHUNKSIZE(5MiB) but less than twice
- - 10MiB - exactly twice as the new WRITE_CHUNKSIZE(5MiB)
- - 128MiB, 512MiB, 1024MiB - some large images
-
-
-
-
- I have a test Ceph cluster as an object storage with both Swift and S3
- protocols enabled for Glance (Ussuri). When I use Swift backend with
- Glance, an image upload completes quickly enough. But with S3 backend
- Glance, it takes much more time to upload an image and it seems to rise
- exponentially.
+ [Test Case]
+ I have a test Ceph cluster as an object storage with both Swift and S3
protocols enabled for Glance (Ussuri). When I use Swift backend with Glance, an
image upload completes quickly enough. But with S3 backend Glance, it takes
much more time to upload an image and it seems to rise exponentially.
It's worth noting that when uploading an image with S3 backend, a single
core is consumed 100% by glance-api process.
+
+ for backend in swift s3; do
+ for i in {8,16,32,64,128,512}; do
+ dd if=/dev/zero of=my-image.img bs=1M count=$i oflag=sync
+ /usr/bin/time --format=%E glance image-create \
+ --store $backend \
+ --file my-image.img --name my-image \
+ --disk-format raw --container-format bare \
+ --progress
+ done
+ done
[swift]
8MB - 2.4s
16MB - 2.8s
32MB - 2.6s
64MB - 2.7s
128MB - 3.1s
...
512MB - 5.9s
[s3]
8MB - 2.2s
16MB - 2.9s
32MB - 5.5s
64MB - 16.3s
128MB - 54.9s
...
512MB - 14m26s
Btw, downloading of 512MB image with S3 backend can complete with less
than 10 seconds.
$ time openstack image save --file downloaded.img
917c5424-4350-4bc5-98ca-66d40e101843
real0m5.673s
$ du -h downloaded.img
512Mdownloaded.img
[/etc/glance/glance-api.conf]
enabled_backends = local:file, ceph:rbd, swift:swift, s3:s3
[swift]
auth_version = 3
auth_address = http://192.168.151.131:5000/v3
...
container = glance
large_object_size = 5120
large_object_chunk_size = 200
[s3]
s3_store_host = http://192.168.151.137:80/
...
s3_store_bucket = zaza-glance-s3-test
s3_store_large_object_size = 5120
s3_store_large_object_chunk_size = 200
ProblemType: BugDistroRelease: Ubuntu 20.04
Package: python3-glance-store 2.0.0-0ubuntu1
ProcVersionSignature: Ubuntu 5.4.0-77.86-generic 5.4.119
Uname: Linux 5.4.0-77-generic x86_64
+ NonfreeKernelModules: bluetooth ecdh_generic ecc tcp_diag inet_diag
binfmt_misc veth zfs zunicode zlua zavl icp zcommon znvpair spl unix_diag
nft_masq nft_chain_nat bridge stp llc vhost_vsock
vmw_vsock_virtio_transport_common vhost vsock ebtable_filter ebtables
ip6table_raw ip6table_mangle ip6table_nat ip6table_filter ip6_tables
iptable_raw iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6
nf_defrag_ipv4 iptable_filter bpfilter nf_tables nfnetlink dm_multipath
scsi_dh_rdac scsi_dh_emc scsi_dh_alua kvm_amd ccp input_leds kvm joydev mac_hid
serio_raw qemu_fw_cfg sch_fq_codel ip_tables x_tables autofs4 btrfs
zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul
crc32_pclmul cirrus ghash_clmulni_intel
[Bug 1934849] Re: s3 backend takes time exponentially
** Description changed:
[Impact]
- [Test Case]
- I have a test Ceph cluster as an object storage with both Swift and S3
protocols enabled for Glance (Ussuri). When I use Swift backend with Glance, an
image upload completes quickly enough. But with S3 backend Glance, it takes
much more time to upload an image and it seems to rise exponentially.
+
+ Glance with S3 backend cannot accept image uploads in a realistic time
+ frame. For example, an 1GB image upload takes ~60 minutes although other
+ backends such as swift can complete it with 10 seconds.
+
+ [Test Plan]
+
+ 1. Deploy a partial OpenStack with multiple Glance backends including S3
+ (zaza test bundles can be used with "ceph" which will set up "rbd",
"swift", and "s3" backends -
https://opendev.org/openstack/charm-glance/src/branch/master/tests/tests.yaml)
+ 2. Upload multiple images with variety of sizes
+ 3. confirm the duration of uploading images are shorter in general after
applying the updated package
+ (expected duration of 1GB is from ~60 minutes to 1-3 minutes)
+
+ for backend in ceph swift s3; do
+ echo "[$backend]"
+ for i in {0,3,5,9,10,128,512,1024}; do
+ dd if=/dev/zero of=my-image.img bs=1M count=$i oflag=sync
+ echo "${i}MiB"
+ time glance image-create \
+ --store $backend \
+ --file my-image.img --name "my-image-${backend}-${i}MiB" \
+ --disk-format raw --container-format bare \
+ --progress
+ done
+ done
+
+
+ [Where problems could occur]
+
+ Since we bump WRITE_CHUNKSIZE from 64KiB to 5MiB, there might be a case where
image uploads fail if the size of the image is less than WRITE_CHUNKSIZE. Or
there might be an unexpected latency in the worst case scenario. We will try to
address the concerns by testing multiple images uploads with multiple sizes
including some corner cases as follows:
+ - 0 - zero
+ - 3MiB - less than the new WRITE_CHUNKSIZE(5MiB)
+ - 5MiB - exactly same as the new WRITE_CHUNKSIZE(5MiB)
+ - 9MiB - bigger than new WRITE_CHUNKSIZE(5MiB) but less than twice
+ - 10MiB - exactly twice as the new WRITE_CHUNKSIZE(5MiB)
+ - 128MiB, 512MiB, 1024MiB - some large images
+
+
+
+
+ I have a test Ceph cluster as an object storage with both Swift and S3
+ protocols enabled for Glance (Ussuri). When I use Swift backend with
+ Glance, an image upload completes quickly enough. But with S3 backend
+ Glance, it takes much more time to upload an image and it seems to rise
+ exponentially.
It's worth noting that when uploading an image with S3 backend, a single
core is consumed 100% by glance-api process.
-
- for backend in swift s3; do
- for i in {8,16,32,64,128,512}; do
- dd if=/dev/zero of=my-image.img bs=1M count=$i oflag=sync
- time glance image-create \
- --store $backend \
- --file my-image.img --name my-image \
- --disk-format raw --container-format bare \
- --progress
- done
- done
[swift]
8MB - 2.4s
16MB - 2.8s
32MB - 2.6s
64MB - 2.7s
128MB - 3.1s
...
512MB - 5.9s
[s3]
8MB - 2.2s
16MB - 2.9s
32MB - 5.5s
64MB - 16.3s
128MB - 54.9s
...
512MB - 14m26s
Btw, downloading of 512MB image with S3 backend can complete with less
than 10 seconds.
$ time openstack image save --file downloaded.img
917c5424-4350-4bc5-98ca-66d40e101843
real0m5.673s
$ du -h downloaded.img
512Mdownloaded.img
[/etc/glance/glance-api.conf]
enabled_backends = local:file, ceph:rbd, swift:swift, s3:s3
[swift]
auth_version = 3
auth_address = http://192.168.151.131:5000/v3
...
container = glance
large_object_size = 5120
large_object_chunk_size = 200
[s3]
s3_store_host = http://192.168.151.137:80/
...
s3_store_bucket = zaza-glance-s3-test
s3_store_large_object_size = 5120
s3_store_large_object_chunk_size = 200
- ProblemType: Bug
- DistroRelease: Ubuntu 20.04
+ ProblemType: BugDistroRelease: Ubuntu 20.04
Package: python3-glance-store 2.0.0-0ubuntu1
ProcVersionSignature: Ubuntu 5.4.0-77.86-generic 5.4.119
Uname: Linux 5.4.0-77-generic x86_64
- NonfreeKernelModules: bluetooth ecdh_generic ecc tcp_diag inet_diag
binfmt_misc veth zfs zunicode zlua zavl icp zcommon znvpair spl unix_diag
nft_masq nft_chain_nat bridge stp llc vhost_vsock
vmw_vsock_virtio_transport_common vhost vsock ebtable_filter ebtables
ip6table_raw ip6table_mangle ip6table_nat ip6table_filter ip6_tables
iptable_raw iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6
nf_defrag_ipv4 iptable_filter bpfilter nf_tables nfnetlink dm_multipath
scsi_dh_rdac scsi_dh_emc scsi_dh_alua kvm_amd ccp input_leds kvm joydev mac_hid
serio_raw qemu_fw_cfg sch_fq_codel ip_tables x_tables autofs4 btrfs
zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul
crc32_pclmul cirrus
[Bug 1934849] Re: s3 backend takes time exponentially
Uploaded to hirsute unapproved queue, focal unapproved queue, and victoria-staging PPA: https://launchpad.net/ubuntu/hirsute/+queue?queue_state=1_text=python-glance-store https://launchpad.net/ubuntu/focal/+queue?queue_state=1_text=python-glance-store https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/victoria-staging/+packages?field.name_filter=glance-store_filter=published_filter= ** Also affects: cloud-archive Importance: Undecided Status: New ** Also affects: cloud-archive/wallaby Importance: Undecided Status: New ** Also affects: cloud-archive/victoria Importance: Undecided Status: New ** Also affects: cloud-archive/ussuri Importance: Undecided Status: New ** Changed in: cloud-archive Status: New => Fix Released ** Changed in: cloud-archive/ussuri Status: New => Triaged ** Changed in: cloud-archive/victoria Status: New => Triaged ** Changed in: cloud-archive/wallaby Status: New => Triaged ** Changed in: cloud-archive/wallaby Importance: Undecided => High ** Changed in: cloud-archive/victoria Importance: Undecided => High ** Changed in: cloud-archive/ussuri Importance: Undecided => High ** Also affects: python-glance-store (Ubuntu Hirsute) Importance: Undecided Status: New ** Also affects: python-glance-store (Ubuntu Focal) Importance: Undecided Status: New ** Changed in: python-glance-store (Ubuntu) Status: Confirmed => Fix Released ** Changed in: python-glance-store (Ubuntu Focal) Status: New => Triaged ** Changed in: python-glance-store (Ubuntu Hirsute) Status: New => Triaged ** Changed in: python-glance-store (Ubuntu Hirsute) Importance: Undecided => High ** Changed in: python-glance-store (Ubuntu Focal) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
** Description changed:
- I have a test Ceph cluster as an object storage with both Swift and S3
- protocols enabled for Glance (Ussuri). When I use Swift backend with
- Glance, an image upload completes quickly enough. But with S3 backend
- Glance, it takes much more time to upload an image and it seems to rise
- exponentially.
+ [Impact]
+ [Test Case]
+ I have a test Ceph cluster as an object storage with both Swift and S3
protocols enabled for Glance (Ussuri). When I use Swift backend with Glance, an
image upload completes quickly enough. But with S3 backend Glance, it takes
much more time to upload an image and it seems to rise exponentially.
It's worth noting that when uploading an image with S3 backend, a single
core is consumed 100% by glance-api process.
for backend in swift s3; do
- for i in {8,16,32,64,128,512}; do
- dd if=/dev/zero of=my-image.img bs=1M count=$i oflag=sync
- time glance image-create \
- --store $backend \
- --file my-image.img --name my-image \
- --disk-format raw --container-format bare \
- --progress
- done
+ for i in {8,16,32,64,128,512}; do
+ dd if=/dev/zero of=my-image.img bs=1M count=$i oflag=sync
+ time glance image-create \
+ --store $backend \
+ --file my-image.img --name my-image \
+ --disk-format raw --container-format bare \
+ --progress
+ done
done
[swift]
8MB - 2.4s
16MB - 2.8s
32MB - 2.6s
64MB - 2.7s
128MB - 3.1s
...
512MB - 5.9s
[s3]
8MB - 2.2s
16MB - 2.9s
32MB - 5.5s
64MB - 16.3s
128MB - 54.9s
...
512MB - 14m26s
Btw, downloading of 512MB image with S3 backend can complete with less
than 10 seconds.
$ time openstack image save --file downloaded.img
917c5424-4350-4bc5-98ca-66d40e101843
real0m5.673s
- $ du -h downloaded.img
+ $ du -h downloaded.img
512Mdownloaded.img
-
[/etc/glance/glance-api.conf]
enabled_backends = local:file, ceph:rbd, swift:swift, s3:s3
[swift]
auth_version = 3
auth_address = http://192.168.151.131:5000/v3
...
container = glance
large_object_size = 5120
large_object_chunk_size = 200
-
[s3]
s3_store_host = http://192.168.151.137:80/
...
s3_store_bucket = zaza-glance-s3-test
s3_store_large_object_size = 5120
s3_store_large_object_chunk_size = 200
ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: python3-glance-store 2.0.0-0ubuntu1
ProcVersionSignature: Ubuntu 5.4.0-77.86-generic 5.4.119
Uname: Linux 5.4.0-77-generic x86_64
NonfreeKernelModules: bluetooth ecdh_generic ecc tcp_diag inet_diag
binfmt_misc veth zfs zunicode zlua zavl icp zcommon znvpair spl unix_diag
nft_masq nft_chain_nat bridge stp llc vhost_vsock
vmw_vsock_virtio_transport_common vhost vsock ebtable_filter ebtables
ip6table_raw ip6table_mangle ip6table_nat ip6table_filter ip6_tables
iptable_raw iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6
nf_defrag_ipv4 iptable_filter bpfilter nf_tables nfnetlink dm_multipath
scsi_dh_rdac scsi_dh_emc scsi_dh_alua kvm_amd ccp input_leds kvm joydev mac_hid
serio_raw qemu_fw_cfg sch_fq_codel ip_tables x_tables autofs4 btrfs
zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul
crc32_pclmul cirrus ghash_clmulni_intel drm_kms_helper virtio_net syscopyarea
aesni_intel sysfillrect sysimgblt fb_sys_fops crypto_simd cryptd drm virtio_blk
glue_helper net_failover psmouse failover floppy i2c_piix4 pata_acpi
ApportVersion: 2.20.11-0ubuntu27.18
Architecture: amd64
CasperMD5CheckResult: skip
Date: Wed Jul 7 04:46:05 2021
PackageArchitecture: all
ProcEnviron:
- TERM=screen-256color
- PATH=(custom, no user)
- LANG=C.UTF-8
- SHELL=/bin/bash
+ TERM=screen-256color
+ PATH=(custom, no user)
+ LANG=C.UTF-8
+ SHELL=/bin/bash
SourcePackage: python-glance-store
UpgradeStatus: No upgrade log present (probably fresh install)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1934849
Title:
s3 backend takes time exponentially
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance-store/+bug/1934849/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
Subscribing Canonical's ~field-high to initiate the Ubuntu package's SRU process in a timely manner. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/glance-store/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
Reviewed: https://review.opendev.org/c/openstack/glance_store/+/799870 Committed: https://opendev.org/openstack/glance_store/commit/32f9a1509bc94baa6acaf508c922f3b7edf5b65f Submitter: "Zuul (22348)" Branch:master commit 32f9a1509bc94baa6acaf508c922f3b7edf5b65f Author: Nobuto Murata Date: Thu Jul 8 01:36:38 2021 +0900 s3: Optimize WRITE_CHUNKSIZE to minimize an overhead When processing the image data from a client, the previous and small chunk size had a significant overhead. Let's be aligned with other chunk size related parameters such as DEFAULT_LARGE_OBJECT_MIN_CHUNK_SIZE (=5MiB) and DEFAULT_LARGE_OBJECT_CHUNK_SIZE (=10MiB). The performance difference should be tangible especially with a singlepart upload. And the upload time can be shortened for example as follows: 100 MiB: 36.5s -> 4.0s 200 MiB: 2m16.8s -> 6.1s 300 MiB: 4m50.4s -> 9.1s Closes-Bug: 1934849 Change-Id: Icecac80dd9e4e9f7ffa76bb7ca63d8d112036b70 ** Changed in: glance-store Status: In Progress => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/glance-store/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
I'm facing the same issue and can confirm the suggested fix improves the upload speed dramatically (without it, I had ~3.5min for 200mb image, now the same time is consumed for 10GB image upload). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/glance-store/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
Fix proposed to branch: master Review: https://review.opendev.org/c/openstack/glance_store/+/799870 ** Changed in: glance-store Status: New => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/glance-store/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
> I *think* hash calculation and verifier have to be outside of the loop to avoid the overhead. I will confirm it with a manual testing. This hypothesis wasn't true, it was really about the chunk size. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/glance-store/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
I *think* hash calculation and verifier have to be outside of the loop to avoid the overhead. I will confirm it with a manual testing. for chunk in utils.chunkreadable(image_file, self.WRITE_CHUNKSIZE): image_data += chunk image_size += len(chunk) os_hash_value.update(chunk) checksum.update(chunk) if verifier: verifier.update(chunk) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/glance-store/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: python-glance-store (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/glance-store/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
Yeah, I put the same config on purpose for both s3 and swift. But tweaking large_object_size didn't make any difference. [swift] large_object_size = 5120 large_object_chunk_size = 200 [s3] s3_store_large_object_size = 5120 s3_store_large_object_chunk_size = 200 After digging into the actual environment, it seems image_size is always zero so this condition never goes to multipart upload, but singlepart upload always. https://opendev.org/openstack/glance_store/src/branch/stable/ussuri/glance_store/_drivers/s3.py#L597 I guess Glance will never know the actual image_size at the beginning of getting image data from a client until getting all of the data. I just don't know what kind of cases we will get non-null value in "size". https://docs.openstack.org/api-ref/image/v2/index.html?expanded=create-image-detail#create-image -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/glance-store/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1934849] Re: s3 backend takes time exponentially
Actually that's only used for images over 5GB
On Wed, Jul 7, 2021 at 1:59 PM James Page
wrote:
> I think the swift driver uses 200M:
>
> DEFAULT_LARGE_OBJECT_CHUNK_SIZE = 200 # 200M
>
> On Wed, Jul 7, 2021 at 1:50 PM Nobuto Murata <1934...@bugs.launchpad.net>
> wrote:
>
>> And by using "4 * units.Mi" it can be 20s.
>>
>> --
>> You received this bug notification because you are a member of Ubuntu
>> OpenStack, which is subscribed to python-glance-store in Ubuntu.
>> https://bugs.launchpad.net/bugs/1934849
>>
>> Title:
>> s3 backend takes time exponentially
>>
>> Status in glance_store:
>> New
>> Status in python-glance-store package in Ubuntu:
>> New
>>
>> Bug description:
>> I have a test Ceph cluster as an object storage with both Swift and S3
>> protocols enabled for Glance (Ussuri). When I use Swift backend with
>> Glance, an image upload completes quickly enough. But with S3 backend
>> Glance, it takes much more time to upload an image and it seems to
>> rise exponentially.
>>
>> It's worth noting that when uploading an image with S3 backend, a
>> single core is consumed 100% by glance-api process.
>>
>> for backend in swift s3; do
>> for i in {8,16,32,64,128,512}; do
>> dd if=/dev/zero of=my-image.img bs=1M count=$i oflag=sync
>> time glance image-create \
>> --store $backend \
>> --file my-image.img --name my-image \
>> --disk-format raw --container-format bare \
>> --progress
>> done
>> done
>>
>> [swift]
>> 8MB - 2.4s
>> 16MB - 2.8s
>> 32MB - 2.6s
>> 64MB - 2.7s
>> 128MB - 3.1s
>> ...
>> 512MB - 5.9s
>>
>> [s3]
>> 8MB - 2.2s
>> 16MB - 2.9s
>> 32MB - 5.5s
>> 64MB - 16.3s
>> 128MB - 54.9s
>> ...
>> 512MB - 14m26s
>>
>> Btw, downloading of 512MB image with S3 backend can complete with less
>> than 10 seconds.
>>
>> $ time openstack image save --file downloaded.img
>> 917c5424-4350-4bc5-98ca-66d40e101843
>> real0m5.673s
>>
>> $ du -h downloaded.img
>> 512Mdownloaded.img
>>
>>
>> [/etc/glance/glance-api.conf]
>>
>> enabled_backends = local:file, ceph:rbd, swift:swift, s3:s3
>>
>> [swift]
>> auth_version = 3
>> auth_address = http://192.168.151.131:5000/v3
>> ...
>> container = glance
>> large_object_size = 5120
>> large_object_chunk_size = 200
>>
>>
>> [s3]
>> s3_store_host = http://192.168.151.137:80/
>> ...
>> s3_store_bucket = zaza-glance-s3-test
>> s3_store_large_object_size = 5120
>> s3_store_large_object_chunk_size = 200
>>
>> ProblemType: Bug
>> DistroRelease: Ubuntu 20.04
>> Package: python3-glance-store 2.0.0-0ubuntu1
>> ProcVersionSignature: Ubuntu 5.4.0-77.86-generic 5.4.119
>> Uname: Linux 5.4.0-77-generic x86_64
>> NonfreeKernelModules: bluetooth ecdh_generic ecc tcp_diag inet_diag
>> binfmt_misc veth zfs zunicode zlua zavl icp zcommon znvpair spl unix_diag
>> nft_masq nft_chain_nat bridge stp llc vhost_vsock
>> vmw_vsock_virtio_transport_common vhost vsock ebtable_filter ebtables
>> ip6table_raw ip6table_mangle ip6table_nat ip6table_filter ip6_tables
>> iptable_raw iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6
>> nf_defrag_ipv4 iptable_filter bpfilter nf_tables nfnetlink dm_multipath
>> scsi_dh_rdac scsi_dh_emc scsi_dh_alua kvm_amd ccp input_leds kvm joydev
>> mac_hid serio_raw qemu_fw_cfg sch_fq_codel ip_tables x_tables autofs4 btrfs
>> zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq
>> async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear
>> crct10dif_pclmul crc32_pclmul cirrus ghash_clmulni_intel drm_kms_helper
>> virtio_net syscopyarea aesni_intel sysfillrect sysimgblt fb_sys_fops
>> crypto_simd cryptd drm virtio_blk glue_helper net_failover psmouse failover
>> floppy i2c_piix4 pata_acpi
>> ApportVersion: 2.20.11-0ubuntu27.18
>> Architecture: amd64
>> CasperMD5CheckResult: skip
>> Date: Wed Jul 7 04:46:05 2021
>> PackageArchitecture: all
>> ProcEnviron:
>>TERM=screen-256color
>>PATH=(custom, no user)
>>LANG=C.UTF-8
>>SHELL=/bin/bash
>> SourcePackage: python-glance-store
>> UpgradeStatus: No upgrade log present (probably fresh install)
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/glance-store/+bug/1934849/+subscriptions
>>
>> --
>> Ubuntu-openstack-bugs mailing list
>> ubuntu-openstack-b...@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/ubuntu-openstack-bugs
>>
>
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1934849
Title:
s3 backend takes time exponentially
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance-store/+bug/1934849/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1934849] Re: s3 backend takes time exponentially
I think the swift driver uses 200M:
DEFAULT_LARGE_OBJECT_CHUNK_SIZE = 200 # 200M
On Wed, Jul 7, 2021 at 1:50 PM Nobuto Murata <1934...@bugs.launchpad.net>
wrote:
> And by using "4 * units.Mi" it can be 20s.
>
> --
> You received this bug notification because you are a member of Ubuntu
> OpenStack, which is subscribed to python-glance-store in Ubuntu.
> https://bugs.launchpad.net/bugs/1934849
>
> Title:
> s3 backend takes time exponentially
>
> Status in glance_store:
> New
> Status in python-glance-store package in Ubuntu:
> New
>
> Bug description:
> I have a test Ceph cluster as an object storage with both Swift and S3
> protocols enabled for Glance (Ussuri). When I use Swift backend with
> Glance, an image upload completes quickly enough. But with S3 backend
> Glance, it takes much more time to upload an image and it seems to
> rise exponentially.
>
> It's worth noting that when uploading an image with S3 backend, a
> single core is consumed 100% by glance-api process.
>
> for backend in swift s3; do
> for i in {8,16,32,64,128,512}; do
> dd if=/dev/zero of=my-image.img bs=1M count=$i oflag=sync
> time glance image-create \
> --store $backend \
> --file my-image.img --name my-image \
> --disk-format raw --container-format bare \
> --progress
> done
> done
>
> [swift]
> 8MB - 2.4s
> 16MB - 2.8s
> 32MB - 2.6s
> 64MB - 2.7s
> 128MB - 3.1s
> ...
> 512MB - 5.9s
>
> [s3]
> 8MB - 2.2s
> 16MB - 2.9s
> 32MB - 5.5s
> 64MB - 16.3s
> 128MB - 54.9s
> ...
> 512MB - 14m26s
>
> Btw, downloading of 512MB image with S3 backend can complete with less
> than 10 seconds.
>
> $ time openstack image save --file downloaded.img
> 917c5424-4350-4bc5-98ca-66d40e101843
> real0m5.673s
>
> $ du -h downloaded.img
> 512Mdownloaded.img
>
>
> [/etc/glance/glance-api.conf]
>
> enabled_backends = local:file, ceph:rbd, swift:swift, s3:s3
>
> [swift]
> auth_version = 3
> auth_address = http://192.168.151.131:5000/v3
> ...
> container = glance
> large_object_size = 5120
> large_object_chunk_size = 200
>
>
> [s3]
> s3_store_host = http://192.168.151.137:80/
> ...
> s3_store_bucket = zaza-glance-s3-test
> s3_store_large_object_size = 5120
> s3_store_large_object_chunk_size = 200
>
> ProblemType: Bug
> DistroRelease: Ubuntu 20.04
> Package: python3-glance-store 2.0.0-0ubuntu1
> ProcVersionSignature: Ubuntu 5.4.0-77.86-generic 5.4.119
> Uname: Linux 5.4.0-77-generic x86_64
> NonfreeKernelModules: bluetooth ecdh_generic ecc tcp_diag inet_diag
> binfmt_misc veth zfs zunicode zlua zavl icp zcommon znvpair spl unix_diag
> nft_masq nft_chain_nat bridge stp llc vhost_vsock
> vmw_vsock_virtio_transport_common vhost vsock ebtable_filter ebtables
> ip6table_raw ip6table_mangle ip6table_nat ip6table_filter ip6_tables
> iptable_raw iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6
> nf_defrag_ipv4 iptable_filter bpfilter nf_tables nfnetlink dm_multipath
> scsi_dh_rdac scsi_dh_emc scsi_dh_alua kvm_amd ccp input_leds kvm joydev
> mac_hid serio_raw qemu_fw_cfg sch_fq_codel ip_tables x_tables autofs4 btrfs
> zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq
> async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear
> crct10dif_pclmul crc32_pclmul cirrus ghash_clmulni_intel drm_kms_helper
> virtio_net syscopyarea aesni_intel sysfillrect sysimgblt fb_sys_fops
> crypto_simd cryptd drm virtio_blk glue_helper net_failover psmouse failover
> floppy i2c_piix4 pata_acpi
> ApportVersion: 2.20.11-0ubuntu27.18
> Architecture: amd64
> CasperMD5CheckResult: skip
> Date: Wed Jul 7 04:46:05 2021
> PackageArchitecture: all
> ProcEnviron:
>TERM=screen-256color
>PATH=(custom, no user)
>LANG=C.UTF-8
>SHELL=/bin/bash
> SourcePackage: python-glance-store
> UpgradeStatus: No upgrade log present (probably fresh install)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/glance-store/+bug/1934849/+subscriptions
>
> --
> Ubuntu-openstack-bugs mailing list
> ubuntu-openstack-b...@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-openstack-bugs
>
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1934849
Title:
s3 backend takes time exponentially
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance-store/+bug/1934849/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
And by using "4 * units.Mi" it can be 20s. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/glance-store/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
Okay, as the utils.chunkreadable loop is taking time I've tried a larger WRITE_CHUNKSIZE by hand. It can decrease the amount of time of uploading a 512MB image from 14 minutes to 60 seconds. $ git diff diff --git a/glance_store/_drivers/s3.py b/glance_store/_drivers/s3.py index 1c18531..576c573 100644 --- a/glance_store/_drivers/s3.py +++ b/glance_store/_drivers/s3.py @@ -361,7 +361,7 @@ class Store(glance_store.driver.Store): EXAMPLE_URL = "s3://:@//" READ_CHUNKSIZE = 64 * units.Ki -WRITE_CHUNKSIZE = READ_CHUNKSIZE +WRITE_CHUNKSIZE = 1024 * units.Ki @staticmethod def get_schemes(): -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/glance-store/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
The code part in question is this for loop: https://opendev.org/openstack/glance_store/src/branch/stable/ussuri/glance_store/_drivers/s3.py#L638-L644 2021-07-07 11:50:06.735 - def _add_singlepart 2021-07-07 11:50:06.736 - getting into utils.chunkreadable loop 2021-07-07 11:50:06.736 - loop invoked 2021-07-07 11:50:06.737 - loop invoked 2021-07-07 11:50:06.737 - loop invoked ... 2021-07-07 11:50:22.514 - loop invoked (1024 times in total) 2021-07-07 11:50:22.544 - put_object So it's taking time before data is passed to boto3 using put_object. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/glance-store/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
S3 performance itself is not bad. Uploading 512MB object can complete
within a few seconds. So I suppose it's on how Glance S3 driver is using
boto3.
$ time python3 upload.py
real0m3.644s
user0m3.124s
sys 0m1.835s
$ cat upload.py
import boto3
s3 = boto3.client(
"s3",
endpoint_url="http://192.168.151.137:80/;,
aws_access_key_id="Z99Y0ST1T8F66RUZXSJT",
aws_secret_access_key="S5sYaE6pa8btV1HQLtz99RG2DNvX4CpzjOfDMjMZ",
)
with open("my-image.img", "rb") as f:
s3.upload_fileobj(f, "zaza-glance-s3-test", "my-test-object")
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1934849
Title:
s3 backend takes time exponentially
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance-store/+bug/1934849/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
Debug log of when uploading a 512MB image with S3 backend. ** Attachment added: "glance-api.log" https://bugs.launchpad.net/ubuntu/+source/python-glance-store/+bug/1934849/+attachment/5509534/+files/glance-api.log ** Also affects: glance-store Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/glance-store/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1934849] Re: s3 backend takes time exponentially
python3-boto3 1.9.253-1 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1934849 Title: s3 backend takes time exponentially To manage notifications about this bug go to: https://bugs.launchpad.net/glance-store/+bug/1934849/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
