[Yahoo-eng-team] [Bug 1518431] Re: Glance failed to upload image to swift storage
Stuart, Kairat, This bug is hard to reproduce, TBH. The race happens in a combination of Glance's v2 behavior and swift's driver. This is not *entirely* Glance's fault, though. The Glance "weird" part is not being able to recognize "0-sized" chunks before submitting them. However, this is being refactored a bit by one of our current swift specs (need to find the link). The swift issue is that the DELETE is hitting the swift node before the last chunk is written/available. This could be caused by unsynchronized clocks or just a race condition. I've managed to reproduce it in one of our installers CI systems with some frequency but I have yet to find a deterministic way to do so. Meanwhile, I can confirm this issue exists. While it's true that part of this code is being refactored by the "buffered chunk writer" spec, I still think we should fix it as it's eligible for backport. Hope the above helps clarifying a bit the real problem ** Also affects: glance/liberty Importance: Undecided Status: New ** Changed in: glance Importance: Undecided => Medium ** Changed in: glance/liberty Importance: Undecided => Medium ** Changed in: glance/liberty Status: New => Triaged ** Changed in: glance/liberty Assignee: (unassigned) => Cyril Roelandt (cyril-roelandt) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1518431 Title: Glance failed to upload image to swift storage Status in Glance: In Progress Status in Glance liberty series: Triaged Bug description: When glance configured with swift backend, and swift API provides via RadosGW is unable to upload image. Command: glance --debug image-create --name trusty_ext4 --disk-format raw --container-format bare --file trusty-server-cloudimg-amd64.img --visibility public --progress Logs: http://paste.openstack.org/show/479621/ To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1518431/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1518431] Re: Glance failed to upload image to swift storage
Kairat, I reopened the bug because its reproducible with Swift not just with RadosGW. Please, review a previous comments, especially #5, #8 and solution proposed in comment #9. ** Changed in: glance Status: Invalid => In Progress ** Changed in: glance Assignee: Kairat Kushaev (kkushaev) => (unassigned) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1518431 Title: Glance failed to upload image to swift storage Status in Glance: In Progress Bug description: When glance configured with swift backend, and swift API provides via RadosGW is unable to upload image. Command: glance --debug image-create --name trusty_ext4 --disk-format raw --container-format bare --file trusty-server-cloudimg-amd64.img --visibility public --progress Logs: http://paste.openstack.org/show/479621/ To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1518431/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1518431] Re: Glance failed to upload image to swift storage
It took some amount of time to debug this issue for Glance. It turned out that both Glance and Swift support chunked requests. Unfortunately, chunked uploading is not supported mod_fastcgi that is used by RadosGW. It is very typical for Apache to response with 411 error because some cgi (or wsgi) frameworks always require Content-length to be specified. Request with transfer-encoding=chunked is a part of Http spec so glance_store prepares correct request here. So I would recommend to deploy RadosGW under different CGI (for example mod_proxy_fcgi) that supports chunked requests. I will mark this as Invalid for Glance, please re-open the bug if you don't agree. ** Changed in: glance Assignee: (unassigned) => Kairat Kushaev (kkushaev) ** Changed in: glance Status: Confirmed => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1518431 Title: Glance failed to upload image to swift storage Status in Glance: Invalid Bug description: When glance configured with swift backend, and swift API provides via RadosGW is unable to upload image. Command: glance --debug image-create --name trusty_ext4 --disk-format raw --container-format bare --file trusty-server-cloudimg-amd64.img --visibility public --progress Logs: http://paste.openstack.org/show/479621/ To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1518431/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1518431] Re: Glance failed to upload image to swift storage
** No longer affects: mos -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1518431 Title: Glance failed to upload image to swift storage Status in Glance: Confirmed Bug description: When glance configured with swift backend, and swift API provides via RadosGW is unable to upload image. Command: glance --debug image-create --name trusty_ext4 --disk-format raw --container-format bare --file trusty-server-cloudimg-amd64.img --visibility public --progress Logs: http://paste.openstack.org/show/479621/ To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1518431/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1518431] Re: Glance failed to upload image to swift storage
** Also affects: mos Importance: Undecided Status: New ** Changed in: mos Status: New => Confirmed ** Changed in: mos Assignee: (unassigned) => MOS Glance (mos-glance) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1518431 Title: Glance failed to upload image to swift storage Status in Glance: Confirmed Status in Mirantis OpenStack: Confirmed Bug description: When glance configured with swift backend, and swift API provides via RadosGW is unable to upload image. Command: glance --debug image-create --name trusty_ext4 --disk-format raw --container-format bare --file trusty-server-cloudimg-amd64.img --visibility public --progress Logs: http://paste.openstack.org/show/479621/ To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1518431/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1518431] Re: Glance failed to upload image to swift storage
I've debugged this issue a bit further and it's not related to RadosGW.
The swift driver deletes that last 0-byte chunk when the image is being
uploaded and that's causing this conflict.
---
if bytes_read == 0:
# Delete the last chunk, because it's of zero size.
# This will happen if size == 0.
LOG.debug("Deleting final zero-length chunk")
connection.delete_object(location.container,
chunk_name)
break
chunk_id += 1
combined_chunks_size += bytes_read
---
** Summary changed:
- Glance failed to upload image to swift storage via RadosGW
+ Glance failed to upload image to swift storage
** Changed in: glance
Status: Invalid => Confirmed
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1518431
Title:
Glance failed to upload image to swift storage
Status in Glance:
Confirmed
Bug description:
When glance configured with swift backend, and swift API provides via
RadosGW is unable to upload image.
Command:
glance --debug image-create --name trusty_ext4 --disk-format raw
--container-format bare --file trusty-server-cloudimg-amd64.img --visibility
public --progress
Logs:
http://paste.openstack.org/show/479621/
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1518431/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1518431] Re: Glance failed to upload image to swift storage via RadosGW
The swift driver is meant to be used to talk to Swift. If there's a problem in RadosGW swift API, we should notify the RadosGW maintainers. ** Changed in: glance Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1518431 Title: Glance failed to upload image to swift storage via RadosGW Status in Glance: Invalid Bug description: When glance configured with swift backend, and swift API provides via RadosGW is unable to upload image. Command: glance --debug image-create --name trusty_ext4 --disk-format raw --container-format bare --file trusty-server-cloudimg-amd64.img --visibility public --progress Logs: http://paste.openstack.org/show/479621/ To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1518431/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
