Public bug reported:
a recent bug in the cloud-images service caused the metadata to have an incorrect path, which meant that simplestreams got a 403 when requesting an image. However it swallowed that error and apparently uploaded 384 bytes of something, possibly an error HTML page, as the new image to a user's glance store, when used as part of the glance-simplestreams-sync charm. Here's an edited bit of log from http://paste.ubuntu.com/12112343/ (this snippet starts at line 2303 in there) First we see they have a real 20150810 image, size=258343424. DEBUG * 08-15 06:41:20 [PID:2747] * glanceclient.common.http * curl -i -X GET -H 'X-Auth-Token: fe1ca44f89f24174a9ccf3418fddf9d3' -H 'Content-Type: application/json' -H 'User-Agent: python-glanceclient' http://10.16.100.108:9292/v1/images/detail?limit=20 DEBUG * 08-15 06:41:20 [PID:2747] * glanceclient.common.http * HTTP/1.1 200 OK date: Sat, 15 Aug 2015 06:41:20 GMT content-length: 3746 content-type: application/json; charset=UTF-8 x-openstack-request-id: req-req-1c273480-d084-46d9-b951-64ba5714d605 {"images": [{"status": "active", "deleted_at": null, "name": "auto-sync/ubuntu-trusty-14.04-amd64-server-20150810-disk1.img", "deleted": false, "container_format": "bare", "created_at": "2015-08-12T06:46:58.000000", "disk_format": "qcow2", "updated_at": "2015-08-12T06:47:00.000000", "min_disk": 0, "protected": false, "id": "016346a7-8b29-4f2d-8391-e7ad85968ce0", "min_ram": 0, "checksum": "9cf87eeb3c6595fc3965ccf7d9941360", "owner": "f011a026b2fc4165b99ae4740f467d1d", "is_public": true, "virtual_size": null, "properties": {"item_name": "disk1.img", "architecture": "x86_64", "version_name": "20150810", "content_id": "auto.sync", "product_name": "com.ubuntu.cloud:server:14.04:amd64", "source_content_id": "com.ubuntu.cloud:released:download"}, "size": 258343424}, .... snip, edited out a bunch of non-ubuntu images in their glance ]} ## then we see that it got new metadata and decided to delete 0810 and get 0813. INFO * 08-15 06:41:20 [PID:2747] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:15.04:arm64: to_add=[u'20150729'] to_remove=[] INFO * 08-15 06:41:20 [PID:2747] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:13.04:amd64: to_add=[u'20140111'] to_remove=[] INFO * 08-15 06:41:20 [PID:2747] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:15.04:ppc64el: to_add=[u'20150729'] to_remove=[] INFO * 08-15 06:41:20 [PID:2747] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:15.04:armhf: to_add=[u'20150729'] to_remove=[] INFO * 08-15 06:41:20 [PID:2747] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:15.04:amd64: to_add=[u'20150729'] to_remove=[] INFO * 08-15 06:41:20 [PID:2747] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:10.04:amd64: to_add=[u'20150427'] to_remove=[] INFO * 08-15 06:41:20 [PID:2747] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:14.10:arm64: to_add=[u'20150723'] to_remove=[] INFO * 08-15 06:41:20 [PID:2747] * sstreams * com.ubuntu.cloud:released:download/com.ubuntu.cloud:server:14.04:amd64: to_add=[u'20150813'] to_remove=[u'20150810'] DEBUG * 08-15 06:41:20 [PID:2747] * sstreams * getting local copy of http://cloud-images.ubuntu.com/releases/server/releases/trusty/release-20150813/ubuntu-14.04-server-cloudimg-amd64-disk1.img INFO * 08-15 06:41:20 [PID:2747] * urllib3.connectionpool * Starting new HTTP connection (1): cloud-images.ubuntu.com DEBUG * 08-15 06:41:20 [PID:2747] * urllib3.connectionpool * Setting read timeout to <object object at 0x7fc83ec93170> DEBUG * 08-15 06:41:20 [PID:2747] * urllib3.connectionpool * "GET /releases/server/releases/trusty/release-20150813/ubuntu-14.04-server-cloudimg-amd64-disk1.img HTTP/1.1" 403 289 ## uh oh, that path is bad and we got a 403 with 289 bytes left. we should bail here, but instead we upload to glance: DEBUG * 08-15 06:41:20 [PID:2747] * glanceclient.common.http * curl -i -X POST -H 'x-image-meta-property-source_content_id: com.ubuntu.cloud:released:download' -H 'X-Auth-Token: fe1ca44f89f24174a9ccf3418fddf9d3' -H 'x-image-meta-property-version_name: 20150813' -H 'x-image-meta-property-product_name: com.ubuntu.cloud:server:14.04:amd64' -H 'x-image-meta-checksum: f655dd7ce96dd5b76b663edcec931b8e' -H 'Transfer-Encoding: chunked' -H 'x-image-meta-container_format: bare' -H 'x-image-meta-property-content_id: auto.sync' -H 'User-Agent: python-glanceclient' -H 'x-image-meta-property-item_name: disk1.img' -H 'x-image-meta-property-architecture: x86_64' -H 'x-image-meta-is_public: True' -H 'x-image-meta-size: 384' -H 'Content-Type: application/octet-stream' -H 'x-image-meta-disk_format: qcow2' -H 'x-image-meta-name: auto-sync/ubuntu-trusty-14.04-amd64-server-20150813-disk1.img' -d '<open file '/tmp/tmpcb3szh', mode 'rb' at 0x7fc8390ac300>' http://10.16.100.108:9292/v1/images ## note the image-meta-size: 384 DEBUG * 08-15 06:41:21 [PID:2747] * glanceclient.common.http * HTTP/1.1 201 Created content-length: 761 etag: f655dd7ce96dd5b76b663edcec931b8e location: http://10.16.100.108:9292/v1/images/1c22ab11-168e-42b1-a772-16a761c75f32 date: Sat, 15 Aug 2015 06:41:21 GMT content-type: application/json x-openstack-request-id: req-req-d31a8004-c158-432c-b70c-13cc7811eaac {"image": {"status": "active", "deleted": false, "container_format": "bare", "min_ram": 0, "updated_at": "2015-08-15T06:41:21.000000", "owner": "f011a026b2fc4165b99ae4740f467d1d", "min_disk": 0, "is_public": true, "deleted_at": null, "id": "1c22ab11-168e-42b1-a772-16a761c75f32", "size": 384, "virtual_size": null, "name": "auto-sync/ubuntu- trusty-14.04-amd64-server-20150813-disk1.img", "checksum": "f655dd7ce96dd5b76b663edcec931b8e", "created_at": "2015-08-15T06:41:21.000000", "disk_format": "qcow2", "properties": {"item_name": "disk1.img", "architecture": "x86_64", "version_name": "20150813", "content_id": "auto.sync", "product_name": "com.ubuntu.cloud:server:14.04:amd64", "source_content_id": "com.ubuntu.cloud:released:download"}, "protected": false}} ## note the size: 384. at this point, glance has a broken image but further runs of the sync charm don't know that so they don't get a working image automatically until they either delete that broken image or we release a new image. ** Affects: simplestreams (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to simplestreams in Ubuntu. https://bugs.launchpad.net/bugs/1485785 Title: simplestreams in trusty ignores http 403 when downloading imag To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/simplestreams/+bug/1485785/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
