We're working on that. The original Ceph bug has been fixed in Jewel and backported to Hammer. Now Glance is trying to use image.size() instead of image.stat() to get the size. Here is the link https://review.openstack.org/#/c/378460/
On 01/02/17 01:22, Grant Morley wrote: > > Hi Nick, > > Thanks for the reply, looks like we could be hitting something > similar. We are running Ceph Jewel packages on the Glance node. > > Thanks for the links to the bug reports. > > Regards, > > > On 31/01/17 11:35, Nick Jones wrote: >> Hi Grant. >> >> Could be unrelated but I'm throwing it out there anyway as we had >> similar 'weirdness' with Glance recently... If you're running the >> Ceph Jewel or Hammer packages on your Glance node then you might be >> running into this bug, in which image.stat() calls in librbdpy fail >> intermittently: >> >> http://tracker.ceph.com/issues/17310 >> >> There's a corresponding bug report on Launchpad for the Ubuntu Ceph >> package that has some more detail: >> >> https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1625489 >> >> Cheers. >> >> -- >> >> -Nick >> >> On 31 January 2017 at 10:49, Grant Morley <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi All, >> >> Not sure if anyone has come across this yet. Yesterday we started >> to try to upload some images to our stack and they were failing >> with the following error when trying via command line: >> >> Error finding address for >> http://10.6.0.3:9292/v2/images/e90a4626-4781-4b53-8914-85ff2129f777/file >> >> <http://10.6.0.3:9292/v2/images/e90a4626-4781-4b53-8914-85ff2129f777/file>: >> [Errno 32] Broken pipe >> >> In the Glance logs at the same time we see this: >> >> 20172017-01-31 10:22:45.005 2096 INFO eventlet.wsgi.server >> [req-1837bce5-a90c-44d5-bf65-c1e3a534c793 >> 4c91f07132454a97b21fff35402b7825 4a6213a64312482896130efc3047195c >> - - -] 10.6.1.223,10.6.0.39 - - [31/Jan/2017 10:22:45] "HEAD >> /v1/images/e90a4626-4781-4b53-8914-85ff2129f777 HTTP/1.1" 200 689 >> 0.046623 >> 2017-01-31 10:22:47.595 2099 INFO eventlet.wsgi.server [-] >> 10.6.0.40 - - [31/Jan/2017 10:22:47] "OPTIONS /versions HTTP/1.0" >> 200 785 0.001113 >> 2017-01-31 10:22:47.720 2095 INFO eventlet.wsgi.server >> [req-873dd7df-f8a4-4443-8795-b81dcd54f412 >> 4c91f07132454a97b21fff35402b7825 4a6213a64312482896130efc3047195c >> - - -] 10.6.1.223,10.6.0.39 - - [31/Jan/2017 10:22:47] "HEAD >> /v1/images/e90a4626-4781-4b53-8914-85ff2129f777 HTTP/1.1" 200 689 >> 0.032188-01-31 10:22:45.005 2096 INFO eventlet.wsgi.server >> [req-1837bce5-a90c-44d5-bf65-c1e3a534c793 >> 4c91f07132454a97b21fff35402b7825 4a6213a64312482896130efc3047195c >> - - -] 10.6.1.223,10.6.0.39 - - [31/Jan/2017 10:22:45] "HEAD >> /v1/images/e90a4626-4781-4b53-8914-85ff2129f777 HTTP/1.1" 200 689 >> 0.046623 >> 2017-01-31 10:22:47.595 2099 INFO eventlet.wsgi.server [-] >> 10.6.0.40 - - [31/Jan/2017 10:22:47] "OPTIONS /versions HTTP/1.0" >> 200 785 0.001113 >> 2017-01-31 10:22:47.720 2095 INFO eventlet.wsgi.server >> [req-873dd7df-f8a4-4443-8795-b81dcd54f412 >> 4c91f07132454a97b21fff35402b7825 4a6213a64312482896130efc3047195c >> - - -] 10.6.1.223,10.6.0.39 - - [31/Jan/2017 10:22:47] "HEAD >> /v1/images/e90a4626-4781-4b53-8914-85ff2129f777 HTTP/1.1" 200 689 >> 0.032188 >> >> Interestingly, when you delete the failed image you get this in >> the glance logs: >> >> 2017-01-31 10:23:16.188 2099 INFO eventlet.wsgi.server >> [req-c2191fe1-64de-4252-b4a2-e84643dbed4c >> 4c91f07132454a97b21fff35402b7825 4a6213a64312482896130efc3047195c >> - - -] 10.6.2.190,10.6.0.39 - - [31/Jan/2017 10:23:16] "DELETE >> /v2/images/e90a4626-4781-4b53-8914-85ff2129f777 HTTP/1.1" 204 208 >> 0.08 >> >> >> It seems to be using V1 to try and upload and V2 to remove? - >> However the image doesn't actually get deleted because it remains >> in ceph: >> >> rbd -p images ls | grep e90a4626-4781-4b53-8914-85ff2129f777 >> e90a4626-4781-4b53-8914-85ff2129f777 >> >> It is almost as if, the image uploads and is then forgotten about. >> >> Our compute nodes are also ceph backed and they are working >> absolutely fine, it is just Glance and ceph has all of a sudden >> stopped working. Just wondered if anyone had any ideas? >> >> Regards, >> >> >> -- >> Grant Morley >> Cloud Lead >> Absolute DevOps Ltd >> Units H, J & K, Gateway 1000, Whittle Way, Stevenage, Herts, SG1 2FP >> www.absolutedevops.io <http://www.absolutedevops.io/> >> [email protected] <mailto:[email protected]> 0845 874 >> 0580 >> >> _______________________________________________ >> OpenStack-operators mailing list >> [email protected] >> <mailto:[email protected]> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators> >> >> >> >> DataCentred Limited registered in England and Wales no. 05611763 > > -- > Grant Morley > Cloud Lead > Absolute DevOps Ltd > Units H, J & K, Gateway 1000, Whittle Way, Stevenage, Herts, SG1 2FP > www.absolutedevops.io <http://www.absolutedevops.io/> > [email protected] <mailto:[email protected]> 0845 874 0580 > > > _______________________________________________ > OpenStack-operators mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators -- Cheers & Best regards, FeiLong Wang (王飞龙) -------------------------------------------------------------------------- Senior Cloud Software Engineer Tel: +64-48032246 Email: [email protected] Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington --------------------------------------------------------------------------
_______________________________________________ OpenStack-operators mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
