Hi All,
Not sure if anyone has come across this yet. Yesterday we started to try
to upload some images to our stack and they were failing with the
following error when trying via command line:
Error finding address for
http://10.6.0.3:9292/v2/images/e90a4626-4781-4b53-8914-85ff2129f777/file:
[Errno 32] Broken pipe
In the Glance logs at the same time we see this:
20172017-01-31 10:22:45.005 2096 INFO eventlet.wsgi.server
[req-1837bce5-a90c-44d5-bf65-c1e3a534c793
4c91f07132454a97b21fff35402b7825 4a6213a64312482896130efc3047195c - - -]
10.6.1.223,10.6.0.39 - - [31/Jan/2017 10:22:45] "HEAD
/v1/images/e90a4626-4781-4b53-8914-85ff2129f777 HTTP/1.1" 200 689 0.046623
2017-01-31 10:22:47.595 2099 INFO eventlet.wsgi.server [-] 10.6.0.40 - -
[31/Jan/2017 10:22:47] "OPTIONS /versions HTTP/1.0" 200 785 0.001113
2017-01-31 10:22:47.720 2095 INFO eventlet.wsgi.server
[req-873dd7df-f8a4-4443-8795-b81dcd54f412
4c91f07132454a97b21fff35402b7825 4a6213a64312482896130efc3047195c - - -]
10.6.1.223,10.6.0.39 - - [31/Jan/2017 10:22:47] "HEAD
/v1/images/e90a4626-4781-4b53-8914-85ff2129f777 HTTP/1.1" 200 689
0.032188-01-31 10:22:45.005 2096 INFO eventlet.wsgi.server
[req-1837bce5-a90c-44d5-bf65-c1e3a534c793
4c91f07132454a97b21fff35402b7825 4a6213a64312482896130efc3047195c - - -]
10.6.1.223,10.6.0.39 - - [31/Jan/2017 10:22:45] "HEAD
/v1/images/e90a4626-4781-4b53-8914-85ff2129f777 HTTP/1.1" 200 689 0.046623
2017-01-31 10:22:47.595 2099 INFO eventlet.wsgi.server [-] 10.6.0.40 - -
[31/Jan/2017 10:22:47] "OPTIONS /versions HTTP/1.0" 200 785 0.001113
2017-01-31 10:22:47.720 2095 INFO eventlet.wsgi.server
[req-873dd7df-f8a4-4443-8795-b81dcd54f412
4c91f07132454a97b21fff35402b7825 4a6213a64312482896130efc3047195c - - -]
10.6.1.223,10.6.0.39 - - [31/Jan/2017 10:22:47] "HEAD
/v1/images/e90a4626-4781-4b53-8914-85ff2129f777 HTTP/1.1" 200 689 0.032188
Interestingly, when you delete the failed image you get this in the
glance logs:
2017-01-31 10:23:16.188 2099 INFO eventlet.wsgi.server
[req-c2191fe1-64de-4252-b4a2-e84643dbed4c
4c91f07132454a97b21fff35402b7825 4a6213a64312482896130efc3047195c - - -]
10.6.2.190,10.6.0.39 - - [31/Jan/2017 10:23:16] "DELETE
/v2/images/e90a4626-4781-4b53-8914-85ff2129f777 HTTP/1.1" 204 208 0.08
It seems to be using V1 to try and upload and V2 to remove? - However
the image doesn't actually get deleted because it remains in ceph:
rbd -p images ls | grep e90a4626-4781-4b53-8914-85ff2129f777
e90a4626-4781-4b53-8914-85ff2129f777
It is almost as if, the image uploads and is then forgotten about.
Our compute nodes are also ceph backed and they are working absolutely
fine, it is just Glance and ceph has all of a sudden stopped working.
Just wondered if anyone had any ideas?
Regards,
--
Grant Morley
Cloud Lead
Absolute DevOps Ltd
Units H, J & K, Gateway 1000, Whittle Way, Stevenage, Herts, SG1 2FP
www.absolutedevops.io <http://www.absolutedevops.io/>
[email protected] <mailto:[email protected]> 0845 874 0580
_______________________________________________
OpenStack-operators mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
