Re: [openstack-dev] [nova] [glance] How to deal with aborted image read?

2015-06-09 Thread Chris Friesen
On 06/08/2015 06:26 PM, Robert Collins wrote: On 9 June 2015 at 07:53, Chris Friesen wrote: From what I understand, the iterator (in the glance-api process) normally breaks out of the while loop once the whole file has been read and the read() call returns an empty string. It's not clear to m

Re: [openstack-dev] [nova] [glance] How to deal with aborted image read?

2015-06-08 Thread Robert Collins
On 9 June 2015 at 07:53, Chris Friesen wrote: > On 06/08/2015 12:30 PM, Robert Collins wrote: >> >> On 9 June 2015 at 03:50, Chris Friesen >> wrote: >>> >>> On 06/07/2015 04:22 PM, Robert Collins wrote: >> >> >>> Hi, original reporter here. >>> >>> There's no LB involved. The issue was noticed i

Re: [openstack-dev] [nova] [glance] How to deal with aborted image read?

2015-06-08 Thread Chris Friesen
On 06/08/2015 12:30 PM, Robert Collins wrote: On 9 June 2015 at 03:50, Chris Friesen wrote: On 06/07/2015 04:22 PM, Robert Collins wrote: Hi, original reporter here. There's no LB involved. The issue was noticed in a test lab that is tight on disk space. When an instance failed to boot th

Re: [openstack-dev] [nova] [glance] How to deal with aborted image read?

2015-06-08 Thread Robert Collins
On 9 June 2015 at 03:50, Chris Friesen wrote: > On 06/07/2015 04:22 PM, Robert Collins wrote: > Hi, original reporter here. > > There's no LB involved. The issue was noticed in a test lab that is tight > on disk space. When an instance failed to boot the person using the lab > tried to delete s

Re: [openstack-dev] [nova] [glance] How to deal with aborted image read?

2015-06-08 Thread Chris Friesen
On 06/07/2015 04:22 PM, Robert Collins wrote: On 6 June 2015 at 13:08, Ian Cordasco wrote: So the problem is with how we use ResponseSerializer and the ChunkedFile (https://git.openstack.org/cgit/openstack/glance/tree/glance/api/v2/image_d ata.py#n222). I think the problem we'll have is that

Re: [openstack-dev] [nova] [glance] How to deal with aborted image read?

2015-06-07 Thread Robert Collins
On 6 June 2015 at 13:08, Ian Cordasco wrote: > > > On 6/5/15, 02:55, "Flavio Percoco" wrote: > >>On 04/06/15 11:46 -0600, Chris Friesen wrote: >>>On 06/04/2015 03:01 AM, Flavio Percoco wrote: On 03/06/15 16:46 -0600, Chris Friesen wrote: >We recently ran into an issue where nova couldn't

Re: [openstack-dev] [nova] [glance] How to deal with aborted image read?

2015-06-05 Thread Ian Cordasco
On 6/5/15, 02:55, "Flavio Percoco" wrote: >On 04/06/15 11:46 -0600, Chris Friesen wrote: >>On 06/04/2015 03:01 AM, Flavio Percoco wrote: >>>On 03/06/15 16:46 -0600, Chris Friesen wrote: We recently ran into an issue where nova couldn't write an image file due to lack of space and so

Re: [openstack-dev] [nova] [glance] How to deal with aborted image read?

2015-06-05 Thread Flavio Percoco
On 04/06/15 11:46 -0600, Chris Friesen wrote: On 06/04/2015 03:01 AM, Flavio Percoco wrote: On 03/06/15 16:46 -0600, Chris Friesen wrote: We recently ran into an issue where nova couldn't write an image file due to lack of space and so just quit reading from glance. This caused glance to be st

Re: [openstack-dev] [nova] [glance] How to deal with aborted image read?

2015-06-04 Thread Chris Friesen
On 06/04/2015 07:16 PM, Joshua Harlow wrote: Perhaps someone needs to use (or forgot to use) contextlib.closing? https://docs.python.org/2/library/contextlib.html#contextlib.closing Or contextlib2 also may be useful: http://contextlib2.readthedocs.org/en/latest/#contextlib2.ExitStack The com

Re: [openstack-dev] [nova] [glance] How to deal with aborted image read?

2015-06-04 Thread Joshua Harlow
Perhaps someone needs to use (or forgot to use) contextlib.closing? https://docs.python.org/2/library/contextlib.html#contextlib.closing Or contextlib2 also may be useful: http://contextlib2.readthedocs.org/en/latest/#contextlib2.ExitStack -Josh Chris Friesen wrote: On 06/04/2015 03:01 AM, F

Re: [openstack-dev] [nova] [glance] How to deal with aborted image read?

2015-06-04 Thread Chris Friesen
On 06/04/2015 03:01 AM, Flavio Percoco wrote: On 03/06/15 16:46 -0600, Chris Friesen wrote: We recently ran into an issue where nova couldn't write an image file due to lack of space and so just quit reading from glance. This caused glance to be stuck with an open file descriptor, which meant t

Re: [openstack-dev] [nova] [glance] How to deal with aborted image read?

2015-06-04 Thread Flavio Percoco
On 03/06/15 16:46 -0600, Chris Friesen wrote: We recently ran into an issue where nova couldn't write an image file due to lack of space and so just quit reading from glance. This caused glance to be stuck with an open file descriptor, which meant that the image consumed space even after it wa

[openstack-dev] [nova] [glance] How to deal with aborted image read?

2015-06-03 Thread Chris Friesen
We recently ran into an issue where nova couldn't write an image file due to lack of space and so just quit reading from glance. This caused glance to be stuck with an open file descriptor, which meant that the image consumed space even after it was deleted. I have a crude fix for nova at "ht