Re: [openstack-dev] [glance][rally] Disabling Glance Testing in Rally gates
Initial finding: This is something related to uwsgi socket timeout. Default value for socket-timeout is 30 in /etc/glance/glance-uwsgi.ini file. For testing purpose I have tried to create 10 GB image using create command: $ time glance --debug image-create --name dsl --file gentoo_root.img --disk-format iso --container-format bare real2m48.539s user0m4.076s sys 0m10.012s It has failed with “502 bad gateway” after 3 minutes. Then I have increased this timeout to 60 and restarted glance-api service and ran above command again. $ time glance --debug image-create --name dsl --file gentoo_root.img --disk-format iso --container-format bare After this I was able to create image without any issue. So another workaround is to set socket-timeout to higher value in /etc/glance/glance-uwsgi.ini file. Thank you, Abhishek Kekane From: Kekane, Abhishek [mailto:abhishek.kek...@nttdata.com] Sent: Thursday, July 13, 2017 2:45 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [glance][rally] Disabling Glance Testing in Rally gates Someone has reported this issue in glance recently. Please refer, https://bugs.launchpad.net/glance/+bug/1703856 I think it is same kind of failure which Andrey was mentioning. As mentioned in the bug temporary solution is to enable the v1 api using the v1 parameter, --location instead of using --file. But this will not be applicable to rally scenarios I guess. I will spend some time to understand the failure cause. Thank you, Abhishek From: Andrey Kurilin [mailto:andr.kuri...@gmail.com] Sent: Thursday, July 13, 2017 2:25 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [glance][rally] Disabling Glance Testing in Rally gates Hi Flavio, 2017-07-13 11:16 GMT+03:00 Flavio Percoco mailto:fla...@redhat.com>>: On 13/07/17 00:56 -0700, Boris Pavlovic wrote: Hi stackers, Unfortunately what was discussed in other thread (situation in glance is critical) happened. Glance stopped working and Rally team is forced to disable checking of it in Rally gates. P.S. Seems like this patch is casing the problems: https://github.com/openstack-dev/devstack/commit/1fa653635781cd975a1031e212b35b6c38196ba4 Hey Boris, Has this been brought up to the Glance team? Or is this email meant to do that? Yes http://eavesdrop.openstack.org/irclogs/%23openstack-glance/%23openstack-glance.2017-07-11.log.html#t2017-07-11T07:44:13 http://eavesdrop.openstack.org/irclogs/%23openstack-glance/%23openstack-glance.2017-07-11.log.html#t2017-07-11T22:12:47 FWIW, the switch to uwsgi is a community goal and not so much a Glance thing. Would you mind elaborating on what exactly is failing and how the glance team can help? I understand that switching to uwsgi is a community goal, but we didn't have any problems with Glance for years until now. As from IRC logs: > HTTPBadGateway: 502 Bad Gateway: Bad Gateway: The proxy server received an > invalid: response from an upstream server.: Apache/2.4.18 (Ubuntu) Server at > 158.69.73.26 Port 80 (HTTP 502) Trace: http://paste.openstack.org/show/615234/ Flavio -- @flaper87 Flavio Percoco __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best regards, Andrey Kurilin. __ Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding. __ Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance][rally] Disabling Glance Testing in Rally gates
Someone has reported this issue in glance recently. Please refer, https://bugs.launchpad.net/glance/+bug/1703856 I think it is same kind of failure which Andrey was mentioning. As mentioned in the bug temporary solution is to enable the v1 api using the v1 parameter, --location instead of using --file. But this will not be applicable to rally scenarios I guess. I will spend some time to understand the failure cause. Thank you, Abhishek From: Andrey Kurilin [mailto:andr.kuri...@gmail.com] Sent: Thursday, July 13, 2017 2:25 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [glance][rally] Disabling Glance Testing in Rally gates Hi Flavio, 2017-07-13 11:16 GMT+03:00 Flavio Percoco mailto:fla...@redhat.com>>: On 13/07/17 00:56 -0700, Boris Pavlovic wrote: Hi stackers, Unfortunately what was discussed in other thread (situation in glance is critical) happened. Glance stopped working and Rally team is forced to disable checking of it in Rally gates. P.S. Seems like this patch is casing the problems: https://github.com/openstack-dev/devstack/commit/1fa653635781cd975a1031e212b35b6c38196ba4 Hey Boris, Has this been brought up to the Glance team? Or is this email meant to do that? Yes http://eavesdrop.openstack.org/irclogs/%23openstack-glance/%23openstack-glance.2017-07-11.log.html#t2017-07-11T07:44:13 http://eavesdrop.openstack.org/irclogs/%23openstack-glance/%23openstack-glance.2017-07-11.log.html#t2017-07-11T22:12:47 FWIW, the switch to uwsgi is a community goal and not so much a Glance thing. Would you mind elaborating on what exactly is failing and how the glance team can help? I understand that switching to uwsgi is a community goal, but we didn't have any problems with Glance for years until now. As from IRC logs: > HTTPBadGateway: 502 Bad Gateway: Bad Gateway: The proxy server received an > invalid: response from an upstream server.: Apache/2.4.18 (Ubuntu) Server at > 158.69.73.26 Port 80 (HTTP 502) Trace: http://paste.openstack.org/show/615234/ Flavio -- @flaper87 Flavio Percoco __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best regards, Andrey Kurilin. __ Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance][rally] Disabling Glance Testing in Rally gates
Hi Flavio, 2017-07-13 11:16 GMT+03:00 Flavio Percoco : > On 13/07/17 00:56 -0700, Boris Pavlovic wrote: > >> Hi stackers, >> >> >> Unfortunately what was discussed in other thread (situation in glance is >> critical) happened. >> Glance stopped working and Rally team is forced to disable checking of it >> in Rally gates. >> >> P.S. Seems like this patch is casing the problems: >> https://github.com/openstack-dev/devstack/commit/1fa65363578 >> 1cd975a1031e212b35b6c38196ba4 >> > > Hey Boris, > > Has this been brought up to the Glance team? Or is this email meant to do > that? > > Yes http://eavesdrop.openstack.org/irclogs/%23openstack-glance/%23openstack-glance.2017-07-11.log.html#t2017-07-11T07:44:13 http://eavesdrop.openstack.org/irclogs/%23openstack-glance/%23openstack-glance.2017-07-11.log.html#t2017-07-11T22:12:47 > FWIW, the switch to uwsgi is a community goal and not so much a Glance > thing. > Would you mind elaborating on what exactly is failing and how the glance > team > can help? > > I understand that switching to uwsgi is a community goal, but we didn't have any problems with Glance for years until now. As from IRC logs: > HTTPBadGateway: 502 Bad Gateway: Bad Gateway: The proxy server received an invalid: response from an upstream server.: Apache/2.4.18 (Ubuntu) Server at 158.69.73.26 Port 80 (HTTP 502) Trace: http://paste.openstack.org/show/615234/ Flavio > > -- > @flaper87 > Flavio Percoco > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- Best regards, Andrey Kurilin. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [glance][rally] Disabling Glance Testing in Rally gates
On 13/07/17 00:56 -0700, Boris Pavlovic wrote: Hi stackers, Unfortunately what was discussed in other thread (situation in glance is critical) happened. Glance stopped working and Rally team is forced to disable checking of it in Rally gates. P.S. Seems like this patch is casing the problems: https://github.com/openstack-dev/devstack/commit/1fa653635781cd975a1031e212b35b6c38196ba4 Hey Boris, Has this been brought up to the Glance team? Or is this email meant to do that? FWIW, the switch to uwsgi is a community goal and not so much a Glance thing. Would you mind elaborating on what exactly is failing and how the glance team can help? Flavio -- @flaper87 Flavio Percoco signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [glance][rally] Disabling Glance Testing in Rally gates
Hi stackers, Unfortunately what was discussed in other thread (situation in glance is critical) happened. Glance stopped working and Rally team is forced to disable checking of it in Rally gates. P.S. Seems like this patch is casing the problems: https://github.com/openstack-dev/devstack/commit/1fa653635781cd975a1031e212b35b6c38196ba4 Best regards, Boris Pavlovic __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev