Re: [openstack-dev] Problems with all OpenStack APIs & uwsgi with Content-Lenght and connection reset by peer (ie: 104)

2018-05-02 Thread Erik Olof Gunnar Andersson 
I noticed something similar when deploying Keystone using nginx in the lab, and 
pretty sure I fixed it by setting  uwsgi_ignore_client_abort to on.
http://nginx.org/en/docs/http/ngx_http_uwsgi_module.html

In addition to that flag I also have
client_header_buffer_size 64k;

uwsgi_buffer_size 8k;
uwsgi_read_timeout600;
uwsgi_send_timeout600;

Best Regards, Erik Olof Gunnar Andersson

-Original Message-
From: Thomas Goirand  
Sent: Wednesday, May 2, 2018 11:28 AM
To: OpenStack Development Mailing List (not for usage questions) 

Subject: Re: [openstack-dev] Problems with all OpenStack APIs & uwsgi with 
Content-Lenght and connection reset by peer (ie: 104)

On 05/02/2018 10:25 AM, Chris Dent wrote:
> On Wed, 2 May 2018, Thomas Goirand wrote:
> 
>> What was disturbing was that, doing the same request with curl worked 
>> perfectly. Even more disturbing, it looked like I was having the 
>> issue nearly always in virtualbox, but not always in real hardware, 
>> where it sometimes worked.
> 
> What was making the request in the first place? It fails in X, but 
> works in curl. What is X?

For example, nova-compute querying nova-placement-api.
Another example: openstackclient. It happened to me trying to configure 
keystone when running puppet-openstack, for example, but on the command line 
directly as well, simply trying to add users, projects, etc. This looks like to 
me as a general problem in all of the OpenStack WSGI applications.

>> Anyway, finally, I figured out that adding:
>>
>> --rem-header Content-Lenght
> 
> You added this arg to what?

As a parameter to uwsgi, so that it removes the Content-Lenght that the WSGI 
application sends.

>> This however, looks like a workaround rather than a fix, and I wonder 
>> if there's a real issue somewhere that needs to be fixed in a better 
>> way, maybe in openstackclient or some other component...
> 
> Yeah, it sounds like something could be setting a bad value for the 
> content length header and uwsgi is timing out while trying to read 
> that much data (meaning, it is believing the content-length header) 
> but there isn't anything actually there.
> 
> Another option is that there are buffer size problems in the uwsgi 
> configuration but it's hard to speculate because it is not clear what 
> requests and tools you're actually talking about here.

When attempting to google for the issue, I saw a lot of people that had this 
problem fixed by adding --buffer-size 65535, as the default 4k header of uwsgi 
was not enough. I also have this option set, as it seems a reasonable thing to 
have, but that was not enough to fix the problem.
Only the --rem-header thing did.

If you want to try, you can simply use the stretch-queens.debian.net repository 
with Glance (or simply Debian Sid), and edit /etc/glance/glance-api-wsgi.ini to 
change the uwsgi parameters (I've just switched Glance to uwsgi, since it now 
works...). I haven't checked with Glance, but since I saw the problem with 
nova-placement-api, cinder-api and keystone, I don't see why it wouldn't happen 
there.

Cheers,

Thomas Goirand (zigo)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Problems with all OpenStack APIs & uwsgi with Content-Lenght and connection reset by peer (ie: 104)

2018-05-02 Thread Thomas Goirand 
On 05/02/2018 10:25 AM, Chris Dent wrote:
> On Wed, 2 May 2018, Thomas Goirand wrote:
> 
>> What was disturbing was that, doing the same request with curl worked
>> perfectly. Even more disturbing, it looked like I was having the issue
>> nearly always in virtualbox, but not always in real hardware, where it
>> sometimes worked.
> 
> What was making the request in the first place? It fails in X, but
> works in curl. What is X?

For example, nova-compute querying nova-placement-api.
Another example: openstackclient. It happened to me trying to configure
keystone when running puppet-openstack, for example, but on the command
line directly as well, simply trying to add users, projects, etc. This
looks like to me as a general problem in all of the OpenStack WSGI
applications.

>> Anyway, finally, I figured out that adding:
>>
>> --rem-header Content-Lenght
> 
> You added this arg to what?

As a parameter to uwsgi, so that it removes the Content-Lenght that the
WSGI application sends.

>> This however, looks like a workaround rather than a fix, and I wonder if
>> there's a real issue somewhere that needs to be fixed in a better way,
>> maybe in openstackclient or some other component...
> 
> Yeah, it sounds like something could be setting a bad value for the
> content length header and uwsgi is timing out while trying to read
> that much data (meaning, it is believing the content-length header)
> but there isn't anything actually there.
> 
> Another option is that there are buffer size problems in the uwsgi
> configuration but it's hard to speculate because it is not clear
> what requests and tools you're actually talking about here.

When attempting to google for the issue, I saw a lot of people that had
this problem fixed by adding --buffer-size 65535, as the default 4k
header of uwsgi was not enough. I also have this option set, as it seems
a reasonable thing to have, but that was not enough to fix the problem.
Only the --rem-header thing did.

If you want to try, you can simply use the stretch-queens.debian.net
repository with Glance (or simply Debian Sid), and edit
/etc/glance/glance-api-wsgi.ini to change the uwsgi parameters (I've
just switched Glance to uwsgi, since it now works...). I haven't checked
with Glance, but since I saw the problem with nova-placement-api,
cinder-api and keystone, I don't see why it wouldn't happen there.

Cheers,

Thomas Goirand (zigo)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Problems with all OpenStack APIs & uwsgi with Content-Lenght and connection reset by peer (ie: 104)

2018-05-02 Thread Chris Dent 

On Wed, 2 May 2018, Thomas Goirand wrote:


What was disturbing was that, doing the same request with curl worked
perfectly. Even more disturbing, it looked like I was having the issue
nearly always in virtualbox, but not always in real hardware, where it
sometimes worked.


What was making the request in the first place? It fails in X, but
works in curl. What is X?


Anyway, finally, I figured out that adding:

--rem-header Content-Lenght


You added this arg to what?

In both cases do you mean openstackclient?


This however, looks like a workaround rather than a fix, and I wonder if
there's a real issue somewhere that needs to be fixed in a better way,
maybe in openstackclient or some other component...


Yeah, it sounds like something could be setting a bad value for the
content length header and uwsgi is timing out while trying to read
that much data (meaning, it is believing the content-length header)
but there isn't anything actually there.

Another option is that there are buffer size problems in the uwsgi
configuration but it's hard to speculate because it is not clear
what requests and tools you're actually talking about here.

--
Chris Dent   ٩◔̯◔۶   https://anticdent.org/
freenode: cdent tw: @anticdent__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Problems with all OpenStack APIs & uwsgi with Content-Lenght and connection reset by peer (ie: 104)

2018-05-02 Thread Thomas Goirand 
Hi there!

It's been a month I was knocking my head on the wall trying to get uwsgi
working with all of OpenStack API uwsgi applications. Indeed, when
OpenStack component (like for example, nova-compute) were talking to
uwsgi, then they were receiving a 104 error (ie: connection reset by
peer) before getting an answer.

What was disturbing was that, doing the same request with curl worked
perfectly. Even more disturbing, it looked like I was having the issue
nearly always in virtualbox, but not always in real hardware, where it
sometimes worked.

Anyway, finally, I figured out that adding:

--rem-header Content-Lenght

fixed everything. I was able to spawn instances in virtualbox.

This however, looks like a workaround rather than a fix, and I wonder if
there's a real issue somewhere that needs to be fixed in a better way,
maybe in openstackclient or some other component...

Thoughts anyone?

Cheers,

Thomas Goirand (zigo)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev