Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

That's what I tried first :) For some reason load distribution was still uneven. I'll check this again, maybe I missed something. On Tue, Feb 23, 2016 at 5:37 PM, Chris Friesen wrote: > On 02/23/2016 05:25 AM, Roman Podoliaka wrote: > >> So looks like it's two

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

On 02/23/2016 05:25 AM, Roman Podoliaka wrote: So looks like it's two related problems here: 1) the distribution of load between workers is uneven. One way to fix this is to decrease the default number of greenlets in pool [2], which will effectively cause a particular worker to give up new

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

Hi all, I've taken another look at this in order to propose patches to oslo.service/oslo.db, so that we have better defaults for WSGI greenlets number / max DB connections overflow [1] [2], which would be more suitable for DB oriented services like our APIs are. I used the Mike's snippet [3] for

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

On 01/08/2016 09:51 PM, Mike Bayer wrote: On 01/08/2016 04:44 AM, Radomir Dopieralski wrote: On 01/07/2016 05:55 PM, Mike Bayer wrote: but also even if you're under something like mod_wsgi, you can spawn a child process or worker thread regardless. You always have a Python interpreter

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

On 01/11/2016 05:39 AM, Radomir Dopieralski wrote: > On 01/08/2016 09:51 PM, Mike Bayer wrote: >> >> >> On 01/08/2016 04:44 AM, Radomir Dopieralski wrote: >>> On 01/07/2016 05:55 PM, Mike Bayer wrote: >>> but also even if you're under something like mod_wsgi, you can spawn a child

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

On 01/08/2016 04:44 AM, Radomir Dopieralski wrote: > On 01/07/2016 05:55 PM, Mike Bayer wrote: > >> but also even if you're under something like >> mod_wsgi, you can spawn a child process or worker thread regardless. >> You always have a Python interpreter running and all the things it can >>

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

On 01/07/2016 06:55 PM, Mike Bayer wrote: On 01/07/2016 11:02 AM, Sean Dague wrote: On 01/07/2016 09:56 AM, Brant Knudson wrote: On Thu, Jan 7, 2016 at 6:39 AM, Clayton O'Neill > wrote: On Thu, Jan 7, 2016 at 2:49 AM, Roman Podoliaka

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

On 01/07/2016 05:44 PM, Mike Bayer wrote: On 01/07/2016 07:39 AM, Clayton O'Neill wrote: On Thu, Jan 7, 2016 at 2:49 AM, Roman Podoliaka wrote: Linux gurus please correct me here, but my understanding is that Linux kernel queues up to $backlog number of connections

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

On 01/07/2016 05:55 PM, Mike Bayer wrote: but also even if you're under something like mod_wsgi, you can spawn a child process or worker thread regardless. You always have a Python interpreter running and all the things it can do. Actually you can't, reliably. Or, more precisely, you really

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

On 01/07/2016 09:49 AM, Roman Podoliaka wrote: Actually we already do that in the parent process. The parent process: 1) starts and creates a socket 2) binds the socket and calls listen() on it passing the backlog value (http://linux.die.net/man/2/listen) 3) passes the socket to the eventlet

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

On 01/07/2016 11:02 AM, Sean Dague wrote: > On 01/07/2016 09:56 AM, Brant Knudson wrote: >> >> >> On Thu, Jan 7, 2016 at 6:39 AM, Clayton O'Neill > > wrote: >> >> On Thu, Jan 7, 2016 at 2:49 AM, Roman Podoliaka >>

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

On Thu, Jan 7, 2016 at 10:44 AM, Mike Bayer wrote: > On 01/07/2016 07:39 AM, Clayton O'Neill wrote: >> On Thu, Jan 7, 2016 at 2:49 AM, Roman Podoliaka >> wrote: >>> In each child process eventlet WSGI server calls accept() in a loop to >>> get a

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

Hi Mike, Thank you for this brilliant analysis! We've been seeing such timeout errors in downstream periodically and this is the first time someone has analysed the root cause thoroughly. On Fri, Dec 18, 2015 at 10:33 PM, Mike Bayer wrote: > Hi all - > > Let me start out with

Re: [openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

On 01/06/2016 09:11 AM, Roman Podoliaka wrote: > Hi Mike, > > Thank you for this brilliant analysis! We've been seeing such timeout > errors in downstream periodically and this is the first time someone > has analysed the root cause thoroughly. > > On Fri, Dec 18, 2015 at 10:33 PM, Mike Bayer

[openstack-dev] [nova] [all] Excessively high greenlet default + excessively low connection pool defaults leads to connection pool latency, timeout errors, idle database connections / workers

Hi all - Let me start out with the assumptions I'm going from for what I want to talk about. 1. I'm looking at Nova right now, but I think similar things are going on in other Openstack apps. 2. Settings that we see in nova.conf, including: #wsgi_default_pool_size = 1000 #max_pool_size =