On 9/18/2014 12:35 AM, Morgan Fainberg wrote:
-----Original Message-----
From: Dean Troyer <dtro...@gmail.com>
Reply: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>>
Date: September 17, 2014 at 21:21:47
To: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>>
Subject:  Re: [openstack-dev] PostgreSQL jobs slow in the gate


Clark Boylan Wrotr :

It appears that keystone eventlet is the source of the slowness in this
job. With keystone eventlet all of the devstack jobs are slower and with
keystone wsgi all of the jobs are quicker. Probably need to collect a
bit more data but this doesn't look good for keystone eventlet.


On Wed, Sep 17, 2014 at 11:02 PM, Morgan Fainberg <
morgan.fainb...@gmail.com> wrote:

I've kicked off a test[1] as well to check into some tunable options
(eventlet workers) for improving keystone eventlet performance. I'll circle
back with the infra team once we have a little more data on both fronts.
The Keystone team will use this data to figure out the best way to approach
this issue.


Brant submitted https://review.openstack.org/#/c/121384/ to up the Keystone
workers when API_WORKERS is set.

I submitted https://review.openstack.org/#/c/122013/ to set a scaling
default for API_WORKERS based on the available CPUs ((nproc+1)/2). There
is a summary in that commit message of the current reviews addressing the
workers in various projects.

I think it has become clear that DevStack needs to set a default for most
services that are currently either too big (nproc) or too small (Keystone
at 1). Of course, moving things to mod_wsgi moots all of that, but it'll
be a while before everything moves.

dt

--

Dean Troyer
dtro...@gmail.com

Dean,

We should probably look at increasing the default number of workers as well in 
the Keystone configuration (rather than just devstack). It looks like, with 
limited datasets, we are seeing a real improvement with Keystone and 4 workers 
(from my previously mentioned quick test). Thanks for the extra data points. 
This helps to confirm the issue is Keystone under eventlet.

Cheers,
Morgan

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


I pointed this out in Brant's devstack patch but we had a product team internally bring up this same point in Icehouse, they were really limited due to the eventlet workers issue in Keystone and once we provided the option (backported) it increased their throughput by 20%. We've been running with that in our internal Tempest runs (setting workers equal to number of CPUs / 2) and so far so good.

--

Thanks,

Matt Riedemann


_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to