Clark Boylan <cboy...@sapwetik.org> Wrotr : > On Wed, Sep 17, 2014, at 06:48 PM, Clark Boylan wrote: > > On Wed, Sep 17, 2014, at 06:37 PM, Matt Riedemann wrote: > > > > > > > > > On 9/17/2014 7:59 PM, Ian Wienand wrote: > > > > On 09/18/2014 09:49 AM, Clark Boylan wrote: > > > >> Recent sampling of test run times shows that our tempest jobs run > > > >> against clouds using PostgreSQL are significantly slower than jobs > run > > > >> against clouds using MySQL. > > > > > > > > FYI There is a possibly relevant review out for max_connections > limits > > > > [1], although it seems to have some issues with shmem usage > > > > > > > > -i > > > > > > > > [1] https://review.openstack.org/#/c/121952/ > > > > > > > > _______________________________________________ > > > > OpenStack-dev mailing list > > > > OpenStack-dev@lists.openstack.org <javascript:;> > > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > > > That's a backport of a fix from master where we were hitting fatal > > > errors due to too many DB connections which was brought on by the > > > changes to cinder and glance to run as many workers as there were CPUs > > > available. So I don't think it probably plays here... > > > > > > The errors pointed out in another part of the thread have been around > > > for awhile, I think they are due to negative tests where we're hitting > > > unique constraints because of the negative tests, so they are expected. > > > > > > We should also note that the postgresql jobs run with the nova metadata > > > API service, I'm not sure how much of a factor that would have here. > > > > > > Is there anything else unique about those jobs from the MySQL ones? > > > > > Good question. There are apparently other differences. The postgres job > > runs Keystone under eventlet instead of via apache mod_wsgi. It also > > sets FORCE_CONFIGDRIVE=False instead of always. And the final difference > > I can find is the one you point out, nova api metadata service is run as > > an independent thing. > > > > Could these things be related? It would be relatively simple to push a > > change or two to devstack-gate to test this but there are enough options > > here that I probably won't do that until we think at least one of these > > options is at fault. > I am starting to feel bad that I picked on PostgreSQL and completely > forgot that there were other items in play here. I went ahead and > uploaded [0] to run all devstack jobs without keystone wsgi services > (eventlet) and [1] to run all devstack job with keystone wsgi services > and the initial results are pretty telling. > > It appears that keystone eventlet is the source of the slowness in this > job. With keystone eventlet all of the devstack jobs are slower and with > keystone wsgi all of the jobs are quicker. Probably need to collect a > bit more data but this doesn't look good for keystone eventlet. > > Thank you Matt for pointing me in this direction. > > [0] https://review.openstack.org/#/c/122299/ > [1] https://review.openstack.org/#/c/122300/ > > > > > > -- > > > > > > Thanks, > > > > > > Matt Riedemann > > > >
I've kicked off a test[1] as well to check into some tunable options (eventlet workers) for improving keystone eventlet performance. I'll circle back with the infra team once we have a little more data on both fronts. The Keystone team will use this data to figure out the best way to approach this issue. --Morgan [1] https://review.openstack.org/#/c/122308/
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev