Excerpts from Sean Dague's message of 2015-06-18 12:53:16 -0700: > On 06/18/2015 03:47 PM, Mike Bayer wrote: > > > > > > On 6/18/15 1:28 PM, Matt Riedemann wrote: > >> > >> > >>> > >> > >> It's not only neutron, I saw some pymysql failures in nova the other > >> day for 'too many connections' or some such related error. > > > > "too many connections" is an error raised by MySQL when more connections > > are attempting to connect than the max_connections setting. It defaults > > to 100 but it is often set at 1024 in distributions. An Openstack > > application that uses oslo.db will go through oslo.db's create_engine() > > routine which defaults the max number of connections per python process > > to 15, and when the process is idle, this drops to 5. > > > > Which means, there have to be over 60 python processes all hitting the > > DB at the max to get that error, if max_connections is at 1024. Or > > the connection pool is totally broken (e.g. with eventlet monkeypatching > > perhaps), which would be a critical issue for me to fix. > > > > So I'd love to see how the "too many connections" error actually > > happens. Please share whatever you have on that. > > I believe the distro defaults in Ubuntu are the mysql defaults, which is > 151 - https://dev.mysql.com/doc/refman/5.5/en/too-many-connections.html > > If we're allowing 15 connections per python process, and the APIs are > running in a worker mode, which means 8 workers per API server, yes, > that's going to add up really quick. > > Perhaps under these scenarios we need to raise this to 1024 in devstack?
Indeed, mysqld needs tuning for better concurrency. I submitted this which should help avoid the low connection count and also will unserialize read/writes on a single table which should lead to tests doing multiple operations asynchronously finishing a bit quicker. https://review.openstack.org/193295 Note that we MIGHT want to raise the innodb_buffer_pool_size. The default since 5.5 has been 128MB (It used to be _8MB_). However, I imagine memory can get tight on devstack nodes and laptops, so it's also perfectly valid to keep it at 128MB and just let operations go to disk more often. __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
