On 04/17/2015 05:26 AM, Qiming Teng wrote:
On Thu, Apr 16, 2015 at 01:27:51PM -0400, Jay Pipes wrote:
On 04/16/2015 09:54 AM, Sean Dague wrote:
On 04/16/2015 05:20 PM, Qiming Teng wrote:
Wondering if there is something misconfigured in my devstack
environment, which was reinstalled on RHEL7 about 10 days ago.
I'm often running into mysql connections problem as shown below:
$ mysql
ERROR 1040 (HY000): Too many connections
When I try dump the mysql connection list, I'm getting the followng
result after a 'systemctl restart mariadb.service':
$ mysqladmin processlist | grep nova | wc -l
125
Most of the connections are at Sleep status:
$ mysqladmin processlist | grep nova | grep Sleep | wc -l
123
As for the workload, I'm currently only running two VMs in a multi-host
devstack environment.
So, my questions:
- Why do we have so many mysql connections from nova?
- Is it possible this is caused some misconfigurations?
- 125 connections in such a toy setup is insane, any hints on nailing
down the connections to the responsible nova components?
Thanks.
Regards,
Qiming
No, that's about right. It's 1 connection per worker. By default most
daemons start 1 worker per processor. Each OpenStack service has a bunch
of daemons. It all adds up pretty quick.
And just to add to what Sean says above, there's nothing inherently
wrong with sleeping connections to MySQL. What *is* wrong, however,
is that the default max_connections setting in my.cnf is 150. :( I
frequently recommend upping that to 2000 or more on any modern
hardware or decent sized VM.
Best,
-jay
Thanks, guys. So the key takeaway for me is:
- 100~200 mysql connections is not big problem, provided those
connections are sleeping;
Yep, correct, for MySQL.
- Tuning mysql to support larger number of user connections is a
must;
Yes, because the default is a measly 150 max_connections.
- Number of mysql connections is not proportional to the number of
VMs, it is more related to the number of cores, number of workers
etc.
Yes. nova-compute workers (which are on each compute host that houses
VMs) actually do not contact the database directly. Instead, they
communicate with the nova-conductor service, which itself communicates
with the database. The nova-conductor service by default will maintain a
number of processes equal to the number of cores on the VM or baremetal
machine. Each of these processes will keep a pool of connections to MySQL.
In addition to nova-conductor, there are various other Nova services
that also make connections to MySQL.
Best,
-jay
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
