On 02/20/2015 03:10 PM, Sean Lynn wrote:
Sorry, that wasn't exactly clear and my memory was a bit foggy, Jay.
max_connections possibly paired with open_files_limit
https://dev.mysql.com/doc/refman/5.6/en/too-many-connections.html
No worries :) I'd be surprised if you were hitting open_files_limit, as
that typically is seen when the schemas have hundreds or thousands of
tables in them. More likely you hit issues with max_connections, which,
for some reason I cannot fathom, is set to 151 by default in MySQL. :(
I recommend using 2000 or more as the value for max_connections for any
database server used in production deployments.
Best,
-jay
On 02/20/15 10:20, Jay Pipes wrote:
On 02/20/2015 10:39 AM, Sean Lynn wrote:
We finished upgrading to Juno about the time you guys did. Just checked
logs across all environments since the time of the Juno upgrade and I'm
*not* seeing the same errors.
For comparison here's what we have (mostly out-of-the-box):
api_workers and rpc_workers = 32
metadata_workers = 32
url_timeout = 30
oslo version = 1.4.1
Any related errors in the Neutron logs?
Couple seemingly dumb questions related to system limits, but:
1. Could this be a file descriptors limit for the neutron processes?
2. Recently we ran into the file descriptors limit in MySQL which
showed up with "sporadic but frequent errors" in Neutron. Under
load is your MySQL fd limit being hit?
I'm not familiar with file descriptor limit issues with MySQL. Do you
mean max_connections issues?
Best,
-jay
3. Similar limit question for RabbitMQ.
Let me know if you want any more comparison info.
Sean Lynn
Time Warner Cable, Inc.
----------------------
Kris G. Lindgren:
"
After our icehouse -> juno upgrade we are noticing sporadic but
frequent errors from nova-metadata when trying to serve metadata
requests. The error is the following:
[req-594325c6-44ed-465c-a8e4-bd5a8e5dbdcb None] Failed to get
metadata for ip: x.x.x.x 2015-02-19 12:16:45.903 25007 TRACE
nova.api.metadata.handler Traceback (most recent call last):
2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File
/usr/lib/python2.6/site-packages/nova/api/metadata/handler.py, line
150, in _handle_remote_ip_request 2015-02-19 12:16:45.903 25007 TRACE
nova.api.metadata.handler meta_data =
self.get_metadata_by_remote_address(remote_address) 2015-02-19
12:16:45.903 25007 TRACE nova.api.metadata.handler File
/usr/lib/python2.6/site-packages/nova/api/metadata/handler.py, line
82, in get_metadata_by_remote_address 2015-02-19 12:16:45.903 25007
TRACE nova.api.metadata.handler data =
base.get_metadata_by_address(self.conductor_api, address)
...
We have increased the number of neutron workers (40 API and 40
RPC), the Neutron url_timeout interval in nova from 30 to 60 seconds.
We are only seeing this issue in production or pre-prod environments
are fine.
Is anyone else noticing this or frequent read timeouts when
talking to neutron? Have you found a solution? What have you tried?
I am thinking of updating a bunch of the oslo (db, messaging, ect
ect) packages to the latest versions to see if things get better.
"
_______________________________________________
OpenStack-operators mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
_______________________________________________
OpenStack-operators mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
_______________________________________________
OpenStack-operators mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
_______________________________________________
OpenStack-operators mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
