Hi, when using Mysql behind Haproxy, i have a problem on reboot when some nova services start after Haproxy service, but before Mysql service. These service failed: (i re-checked for sure in /var/log/boot.log) * Starting Nova cert [fail] * Starting Nova conductor [fail] * Starting Nova scheduler [fail] * Starting Cinder scheduler server [fail]
I must login to server and re-start these services manually. When check log of Nova-cert, I saw: *2013-07-12 15:20:05.020 2490 CRITICAL nova [-] (OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communic ation packet', system error: 0") None None* 2013-07-12 15:20:05.020 2490 TRACE nova Traceback (most recent call last): 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/bin/nova-cert", line 51, in <module> 2013-07-12 15:20:05.020 2490 TRACE nova service.wait() 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 689, in wait 2013-07-12 15:20:05.020 2490 TRACE nova _launcher.wait() 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 209, in wait 2013-07-12 15:20:05.020 2490 TRACE nova super(ServiceLauncher, self).wait() 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 179, in wait 2013-07-12 15:20:05.020 2490 TRACE nova service.wait() 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wait 2013-07-12 15:20:05.020 2490 TRACE nova return self._exit_event.wait() 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait 2013-07-12 15:20:05.020 2490 TRACE nova return hubs.get_hub().switch() 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch 2013-07-12 15:20:05.020 2490 TRACE nova return self.greenlet.switch() 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main 2013-07-12 15:20:05.020 2490 TRACE nova result = function(*args, **kwargs) 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 147, in run_server 2013-07-12 15:20:05.020 2490 TRACE nova server.start() 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 434, in start 2013-07-12 15:20:05.020 2490 TRACE nova self.host, self.binary) 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/conductor/api.py", line 261, in service_get_by_a rgs 2013-07-12 15:20:05.020 2490 TRACE nova binary=binary) 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 1348, in wrapper 2013-07-12 15:20:05.020 2490 TRACE nova return func(*args, **kwargs) 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/common.py", line 424, in in ner 2013-07-12 15:20:05.020 2490 TRACE nova return catch_client_exception(exceptions, func, *args, **kwargs) 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/common.py", line 407, in ca tch_client_exception 2013-07-12 15:20:05.020 2490 TRACE nova return func(*args, **kwargs) 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 325, in service_get_ all_by 2013-07-12 15:20:05.020 2490 TRACE nova result = self.db.service_get_by_args(context, host, binary) 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 155, in service_get_by_args 2013-07-12 15:20:05.020 2490 TRACE nova return IMPL.service_get_by_args(context, host, binary) 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 96, in wrapper 2013-07-12 15:20:05.020 2490 TRACE nova return f(*args, **kwargs) 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 409, in service_get_by_args 2013-07-12 15:20:05.020 2490 TRACE nova result = model_query(context, models.Service).\ 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 177, in model_query 2013-07-12 15:20:05.020 2490 TRACE nova session = kwargs.get('session') or get_session() 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/openstack/common/db/sqlalchemy/session.py", line 325, in get_session 2013-07-12 15:20:05.020 2490 TRACE nova engine = get_engine() 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/openstack/common/db/sqlalchemy/session.py", line 446, in get_engine 2013-07-12 15:20:05.020 2490 TRACE nova _ENGINE = create_engine(CONF.sql_connection) 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/openstack/common/db/sqlalchemy/session.py", line 562, in create_engine 2013-07-12 15:20:05.020 2490 TRACE nova engine.connect() 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2471, in connect 2013-07-12 15:20:05.020 2490 TRACE nova return self._connection_cls(self, **kwargs) 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 878, in __init__ 2013-07-12 15:20:05.020 2490 TRACE nova self.__connection = connection or engine.raw_connection() 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2557, in raw_connection 2013-07-12 15:20:05.020 2490 TRACE nova return self.pool.unique_connection() 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 184, in unique_connection 2013-07-12 15:20:05.020 2490 TRACE nova return _ConnectionFairy(self).checkout() 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 401, in __init__ 2013-07-12 15:20:05.020 2490 TRACE nova rec = self._connection_record = pool._do_get() 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 746, in _do_get 2013-07-12 15:20:05.020 2490 TRACE nova con = self._create_connection() 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 189, in _create_connection 2013-07-12 15:20:05.020 2490 TRACE nova return _ConnectionRecord(self) 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 282, in __init__ 2013-07-12 15:20:05.020 2490 TRACE nova self.connection = self.__connect() 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 344, in __connect 2013-07-12 15:20:05.020 2490 TRACE nova connection = self.__pool._creator() 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 80, in connect 2013-07-12 15:20:05.020 2490 TRACE nova return dialect.connect(*cargs, **cparams) 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 281, in connect 2013-07-12 15:20:05.020 2490 TRACE nova return self.dbapi.connect(*cargs, **cparams) 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect 2013-07-12 15:20:05.020 2490 TRACE nova return Connection(*args, **kwargs) 2013-07-12 15:20:05.020 2490 TRACE nova File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__ 2013-07-12 15:20:05.020 2490 TRACE nova super(Connection, self).__init__(*args, **kwargs2) *2013-07-12 15:20:05.020 2490 TRACE nova OperationalError: (OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0") None None * Do you know a quick fix for this problem? (I also send this email to report the problem) Thanks you! PS: I'm runing mysql, haproxy, nova-* services, cinder-* services on the same server, using Ubuntu 12.04.
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev