Public bug reported: nova-conductor service to access the DB service to delay 7230.58s
The more information is： 2023-03-02 18:39:53.726 22 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status. 2023-03-02 18:39:53.727 19 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status. 2023-03-02 18:39:53.729 23 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status. 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db [-] Unexpected error while reporting service status: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 812, in _checkout 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db raise exc.InvalidatePoolError() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db sqlalchemy.exc.InvalidatePoolError: () 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db During handling of the above exception, another exception occurred: 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2285, in _wrap_pool_connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return fn() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 363, in connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 842, in _checkout 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db fairy._connection_record._checkin_failed(err) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db exc_value, with_traceback=exc_tb, 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db raise exception 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 838, in _checkout 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db fairy._connection_record.get_connection() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 606, in get_connection 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db self.__connect() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 657, in __connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db pool.logger.debug("Error on connect(): %s", e) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db exc_value, with_traceback=exc_tb, 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db raise exception 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 652, in __connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 490, in connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/__init__.py", line 94, in Connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 325, in __init__ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db self.connect() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 598, in connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db self._get_server_information() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 975, in _get_server_information 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db packet = self._read_packet() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 657, in _read_packet 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db packet_header = self._read_bytes(4) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 707, in _read_bytes 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db CR.CR_SERVER_LOST, "Lost connection to MySQL server during query") 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query') 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db The above exception was the direct cause of the following exception: 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/servicegroup/drivers/db.py", line 93, in _report_state 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db service.service_ref.save() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/service.py", line 436, in save 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/api.py", line 182, in service_update 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db self.force_reraise() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/usr/local/lib/python3.6/site-packages/six.py", line 703, in reraise 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db raise value 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 223, in wrapped 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 552, in service_update 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 238, in wrapped 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 413, in service_get 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db result = query.first() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3300, in first 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3078, in __getitem__ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return list(res) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3405, in __iter__ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3427, in _execute_and_instances 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db querycontext, self._connection_from_session, close_with_result=True 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3442, in _get_bind_args 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db mapper=self._bind_mapper(), clause=querycontext.statement, **kw 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3420, in _connection_from_session 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 1133, in connection 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db execution_options=execution_options, 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 1139, in _connection_for_bind 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db engine, execution_options 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 432, in _connection_for_bind 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db conn = bind._contextual_connect() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2251, in _contextual_connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2289, in _wrap_pool_connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db e, dialect, self 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1552, in _handle_dbapi_exception_noconnection 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db util.raise_(newraise, with_traceback=exc_info[2], from_=e) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db raise exception 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2285, in _wrap_pool_connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return fn() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 363, in connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 842, in _checkout 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db fairy._connection_record._checkin_failed(err) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db exc_value, with_traceback=exc_tb, 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db raise exception 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 838, in _checkout 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db fairy._connection_record.get_connection() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 606, in get_connection 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db self.__connect() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 657, in __connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db pool.logger.debug("Error on connect(): %s", e) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db exc_value, with_traceback=exc_tb, 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db raise exception 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 652, in __connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 490, in connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/__init__.py", line 94, in Connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 325, in __init__ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db self.connect() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 598, in connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db self._get_server_information() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 975, in _get_server_information 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db packet = self._read_packet() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 657, in _read_packet 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db packet_header = self._read_bytes(4) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 707, in _read_bytes 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db CR.CR_SERVER_LOST, "Lost connection to MySQL server during query") 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db (Background on this error at: http://sqlalche.me/e/e3q8) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db 2023-03-02 18:39:53.737 20 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 7230.58 sec 2023-03-02 18:39:53.741 21 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status. 2023-03-02 18:39:53.745 20 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2009263 Title: 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 7230.58 sec Status in OpenStack Compute (nova): New Bug description: nova-conductor service to access the DB service to delay 7230.58s The more information is： 2023-03-02 18:39:53.726 22 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status. 2023-03-02 18:39:53.727 19 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status. 2023-03-02 18:39:53.729 23 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status. 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db [-] Unexpected error while reporting service status: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 812, in _checkout 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db raise exc.InvalidatePoolError() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db sqlalchemy.exc.InvalidatePoolError: () 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db During handling of the above exception, another exception occurred: 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2285, in _wrap_pool_connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return fn() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 363, in connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 842, in _checkout 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db fairy._connection_record._checkin_failed(err) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db exc_value, with_traceback=exc_tb, 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db raise exception 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 838, in _checkout 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db fairy._connection_record.get_connection() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 606, in get_connection 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db self.__connect() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 657, in __connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db pool.logger.debug("Error on connect(): %s", e) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db exc_value, with_traceback=exc_tb, 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db raise exception 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 652, in __connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 490, in connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/__init__.py", line 94, in Connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 325, in __init__ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db self.connect() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 598, in connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db self._get_server_information() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 975, in _get_server_information 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db packet = self._read_packet() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 657, in _read_packet 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db packet_header = self._read_bytes(4) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 707, in _read_bytes 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db CR.CR_SERVER_LOST, "Lost connection to MySQL server during query") 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query') 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db The above exception was the direct cause of the following exception: 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/servicegroup/drivers/db.py", line 93, in _report_state 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db service.service_ref.save() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/service.py", line 436, in save 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/api.py", line 182, in service_update 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db self.force_reraise() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/usr/local/lib/python3.6/site-packages/six.py", line 703, in reraise 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db raise value 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 223, in wrapped 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 552, in service_update 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 238, in wrapped 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 413, in service_get 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db result = query.first() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3300, in first 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3078, in __getitem__ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return list(res) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3405, in __iter__ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3427, in _execute_and_instances 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db querycontext, self._connection_from_session, close_with_result=True 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3442, in _get_bind_args 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db mapper=self._bind_mapper(), clause=querycontext.statement, **kw 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3420, in _connection_from_session 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 1133, in connection 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db execution_options=execution_options, 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 1139, in _connection_for_bind 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db engine, execution_options 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 432, in _connection_for_bind 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db conn = bind._contextual_connect() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2251, in _contextual_connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2289, in _wrap_pool_connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db e, dialect, self 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1552, in _handle_dbapi_exception_noconnection 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db util.raise_(newraise, with_traceback=exc_info[2], from_=e) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db raise exception 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2285, in _wrap_pool_connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return fn() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 363, in connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 842, in _checkout 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db fairy._connection_record._checkin_failed(err) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db exc_value, with_traceback=exc_tb, 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db raise exception 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 838, in _checkout 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db fairy._connection_record.get_connection() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 606, in get_connection 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db self.__connect() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 657, in __connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db pool.logger.debug("Error on connect(): %s", e) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db exc_value, with_traceback=exc_tb, 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db raise exception 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 652, in __connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 490, in connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/__init__.py", line 94, in Connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 325, in __init__ 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db self.connect() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 598, in connect 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db self._get_server_information() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 975, in _get_server_information 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db packet = self._read_packet() 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 657, in _read_packet 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db packet_header = self._read_bytes(4) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 707, in _read_bytes 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db CR.CR_SERVER_LOST, "Lost connection to MySQL server during query") 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db (Background on this error at: http://sqlalche.me/e/e3q8) 2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db 2023-03-02 18:39:53.737 20 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 7230.58 sec 2023-03-02 18:39:53.741 21 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status. 2023-03-02 18:39:53.745 20 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2009263/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
