Public bug reported: --------------------------Enviornment------------------------------------------------------------
Controller - IP A 10.1.1.100 Compute - IP A 10.1.1.110 Release - Openstack Stein --------------------------------Error ----------------------------------------------------------- root@overcloud-controller etc]# systemctl status openstack-nova-api.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service ● openstack-nova-api.service - OpenStack Nova API Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-api.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2019-11-09 15:44:21 +07; 58min ago Main PID: 1213 (nova-api) CGroup: /system.slice/openstack-nova-api.service ├─ 1213 /usr/bin/python2 /usr/bin/nova-api ├─17185 /usr/bin/python2 /usr/bin/nova-api └─17186 /usr/bin/python2 /usr/bin/nova-api Nov 09 15:42:51 overcloud-controller systemd[1]: Starting OpenStack Nova API Server... Nov 09 15:44:11 overcloud-controller nova-api[1213]: /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:22: PkgResourcesDepreca...ately. Nov 09 15:44:11 overcloud-controller nova-api[1213]: return pkg_resources.EntryPoint.parse("x=" + s).load(False) Nov 09 15:44:21 overcloud-controller systemd[1]: Started OpenStack Nova API Server. ● openstack-nova-scheduler.service - OpenStack Nova Scheduler Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-scheduler.service; enabled; vendor preset: disabled) Active: activating (start) since Sat 2019-11-09 16:42:25 +07; 15s ago Main PID: 17141 (nova-scheduler) CGroup: /system.slice/openstack-nova-scheduler.service └─17141 /usr/bin/python2 /usr/bin/nova-scheduler Nov 09 16:42:25 overcloud-controller systemd[1]: openstack-nova-scheduler.service holdoff time over, scheduling restart. Nov 09 16:42:25 overcloud-controller systemd[1]: Stopped OpenStack Nova Scheduler Server. Nov 09 16:42:25 overcloud-controller systemd[1]: Starting OpenStack Nova Scheduler Server... ● openstack-nova-conductor.service - OpenStack Nova Conductor Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-conductor.service; enabled; vendor preset: disabled) Active: activating (start) since Sat 2019-11-09 16:42:31 +07; 9s ago Main PID: 17156 (nova-conductor) CGroup: /system.slice/openstack-nova-conductor.service └─17156 /usr/bin/python2 /usr/bin/nova-conductor Nov 09 16:42:31 overcloud-controller systemd[1]: openstack-nova-conductor.service holdoff time over, scheduling restart. Nov 09 16:42:31 overcloud-controller systemd[1]: Stopped OpenStack Nova Conductor Server. Nov 09 16:42:31 overcloud-controller systemd[1]: Starting OpenStack Nova Conductor Server... ● openstack-nova-novncproxy.service - OpenStack Nova NoVNC Proxy Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-novncproxy.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2019-11-09 15:42:51 +07; 59min ago Main PID: 1210 (nova-novncproxy) CGroup: /system.slice/openstack-nova-novncproxy.service └─1210 /usr/bin/python2 /usr/bin/nova-novncproxy --web /usr/share/novnc/ Nov 09 15:42:51 overcloud-controller systemd[1]: Started OpenStack Nova NoVNC Proxy Server. Hint: Some lines were ellipsized, use -l to show in full. -----------------------------------Expected Result ---------------------------------------- root@overcloud-controller etc]# systemctl status openstack-nova-api.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service ● openstack-nova-api.service - OpenStack Nova API Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-api.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2019-11-09 15:44:21 +07; 58min ago Main PID: 1213 (nova-api) CGroup: /system.slice/openstack-nova-api.service ├─ 1213 /usr/bin/python2 /usr/bin/nova-api ├─17193 /usr/bin/python2 /usr/bin/nova-api └─17194 /usr/bin/python2 /usr/bin/nova-api Nov 09 15:42:51 overcloud-controller systemd[1]: Starting OpenStack Nova API Server... Nov 09 15:44:11 overcloud-controller nova-api[1213]: /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:22: PkgResourcesDepreca...ately. Nov 09 15:44:11 overcloud-controller nova-api[1213]: return pkg_resources.EntryPoint.parse("x=" + s).load(False) Nov 09 15:44:21 overcloud-controller systemd[1]: Started OpenStack Nova API Server. ● openstack-nova-scheduler.service - OpenStack Nova Scheduler Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-scheduler.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2019-11-09 16:42:42 +07; 1s ago Main PID: 17141 (nova-scheduler) CGroup: /system.slice/openstack-nova-scheduler.service Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: first_packet = self.connection._read_packet() Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 683, ...packet Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: packet.check_error() Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: File "/usr/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in ..._error Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: err.raise_mysql_exception(self._data) Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise...eption Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: raise errorclass(errno, errval) Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'nova.comp...local_ Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: Removing descriptor: 5 Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: Exception AssertionError: AssertionError('do not call blocking functions f...gnored ● openstack-nova-conductor.service - OpenStack Nova Conductor Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-conductor.service; enabled; vendor preset: disabled) Active: activating (start) since Sat 2019-11-09 16:42:31 +07; 11s ago Main PID: 17156 (nova-conductor) CGroup: /system.slice/openstack-nova-conductor.service └─17156 /usr/bin/python2 /usr/bin/nova-conductor Nov 09 16:42:31 overcloud-controller systemd[1]: openstack-nova-conductor.service holdoff time over, scheduling restart. Nov 09 16:42:31 overcloud-controller systemd[1]: Stopped OpenStack Nova Conductor Server. Nov 09 16:42:31 overcloud-controller systemd[1]: Starting OpenStack Nova Conductor Server... ● openstack-nova-novncproxy.service - OpenStack Nova NoVNC Proxy Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-novncproxy.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2019-11-09 15:42:51 +07; 59min ago Main PID: 1210 (nova-novncproxy) CGroup: /system.slice/openstack-nova-novncproxy.service └─1210 /usr/bin/python2 /usr/bin/nova-novncproxy --web /usr/share/novnc/ Nov 09 15:42:51 overcloud-controller systemd[1]: Started OpenStack Nova NoVNC Proxy Server. Hint: Some lines were ellipsized, use -l to show in full. -----------------------------------NOVA SCHEDULAR LOGS------------------------------------- 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service Traceback (most recent call last): 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 796, in run_service 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service service.start() 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 166, in start 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service ctxt, self.host, self.binary) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service result = fn(cls, context, *args, **kwargs) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 334, in get_by_host_and_binary 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service host, binary) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 127, in service_get_by_host_and_binary 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return IMPL.service_get_by_host_and_binary(context, host, binary) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 251, in wrapped 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return f(context, *args, **kwargs) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 508, in service_get_by_host_and_binary 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service filter_by(binary=binary).\ 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2979, in first 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service ret = list(self[0:1]) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2771, in __getitem__ 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return list(res) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3081, in __iter__ 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return self._execute_and_instances(context) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3106, in _execute_and_instances 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service result = conn.execute(querycontext.statement, self._params) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 980, in execute 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return meth(self, multiparams, params) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return connection._execute_clauseelement(self, multiparams, params) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service distilled_params, 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service e, statement, parameters, cursor, context 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1456, in _handle_dbapi_exception 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service util.raise_from_cause(newraise, exc_info) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service reraise(type(exception), exception, tb=exc_tb, cause=cause) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service cursor, statement, parameters, context 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service cursor.execute(statement, parameters) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 170, in execute 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service result = self._query(query) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 328, in _query 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service conn.query(q) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 516, in query 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 727, in _read_query_result 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service result.read() 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1066, in read 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service first_packet = self.connection._read_packet() 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 683, in _read_packet 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service packet.check_error() 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in check_error 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service err.raise_mysql_exception(self._data) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service raise errorclass(errno, errval) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'nova.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.uuid AS services_uuid, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.disabled_reason AS services_disabled_reason, services.last_seen_up AS services_last_seen_up, services.forced_down AS services_forced_down, services.version AS services_version \nFROM services \nWHERE services.deleted = %(deleted_1)s AND services.host = %(host_1)s AND services.`binary` = %(binary_1)s \n LIMIT %(param_1)s'] [parameters: {u'host_1': 'overcloud-controller', u'param_1': 1, u'deleted_1': 0, u'binary_1': 'nova-scheduler'}] (Background on this error at: http://sqlalche.me/e/f405) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service --------------------------------------------Steps Taken-------------------------------------- Restart Rabbit mq Created NOVA DB Again, New privileges provided. Restarted nova services ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1851917 Title: Nova Schedular and Nova Conductor restart on repeat Status in OpenStack Compute (nova): New Bug description: --------------------------Enviornment------------------------------------------------------------ Controller - IP A 10.1.1.100 Compute - IP A 10.1.1.110 Release - Openstack Stein --------------------------------Error ----------------------------------------------------------- root@overcloud-controller etc]# systemctl status openstack-nova-api.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service ● openstack-nova-api.service - OpenStack Nova API Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-api.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2019-11-09 15:44:21 +07; 58min ago Main PID: 1213 (nova-api) CGroup: /system.slice/openstack-nova-api.service ├─ 1213 /usr/bin/python2 /usr/bin/nova-api ├─17185 /usr/bin/python2 /usr/bin/nova-api └─17186 /usr/bin/python2 /usr/bin/nova-api Nov 09 15:42:51 overcloud-controller systemd[1]: Starting OpenStack Nova API Server... Nov 09 15:44:11 overcloud-controller nova-api[1213]: /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:22: PkgResourcesDepreca...ately. Nov 09 15:44:11 overcloud-controller nova-api[1213]: return pkg_resources.EntryPoint.parse("x=" + s).load(False) Nov 09 15:44:21 overcloud-controller systemd[1]: Started OpenStack Nova API Server. ● openstack-nova-scheduler.service - OpenStack Nova Scheduler Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-scheduler.service; enabled; vendor preset: disabled) Active: activating (start) since Sat 2019-11-09 16:42:25 +07; 15s ago Main PID: 17141 (nova-scheduler) CGroup: /system.slice/openstack-nova-scheduler.service └─17141 /usr/bin/python2 /usr/bin/nova-scheduler Nov 09 16:42:25 overcloud-controller systemd[1]: openstack-nova-scheduler.service holdoff time over, scheduling restart. Nov 09 16:42:25 overcloud-controller systemd[1]: Stopped OpenStack Nova Scheduler Server. Nov 09 16:42:25 overcloud-controller systemd[1]: Starting OpenStack Nova Scheduler Server... ● openstack-nova-conductor.service - OpenStack Nova Conductor Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-conductor.service; enabled; vendor preset: disabled) Active: activating (start) since Sat 2019-11-09 16:42:31 +07; 9s ago Main PID: 17156 (nova-conductor) CGroup: /system.slice/openstack-nova-conductor.service └─17156 /usr/bin/python2 /usr/bin/nova-conductor Nov 09 16:42:31 overcloud-controller systemd[1]: openstack-nova-conductor.service holdoff time over, scheduling restart. Nov 09 16:42:31 overcloud-controller systemd[1]: Stopped OpenStack Nova Conductor Server. Nov 09 16:42:31 overcloud-controller systemd[1]: Starting OpenStack Nova Conductor Server... ● openstack-nova-novncproxy.service - OpenStack Nova NoVNC Proxy Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-novncproxy.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2019-11-09 15:42:51 +07; 59min ago Main PID: 1210 (nova-novncproxy) CGroup: /system.slice/openstack-nova-novncproxy.service └─1210 /usr/bin/python2 /usr/bin/nova-novncproxy --web /usr/share/novnc/ Nov 09 15:42:51 overcloud-controller systemd[1]: Started OpenStack Nova NoVNC Proxy Server. Hint: Some lines were ellipsized, use -l to show in full. -----------------------------------Expected Result ---------------------------------------- root@overcloud-controller etc]# systemctl status openstack-nova-api.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service ● openstack-nova-api.service - OpenStack Nova API Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-api.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2019-11-09 15:44:21 +07; 58min ago Main PID: 1213 (nova-api) CGroup: /system.slice/openstack-nova-api.service ├─ 1213 /usr/bin/python2 /usr/bin/nova-api ├─17193 /usr/bin/python2 /usr/bin/nova-api └─17194 /usr/bin/python2 /usr/bin/nova-api Nov 09 15:42:51 overcloud-controller systemd[1]: Starting OpenStack Nova API Server... Nov 09 15:44:11 overcloud-controller nova-api[1213]: /usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py:22: PkgResourcesDepreca...ately. Nov 09 15:44:11 overcloud-controller nova-api[1213]: return pkg_resources.EntryPoint.parse("x=" + s).load(False) Nov 09 15:44:21 overcloud-controller systemd[1]: Started OpenStack Nova API Server. ● openstack-nova-scheduler.service - OpenStack Nova Scheduler Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-scheduler.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2019-11-09 16:42:42 +07; 1s ago Main PID: 17141 (nova-scheduler) CGroup: /system.slice/openstack-nova-scheduler.service Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: first_packet = self.connection._read_packet() Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 683, ...packet Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: packet.check_error() Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: File "/usr/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in ..._error Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: err.raise_mysql_exception(self._data) Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise...eption Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: raise errorclass(errno, errval) Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'nova.comp...local_ Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: Removing descriptor: 5 Nov 09 16:42:42 overcloud-controller nova-scheduler[17141]: Exception AssertionError: AssertionError('do not call blocking functions f...gnored ● openstack-nova-conductor.service - OpenStack Nova Conductor Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-conductor.service; enabled; vendor preset: disabled) Active: activating (start) since Sat 2019-11-09 16:42:31 +07; 11s ago Main PID: 17156 (nova-conductor) CGroup: /system.slice/openstack-nova-conductor.service └─17156 /usr/bin/python2 /usr/bin/nova-conductor Nov 09 16:42:31 overcloud-controller systemd[1]: openstack-nova-conductor.service holdoff time over, scheduling restart. Nov 09 16:42:31 overcloud-controller systemd[1]: Stopped OpenStack Nova Conductor Server. Nov 09 16:42:31 overcloud-controller systemd[1]: Starting OpenStack Nova Conductor Server... ● openstack-nova-novncproxy.service - OpenStack Nova NoVNC Proxy Server Loaded: loaded (/usr/lib/systemd/system/openstack-nova-novncproxy.service; enabled; vendor preset: disabled) Active: active (running) since Sat 2019-11-09 15:42:51 +07; 59min ago Main PID: 1210 (nova-novncproxy) CGroup: /system.slice/openstack-nova-novncproxy.service └─1210 /usr/bin/python2 /usr/bin/nova-novncproxy --web /usr/share/novnc/ Nov 09 15:42:51 overcloud-controller systemd[1]: Started OpenStack Nova NoVNC Proxy Server. Hint: Some lines were ellipsized, use -l to show in full. -----------------------------------NOVA SCHEDULAR LOGS------------------------------------- 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service Traceback (most recent call last): 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 796, in run_service 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service service.start() 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 166, in start 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service ctxt, self.host, self.binary) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service result = fn(cls, context, *args, **kwargs) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 334, in get_by_host_and_binary 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service host, binary) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 127, in service_get_by_host_and_binary 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return IMPL.service_get_by_host_and_binary(context, host, binary) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 251, in wrapped 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return f(context, *args, **kwargs) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 508, in service_get_by_host_and_binary 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service filter_by(binary=binary).\ 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2979, in first 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service ret = list(self[0:1]) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2771, in __getitem__ 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return list(res) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3081, in __iter__ 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return self._execute_and_instances(context) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 3106, in _execute_and_instances 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service result = conn.execute(querycontext.statement, self._params) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 980, in execute 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return meth(self, multiparams, params) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service return connection._execute_clauseelement(self, multiparams, params) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service distilled_params, 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service e, statement, parameters, cursor, context 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1456, in _handle_dbapi_exception 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service util.raise_from_cause(newraise, exc_info) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service reraise(type(exception), exception, tb=exc_tb, cause=cause) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service cursor, statement, parameters, context 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service cursor.execute(statement, parameters) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 170, in execute 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service result = self._query(query) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 328, in _query 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service conn.query(q) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 516, in query 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service self._affected_rows = self._read_query_result(unbuffered=unbuffered) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 727, in _read_query_result 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service result.read() 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1066, in read 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service first_packet = self.connection._read_packet() 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 683, in _read_packet 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service packet.check_error() 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in check_error 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service err.raise_mysql_exception(self._data) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service raise errorclass(errno, errval) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'nova.services' doesn't exist") [SQL: u'SELECT services.created_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.uuid AS services_uuid, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services.disabled AS services_disabled, services.disabled_reason AS services_disabled_reason, services.last_seen_up AS services_last_seen_up, services.forced_down AS services_forced_down, services.version AS services_version \nFROM services \nWHERE services.deleted = %(deleted_1)s AND services.host = %(host_1)s AND services.`binary` = %(binary_1)s \n LIMIT %(param_1)s'] [parameters: {u'host_1': 'overcloud-controller', u'param_1': 1, u'deleted_1': 0, u'binary_1': 'nova-scheduler'}] (Background on this error at: http://sqlalche.me/e/f405) 2019-11-09 16:44:29.120 17464 ERROR oslo_service.service --------------------------------------------Steps Taken-------------------------------------- Restart Rabbit mq Created NOVA DB Again, New privileges provided. Restarted nova services To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1851917/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp