** Changed in: nova
Status: Fix Committed => Fix Released
** Changed in: nova
Milestone: None => juno-3
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1342257
Title:
Nova cells can die unexpectedly on boot due to db failure
Status in OpenStack Compute (Nova):
Fix Released
Bug description:
We have seen a crash in the cells booting process with the following
traceback:
2014-07-15 01:00:07.688 3070 CRITICAL nova
[req-badc12a2-4ad9-4209-bcd4-f2429e134820 None] DBError: (1030, 'Got error 28
from storage engine')
2014-07-15 01:00:07.688 3070 TRACE nova Traceback (most recent call last):
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/current/nova/bin/nova-cells", line 13, in <module>
2014-07-15 01:00:07.688 3070 TRACE nova sys.exit(main())
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/cmd/cells.py", line
45, in main
2014-07-15 01:00:07.688 3070 TRACE nova manager=CONF.cells.manager)
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/service.py", line
275, in create
2014-07-15 01:00:07.688 3070 TRACE nova db_allowed=db_allowed)
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/service.py", line
148, in __init__
2014-07-15 01:00:07.688 3070 TRACE nova self.manager =
manager_class(host=self.host, *args, **kwargs)
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/cells/manager.py",
line 90, in __init__
2014-07-15 01:00:07.688 3070 TRACE nova self.state_manager =
cell_state_manager()
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/cells/state.py",
line 161, in __new__
2014-07-15 01:00:07.688 3070 TRACE nova return
CellStateManagerDB(cell_state_cls)
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/cells/state.py",
line 174, in __init__
2014-07-15 01:00:07.688 3070 TRACE nova self._cell_data_sync(force=True)
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/openstack/common/lockutils.py",
line 325, in inner
2014-07-15 01:00:07.688 3070 TRACE nova return f(*args, **kwargs)
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/cells/state.py",
line 436, in _cell_data_sync
2014-07-15 01:00:07.688 3070 TRACE nova db_cells =
self.db.cell_get_all(ctxt)
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/db/api.py", line
1599, in cell_get_all
2014-07-15 01:00:07.688 3070 TRACE nova return IMPL.cell_get_all(context)
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/db/api.py", line
93, in __getattr__
2014-07-15 01:00:07.688 3070 TRACE nova return getattr(self._db_api, key)
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/db/api.py", line
85, in _db_api
2014-07-15 01:00:07.688 3070 TRACE nova backend_mapping=_BACKEND_MAPPING)
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/openstack/common/db/api.py",
line 128, in __init__
2014-07-15 01:00:07.688 3070 TRACE nova self._load_backend()
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/openstack/common/db/api.py",
line 143, in _load_backend
2014-07-15 01:00:07.688 3070 TRACE nova self._backend =
backend_mod.get_backend()
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/db/mysqldb/api.py",
line 42, in get_backend
2014-07-15 01:00:07.688 3070 TRACE nova return API()
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/db/mysqldb/api.py",
line 75, in __init__
2014-07-15 01:00:07.688 3070 TRACE nova self._launch_monitor()
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/db/mysqldb/api.py",
line 89, in _launch_monitor
2014-07-15 01:00:07.688 3070 TRACE nova self._check_schema()
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/db/mysqldb/api.py",
line 56, in inner
2014-07-15 01:00:07.688 3070 TRACE nova result = f(*args, **kwargs)
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/db/mysqldb/api.py",
line 81, in _check_schema
2014-07-15 01:00:07.688 3070 TRACE nova schema = conn.get_schema()
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/db/mysqldb/connection.py",
line 186, in get_schema
2014-07-15 01:00:07.688 3070 TRACE nova tables = self._get_tables()
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/db/mysqldb/connection.py",
line 169, in _get_tables
2014-07-15 01:00:07.688 3070 TRACE nova columns =
self._get_columns(table_name)
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/db/mysqldb/connection.py",
line 159, in _get_columns
2014-07-15 01:00:07.688 3070 TRACE nova cursor = self.execute('DESCRIBE
%s' % name)
2014-07-15 01:00:07.688 3070 TRACE nova File
"/opt/rackstack/863.0/nova/lib/python2.6/site-packages/nova/db/mysqldb/connection.py",
line 79, in inner
2014-07-15 01:00:07.688 3070 TRACE nova raise db_exc.DBError(e)
2014-07-15 01:00:07.688 3070 TRACE nova DBError: (1030, 'Got error 28 from
storage engine')
2014-07-15 01:00:07.688 3070 TRACE nova
Since this is a DB issue it seems the process should at the very least
retry.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1342257/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
