Re: [openstack-dev] [nova] unit test migration failure specific to MySQL/MariaDB - 'uuid': used in a foreign key constraint 'block_device_mapping_instance_uuid_fkey'
OK so it’s looking like sql_mode=‘TRADITIONAL’ that allows it to work. So that
is most of it. My MariaDB has no default sql_mode but oslo.db should be
setting this, but in any case this seems more like a local oslo.db connection
type of thing that I can track down myself, so most of the mystery solved! (at
least the part that I didn’t feel like getting into….which I did anyway).
Mike Bayer mba...@redhat.com wrote:
working with sdague on IRC, the first thing I’m seeing is that my MariaDB
server is disallowing a change in column that is UNIQUE and has an FK
pointing to it, and this is distinctly different from a straight up MySQL
server (see below).
http://paste.openstack.org/raw/155896/
old school MySQL:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 4840
Server version: 5.6.15 Homebrew
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql create table foo (id int, blah int, primary key (id), unique key
(blah)) engine=InnoDB;
Query OK, 0 rows affected (0.01 sec)
mysql create table bar(id int, blah_fk int, primary key (id), foreign key
(blah_fk) references foo(blah)) engine=InnoDB;
Query OK, 0 rows affected (0.01 sec)
mysql alter table foo change column blah blah int not null;
Query OK, 0 rows affected (0.02 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql
MariaDB 10:
MariaDB [test] create table foo (id int, blah int, primary key (id), unique
key (blah));
Query OK, 0 rows affected (0.09 sec)
MariaDB [test] create table bar(id int, blah_fk int, primary key (id),
foreign key (blah_fk) references foo(blah));
Query OK, 0 rows affected (0.12 sec)
MariaDB [test] alter table foo change column blah blah int not null;
ERROR 1833 (HY000): Cannot change column 'blah': used in a foreign key
constraint 'bar_ibfk_1' of table 'test.bar'
MariaDB [test]
Matt Riedemann mrie...@linux.vnet.ibm.com wrote:
On 1/6/2015 5:40 PM, Mike Bayer wrote:
Hello -
Victor Sergeyev and I are both observing the following test failure which
occurs with all the tests underneath
nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.This is
against master with a brand new tox environment and everything at the
default.
It does not seem to be occurring on gates that run these tests and
interestingly the tests seem to complete very quickly (under seven seconds)
on the gate as well; the failures here take between 50-100 seconds to
occur, not fully deterministically, and only on the MySQL backend; the
Postgresql and SQLite versions of these tests pass. I’m running against
MariaDB server 10.0.14 with Python 2.7.8 on Fedora 21.
Below is the test just for test_walk_versions, but the warnings (not
necessarily the failures themselves) here also occur for test_migration_267
as well as test_innodb_tables.
I’m still looking into what the cause of this is, I’d imagine it’s
something related to newer MySQL versions or perhaps MariaDB vs. MySQL, I’m
just putting it up here in case someone already knows what this is or has
some clue to save me some time figuring it out. I apologize if I’m just
doing something dumb, I’ve only recently begun to run Nova’s test suite in
full against all backends, so I haven’t yet put intelligent thought into
this nor have I tried to yet look at the migration in question causing the
problem. Will do that next.
[mbayer@thinkpad nova]$ tox -e py27 --
nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions
py27 develop-inst-noop: /home/mbayer/dev/openstack/nova
py27 runtests: PYTHONHASHSEED='0'
py27 runtests: commands[0] | find . -type f -name *.pyc -delete
py27 runtests: commands[1] | bash tools/pretty_tox.sh
nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions
running testr
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./
${OS_TEST_PATH:-./nova/tests} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./
${OS_TEST_PATH:-./nova/tests} --load-list /tmp/tmpw7zqhE
2015-01-06 18:28:12.913 32435 WARNING oslo.db.sqlalchemy.session
[req-5cc6731f-00ef-43df-8aec-4914a44d12c5 ] MySQL SQL mode is '', consider
enabling TRADITIONAL or STRICT_ALL_TABLES
{0}
nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions
[51.553131s] ... FAILED
Captured traceback:
~~~
Traceback (most recent call last):
File
Re: [openstack-dev] [nova] unit test migration failure specific to MySQL/MariaDB - 'uuid': used in a foreign key constraint 'block_device_mapping_instance_uuid_fkey'
working with sdague on IRC, the first thing I’m seeing is that my MariaDB
server is disallowing a change in column that is UNIQUE and has an FK pointing
to it, and this is distinctly different from a straight up MySQL server (see
below).
http://paste.openstack.org/raw/155896/
old school MySQL:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 4840
Server version: 5.6.15 Homebrew
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql create table foo (id int, blah int, primary key (id), unique key (blah))
engine=InnoDB;
Query OK, 0 rows affected (0.01 sec)
mysql create table bar(id int, blah_fk int, primary key (id), foreign key
(blah_fk) references foo(blah)) engine=InnoDB;
Query OK, 0 rows affected (0.01 sec)
mysql alter table foo change column blah blah int not null;
Query OK, 0 rows affected (0.02 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql
MariaDB 10:
MariaDB [test] create table foo (id int, blah int, primary key (id), unique
key (blah));
Query OK, 0 rows affected (0.09 sec)
MariaDB [test] create table bar(id int, blah_fk int, primary key (id), foreign
key (blah_fk) references foo(blah));
Query OK, 0 rows affected (0.12 sec)
MariaDB [test] alter table foo change column blah blah int not null;
ERROR 1833 (HY000): Cannot change column 'blah': used in a foreign key
constraint 'bar_ibfk_1' of table 'test.bar'
MariaDB [test]
Matt Riedemann mrie...@linux.vnet.ibm.com wrote:
On 1/6/2015 5:40 PM, Mike Bayer wrote:
Hello -
Victor Sergeyev and I are both observing the following test failure which
occurs with all the tests underneath
nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.This is
against master with a brand new tox environment and everything at the
default.
It does not seem to be occurring on gates that run these tests and
interestingly the tests seem to complete very quickly (under seven seconds)
on the gate as well; the failures here take between 50-100 seconds to occur,
not fully deterministically, and only on the MySQL backend; the Postgresql
and SQLite versions of these tests pass. I’m running against MariaDB server
10.0.14 with Python 2.7.8 on Fedora 21.
Below is the test just for test_walk_versions, but the warnings (not
necessarily the failures themselves) here also occur for test_migration_267
as well as test_innodb_tables.
I’m still looking into what the cause of this is, I’d imagine it’s something
related to newer MySQL versions or perhaps MariaDB vs. MySQL, I’m just
putting it up here in case someone already knows what this is or has some
clue to save me some time figuring it out. I apologize if I’m just doing
something dumb, I’ve only recently begun to run Nova’s test suite in full
against all backends, so I haven’t yet put intelligent thought into this nor
have I tried to yet look at the migration in question causing the problem.
Will do that next.
[mbayer@thinkpad nova]$ tox -e py27 --
nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions
py27 develop-inst-noop: /home/mbayer/dev/openstack/nova
py27 runtests: PYTHONHASHSEED='0'
py27 runtests: commands[0] | find . -type f -name *.pyc -delete
py27 runtests: commands[1] | bash tools/pretty_tox.sh
nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions
running testr
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./
${OS_TEST_PATH:-./nova/tests} --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./
${OS_TEST_PATH:-./nova/tests} --load-list /tmp/tmpw7zqhE
2015-01-06 18:28:12.913 32435 WARNING oslo.db.sqlalchemy.session
[req-5cc6731f-00ef-43df-8aec-4914a44d12c5 ] MySQL SQL mode is '', consider
enabling TRADITIONAL or STRICT_ALL_TABLES
{0}
nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions
[51.553131s] ... FAILED
Captured traceback:
~~~
Traceback (most recent call last):
File nova/tests/unit/db/test_migrations.py, line 151, in
test_walk_versions
self.walk_versions(self.snake_walk, self.downgrade)
File
/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py,
line 193, in walk_versions
self.migrate_up(version, with_data=True)
File nova/tests/unit/db/test_migrations.py, line 148, in migrate_up
super(NovaMigrationsCheckers, self).migrate_up(version, with_data)
Re: [openstack-dev] [nova] unit test migration failure specific to MySQL/MariaDB - 'uuid': used in a foreign key constraint 'block_device_mapping_instance_uuid_fkey'
On 1/6/2015 5:40 PM, Mike Bayer wrote:
Hello -
Victor Sergeyev and I are both observing the following test failure which
occurs with all the tests underneath
nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.This is against
master with a brand new tox environment and everything at the default.
It does not seem to be occurring on gates that run these tests and
interestingly the tests seem to complete very quickly (under seven seconds) on
the gate as well; the failures here take between 50-100 seconds to occur, not
fully deterministically, and only on the MySQL backend; the Postgresql and
SQLite versions of these tests pass. I’m running against MariaDB server
10.0.14 with Python 2.7.8 on Fedora 21.
Below is the test just for test_walk_versions, but the warnings (not
necessarily the failures themselves) here also occur for test_migration_267 as
well as test_innodb_tables.
I’m still looking into what the cause of this is, I’d imagine it’s something
related to newer MySQL versions or perhaps MariaDB vs. MySQL, I’m just putting
it up here in case someone already knows what this is or has some clue to save
me some time figuring it out. I apologize if I’m just doing something dumb,
I’ve only recently begun to run Nova’s test suite in full against all backends,
so I haven’t yet put intelligent thought into this nor have I tried to yet look
at the migration in question causing the problem. Will do that next.
[mbayer@thinkpad nova]$ tox -e py27 --
nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions
py27 develop-inst-noop: /home/mbayer/dev/openstack/nova
py27 runtests: PYTHONHASHSEED='0'
py27 runtests: commands[0] | find . -type f -name *.pyc -delete
py27 runtests: commands[1] | bash tools/pretty_tox.sh
nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions
running testr
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./nova/tests}
--list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./nova/tests}
--load-list /tmp/tmpw7zqhE
2015-01-06 18:28:12.913 32435 WARNING oslo.db.sqlalchemy.session
[req-5cc6731f-00ef-43df-8aec-4914a44d12c5 ] MySQL SQL mode is '', consider
enabling TRADITIONAL or STRICT_ALL_TABLES
{0}
nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions
[51.553131s] ... FAILED
Captured traceback:
~~~
Traceback (most recent call last):
File nova/tests/unit/db/test_migrations.py, line 151, in
test_walk_versions
self.walk_versions(self.snake_walk, self.downgrade)
File
/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py,
line 193, in walk_versions
self.migrate_up(version, with_data=True)
File nova/tests/unit/db/test_migrations.py, line 148, in migrate_up
super(NovaMigrationsCheckers, self).migrate_up(version, with_data)
File
/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py,
line 263, in migrate_up
self.REPOSITORY, version)
File
/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py,
line 186, in upgrade
return _migrate(url, repository, version, upgrade=True, err=err,
**opts)
File string, line 2, in _migrate
File
/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/util/__init__.py,
line 160, in with_engine
return f(*a, **kw)
File
/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py,
line 366, in _migrate
schema.runchange(ver, change, changeset.step)
File
/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/schema.py,
line 93, in runchange
change.run(self.engine, step)
File
/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/script/py.py,
line 148, in run
script_func(engine)
File
/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py,
line 103, in upgrade
process_null_records(meta, scan=False)
File
/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py,
line 89, in process_null_records
table.columns.uuid.alter(nullable=False)
File
/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py,
line 534, in alter
return alter_column(self, *p, **k)
File
[openstack-dev] [nova] unit test migration failure specific to MySQL/MariaDB - 'uuid': used in a foreign key constraint 'block_device_mapping_instance_uuid_fkey'
Hello -
Victor Sergeyev and I are both observing the following test failure which
occurs with all the tests underneath
nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.This is against
master with a brand new tox environment and everything at the default.
It does not seem to be occurring on gates that run these tests and
interestingly the tests seem to complete very quickly (under seven seconds) on
the gate as well; the failures here take between 50-100 seconds to occur, not
fully deterministically, and only on the MySQL backend; the Postgresql and
SQLite versions of these tests pass. I’m running against MariaDB server
10.0.14 with Python 2.7.8 on Fedora 21.
Below is the test just for test_walk_versions, but the warnings (not
necessarily the failures themselves) here also occur for test_migration_267 as
well as test_innodb_tables.
I’m still looking into what the cause of this is, I’d imagine it’s something
related to newer MySQL versions or perhaps MariaDB vs. MySQL, I’m just putting
it up here in case someone already knows what this is or has some clue to save
me some time figuring it out. I apologize if I’m just doing something dumb,
I’ve only recently begun to run Nova’s test suite in full against all backends,
so I haven’t yet put intelligent thought into this nor have I tried to yet look
at the migration in question causing the problem. Will do that next.
[mbayer@thinkpad nova]$ tox -e py27 --
nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions
py27 develop-inst-noop: /home/mbayer/dev/openstack/nova
py27 runtests: PYTHONHASHSEED='0'
py27 runtests: commands[0] | find . -type f -name *.pyc -delete
py27 runtests: commands[1] | bash tools/pretty_tox.sh
nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions
running testr
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./nova/tests}
--list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./nova/tests}
--load-list /tmp/tmpw7zqhE
2015-01-06 18:28:12.913 32435 WARNING oslo.db.sqlalchemy.session
[req-5cc6731f-00ef-43df-8aec-4914a44d12c5 ] MySQL SQL mode is '', consider
enabling TRADITIONAL or STRICT_ALL_TABLES
{0}
nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions
[51.553131s] ... FAILED
Captured traceback:
~~~
Traceback (most recent call last):
File nova/tests/unit/db/test_migrations.py, line 151, in
test_walk_versions
self.walk_versions(self.snake_walk, self.downgrade)
File
/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py,
line 193, in walk_versions
self.migrate_up(version, with_data=True)
File nova/tests/unit/db/test_migrations.py, line 148, in migrate_up
super(NovaMigrationsCheckers, self).migrate_up(version, with_data)
File
/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py,
line 263, in migrate_up
self.REPOSITORY, version)
File
/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py,
line 186, in upgrade
return _migrate(url, repository, version, upgrade=True, err=err, **opts)
File string, line 2, in _migrate
File
/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/util/__init__.py,
line 160, in with_engine
return f(*a, **kw)
File
/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py,
line 366, in _migrate
schema.runchange(ver, change, changeset.step)
File
/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/schema.py,
line 93, in runchange
change.run(self.engine, step)
File
/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/script/py.py,
line 148, in run
script_func(engine)
File
/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py,
line 103, in upgrade
process_null_records(meta, scan=False)
File
/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py,
line 89, in process_null_records
table.columns.uuid.alter(nullable=False)
File
/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py,
line 534, in alter
return alter_column(self, *p, **k)
File
/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py,
line 141, in
