Re: [openstack-dev] [nova] unit test migration failure specific to MySQL/MariaDB - 'uuid': used in a foreign key constraint 'block_device_mapping_instance_uuid_fkey'
OK so it’s looking like sql_mode=‘TRADITIONAL’ that allows it to work. So that is most of it. My MariaDB has no default sql_mode but oslo.db should be setting this, but in any case this seems more like a local oslo.db connection type of thing that I can track down myself, so most of the mystery solved! (at least the part that I didn’t feel like getting into….which I did anyway). Mike Bayer mba...@redhat.com wrote: working with sdague on IRC, the first thing I’m seeing is that my MariaDB server is disallowing a change in column that is UNIQUE and has an FK pointing to it, and this is distinctly different from a straight up MySQL server (see below). http://paste.openstack.org/raw/155896/ old school MySQL: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 4840 Server version: 5.6.15 Homebrew Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql create table foo (id int, blah int, primary key (id), unique key (blah)) engine=InnoDB; Query OK, 0 rows affected (0.01 sec) mysql create table bar(id int, blah_fk int, primary key (id), foreign key (blah_fk) references foo(blah)) engine=InnoDB; Query OK, 0 rows affected (0.01 sec) mysql alter table foo change column blah blah int not null; Query OK, 0 rows affected (0.02 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql MariaDB 10: MariaDB [test] create table foo (id int, blah int, primary key (id), unique key (blah)); Query OK, 0 rows affected (0.09 sec) MariaDB [test] create table bar(id int, blah_fk int, primary key (id), foreign key (blah_fk) references foo(blah)); Query OK, 0 rows affected (0.12 sec) MariaDB [test] alter table foo change column blah blah int not null; ERROR 1833 (HY000): Cannot change column 'blah': used in a foreign key constraint 'bar_ibfk_1' of table 'test.bar' MariaDB [test] Matt Riedemann mrie...@linux.vnet.ibm.com wrote: On 1/6/2015 5:40 PM, Mike Bayer wrote: Hello - Victor Sergeyev and I are both observing the following test failure which occurs with all the tests underneath nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.This is against master with a brand new tox environment and everything at the default. It does not seem to be occurring on gates that run these tests and interestingly the tests seem to complete very quickly (under seven seconds) on the gate as well; the failures here take between 50-100 seconds to occur, not fully deterministically, and only on the MySQL backend; the Postgresql and SQLite versions of these tests pass. I’m running against MariaDB server 10.0.14 with Python 2.7.8 on Fedora 21. Below is the test just for test_walk_versions, but the warnings (not necessarily the failures themselves) here also occur for test_migration_267 as well as test_innodb_tables. I’m still looking into what the cause of this is, I’d imagine it’s something related to newer MySQL versions or perhaps MariaDB vs. MySQL, I’m just putting it up here in case someone already knows what this is or has some clue to save me some time figuring it out. I apologize if I’m just doing something dumb, I’ve only recently begun to run Nova’s test suite in full against all backends, so I haven’t yet put intelligent thought into this nor have I tried to yet look at the migration in question causing the problem. Will do that next. [mbayer@thinkpad nova]$ tox -e py27 -- nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions py27 develop-inst-noop: /home/mbayer/dev/openstack/nova py27 runtests: PYTHONHASHSEED='0' py27 runtests: commands[0] | find . -type f -name *.pyc -delete py27 runtests: commands[1] | bash tools/pretty_tox.sh nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions running testr running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \ OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \ OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \ ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./nova/tests} --list running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \ OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \ OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \ ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./nova/tests} --load-list /tmp/tmpw7zqhE 2015-01-06 18:28:12.913 32435 WARNING oslo.db.sqlalchemy.session [req-5cc6731f-00ef-43df-8aec-4914a44d12c5 ] MySQL SQL mode is '', consider enabling TRADITIONAL or STRICT_ALL_TABLES {0} nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions [51.553131s] ... FAILED Captured traceback: ~~~ Traceback (most recent call last): File
Re: [openstack-dev] [nova] unit test migration failure specific to MySQL/MariaDB - 'uuid': used in a foreign key constraint 'block_device_mapping_instance_uuid_fkey'
working with sdague on IRC, the first thing I’m seeing is that my MariaDB server is disallowing a change in column that is UNIQUE and has an FK pointing to it, and this is distinctly different from a straight up MySQL server (see below). http://paste.openstack.org/raw/155896/ old school MySQL: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 4840 Server version: 5.6.15 Homebrew Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql create table foo (id int, blah int, primary key (id), unique key (blah)) engine=InnoDB; Query OK, 0 rows affected (0.01 sec) mysql create table bar(id int, blah_fk int, primary key (id), foreign key (blah_fk) references foo(blah)) engine=InnoDB; Query OK, 0 rows affected (0.01 sec) mysql alter table foo change column blah blah int not null; Query OK, 0 rows affected (0.02 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql MariaDB 10: MariaDB [test] create table foo (id int, blah int, primary key (id), unique key (blah)); Query OK, 0 rows affected (0.09 sec) MariaDB [test] create table bar(id int, blah_fk int, primary key (id), foreign key (blah_fk) references foo(blah)); Query OK, 0 rows affected (0.12 sec) MariaDB [test] alter table foo change column blah blah int not null; ERROR 1833 (HY000): Cannot change column 'blah': used in a foreign key constraint 'bar_ibfk_1' of table 'test.bar' MariaDB [test] Matt Riedemann mrie...@linux.vnet.ibm.com wrote: On 1/6/2015 5:40 PM, Mike Bayer wrote: Hello - Victor Sergeyev and I are both observing the following test failure which occurs with all the tests underneath nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.This is against master with a brand new tox environment and everything at the default. It does not seem to be occurring on gates that run these tests and interestingly the tests seem to complete very quickly (under seven seconds) on the gate as well; the failures here take between 50-100 seconds to occur, not fully deterministically, and only on the MySQL backend; the Postgresql and SQLite versions of these tests pass. I’m running against MariaDB server 10.0.14 with Python 2.7.8 on Fedora 21. Below is the test just for test_walk_versions, but the warnings (not necessarily the failures themselves) here also occur for test_migration_267 as well as test_innodb_tables. I’m still looking into what the cause of this is, I’d imagine it’s something related to newer MySQL versions or perhaps MariaDB vs. MySQL, I’m just putting it up here in case someone already knows what this is or has some clue to save me some time figuring it out. I apologize if I’m just doing something dumb, I’ve only recently begun to run Nova’s test suite in full against all backends, so I haven’t yet put intelligent thought into this nor have I tried to yet look at the migration in question causing the problem. Will do that next. [mbayer@thinkpad nova]$ tox -e py27 -- nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions py27 develop-inst-noop: /home/mbayer/dev/openstack/nova py27 runtests: PYTHONHASHSEED='0' py27 runtests: commands[0] | find . -type f -name *.pyc -delete py27 runtests: commands[1] | bash tools/pretty_tox.sh nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions running testr running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \ OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \ OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \ ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./nova/tests} --list running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \ OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \ OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \ ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./nova/tests} --load-list /tmp/tmpw7zqhE 2015-01-06 18:28:12.913 32435 WARNING oslo.db.sqlalchemy.session [req-5cc6731f-00ef-43df-8aec-4914a44d12c5 ] MySQL SQL mode is '', consider enabling TRADITIONAL or STRICT_ALL_TABLES {0} nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions [51.553131s] ... FAILED Captured traceback: ~~~ Traceback (most recent call last): File nova/tests/unit/db/test_migrations.py, line 151, in test_walk_versions self.walk_versions(self.snake_walk, self.downgrade) File /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py, line 193, in walk_versions self.migrate_up(version, with_data=True) File nova/tests/unit/db/test_migrations.py, line 148, in migrate_up super(NovaMigrationsCheckers, self).migrate_up(version, with_data)
Re: [openstack-dev] [nova] unit test migration failure specific to MySQL/MariaDB - 'uuid': used in a foreign key constraint 'block_device_mapping_instance_uuid_fkey'
On 1/6/2015 5:40 PM, Mike Bayer wrote: Hello - Victor Sergeyev and I are both observing the following test failure which occurs with all the tests underneath nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.This is against master with a brand new tox environment and everything at the default. It does not seem to be occurring on gates that run these tests and interestingly the tests seem to complete very quickly (under seven seconds) on the gate as well; the failures here take between 50-100 seconds to occur, not fully deterministically, and only on the MySQL backend; the Postgresql and SQLite versions of these tests pass. I’m running against MariaDB server 10.0.14 with Python 2.7.8 on Fedora 21. Below is the test just for test_walk_versions, but the warnings (not necessarily the failures themselves) here also occur for test_migration_267 as well as test_innodb_tables. I’m still looking into what the cause of this is, I’d imagine it’s something related to newer MySQL versions or perhaps MariaDB vs. MySQL, I’m just putting it up here in case someone already knows what this is or has some clue to save me some time figuring it out. I apologize if I’m just doing something dumb, I’ve only recently begun to run Nova’s test suite in full against all backends, so I haven’t yet put intelligent thought into this nor have I tried to yet look at the migration in question causing the problem. Will do that next. [mbayer@thinkpad nova]$ tox -e py27 -- nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions py27 develop-inst-noop: /home/mbayer/dev/openstack/nova py27 runtests: PYTHONHASHSEED='0' py27 runtests: commands[0] | find . -type f -name *.pyc -delete py27 runtests: commands[1] | bash tools/pretty_tox.sh nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions running testr running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \ OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \ OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \ ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./nova/tests} --list running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \ OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \ OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \ ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./nova/tests} --load-list /tmp/tmpw7zqhE 2015-01-06 18:28:12.913 32435 WARNING oslo.db.sqlalchemy.session [req-5cc6731f-00ef-43df-8aec-4914a44d12c5 ] MySQL SQL mode is '', consider enabling TRADITIONAL or STRICT_ALL_TABLES {0} nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions [51.553131s] ... FAILED Captured traceback: ~~~ Traceback (most recent call last): File nova/tests/unit/db/test_migrations.py, line 151, in test_walk_versions self.walk_versions(self.snake_walk, self.downgrade) File /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py, line 193, in walk_versions self.migrate_up(version, with_data=True) File nova/tests/unit/db/test_migrations.py, line 148, in migrate_up super(NovaMigrationsCheckers, self).migrate_up(version, with_data) File /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py, line 263, in migrate_up self.REPOSITORY, version) File /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py, line 186, in upgrade return _migrate(url, repository, version, upgrade=True, err=err, **opts) File string, line 2, in _migrate File /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/util/__init__.py, line 160, in with_engine return f(*a, **kw) File /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py, line 366, in _migrate schema.runchange(ver, change, changeset.step) File /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/schema.py, line 93, in runchange change.run(self.engine, step) File /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/script/py.py, line 148, in run script_func(engine) File /home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py, line 103, in upgrade process_null_records(meta, scan=False) File /home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py, line 89, in process_null_records table.columns.uuid.alter(nullable=False) File /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py, line 534, in alter return alter_column(self, *p, **k) File