#31055: Omits test_ prefix from database name when running subset of tests
-----------------------------------+--------------------------------------
Reporter: Matthijs Kooijman | Owner: nobody
Type: Bug | Status: new
Component: Testing framework | Version: master
Severity: Normal | Resolution:
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------------+--------------------------------------
Changes (by Matthijs Kooijman):
* cc: Matthijs Kooijman (added)
* version: 3.0 => master
Old description:
> While debugging some test framework issues wrt mysql, I noticed a problem
> where the test runner would try to access the test database without
> prefixing `test_`, leading to an access denied error (because my
> permissions are set up tightly).
>
> What I suspect happens is that this subset of tests only uses the
> `default` database, so only that one is set up by
> `DisoveryRunner.setup_databases`. This is confirmed by using a debugger,
> which shows
> [[https://github.com/django/django/blob/845042b3d9faaefef8855c2bab48bd9532cd00ca/django/test/runner.py#L683|databases]]
> only contains 'default'. Then, it runs the `check` management command,
> which looks at `settings.DATABASES`, which still contains the settings
> for `default` and `other`. This in turn causes a connection to the
> `other` database to be made, but since the name of that database is not
> modified by
> [[https://github.com/django/django/blob/845042b3d9faaefef8855c2bab48bd9532cd00ca/django/db/backends/base/creation.py#L61|create_test_db]],
> that still refers to the original name, and the connection fails.
>
> To reproduce, I have a clean master
> (c33eb6dcd0c211f8f02b2976fe3b3463f0a54498), with the following
> `tests/test_mysql.py`:
>
> {{{
> DATABASES = {
> 'default': {
> 'ENGINE': 'django.db.backends.mysql',
> 'HOST': 'localhost',
> 'USER': 'test_django',
> 'PASSWORD': 'XXX',
> # Django prepends test_ to this name...
> 'NAME': 'django_main',
> },
> 'other': {
> 'ENGINE': 'django.db.backends.mysql',
> 'HOST': 'localhost',
> 'USER': 'test_django',
> 'PASSWORD': 'XXX',
> # Django prepends test_ to this name...
> 'NAME': 'django_other',
> }
> }
>
> SECRET_KEY = "django_tests_secret_key"
>
> # Use a fast hasher to speed up tests.
> PASSWORD_HASHERS = [
> 'django.contrib.auth.hashers.MD5PasswordHasher',
> ]
> }}}
>
> Then inside `tests`, I run:
>
> ./runtests.py --settings test_mysql --parallel 1 timezones
>
> I think the `--parallel 1` is not strictly needed, but might make things
> easier to debug. With the above, I get:
>
> {{{
> Creating test database for alias 'default'...
> Destroying test database for alias 'default'...
> Testing against Django installed in
> '/home/matthijs/docs/src/upstream/django/django'
> Traceback (most recent call last):
> File
> "/home/matthijs/docs/src/upstream/django/django/db/backends/base/base.py",
> line 220, in ensure_connection
> self.connect()
> File "/home/matthijs/docs/src/upstream/django/django/utils/asyncio.py",
> line 24, in inner
> return func(*args, **kwargs)
> File
> "/home/matthijs/docs/src/upstream/django/django/db/backends/base/base.py",
> line 197, in connect
> self.connection = self.get_new_connection(conn_params)
> File "/home/matthijs/docs/src/upstream/django/django/utils/asyncio.py",
> line 24, in inner
> return func(*args, **kwargs)
> File
> "/home/matthijs/docs/src/upstream/django/django/db/backends/mysql/base.py",
> line 233, in get_new_connection
> return Database.connect(**conn_params)
> File "/home/matthijs/docs/src/upstream/django/venv/lib/python3.7/site-
> packages/MySQLdb/__init__.py", line 84, in Connect
> return Connection(*args, **kwargs)
> File "/home/matthijs/docs/src/upstream/django/venv/lib/python3.7/site-
> packages/MySQLdb/connections.py", line 179, in __init__
> super(Connection, self).__init__(*args, **kwargs2)
> MySQLdb._exceptions.OperationalError: (1044, "Access denied for user
> 'test_django'@'localhost' to database 'django_other'")
>
> The above exception was the direct cause of the following exception:
>
> Traceback (most recent call last):
> File "./runtests.py", line 566, in <module>
> options.start_at, options.start_after, options.pdb,
> File "./runtests.py", line 308, in django_tests
> extra_tests=extra_tests,
> File "/home/matthijs/docs/src/upstream/django/django/test/runner.py",
> line 687, in run_tests
> self.run_checks()
> File "/home/matthijs/docs/src/upstream/django/django/test/runner.py",
> line 625, in run_checks
> call_command('check', verbosity=self.verbosity)
> File
> "/home/matthijs/docs/src/upstream/django/django/core/management/__init__.py",
> line 168, in call_command
> return command.execute(*args, **defaults)
> File
> "/home/matthijs/docs/src/upstream/django/django/core/management/base.py",
> line 369, in execute
> output = self.handle(*args, **options)
> File
> "/home/matthijs/docs/src/upstream/django/django/core/management/commands/check.py",
> line 64, in handle
> fail_level=getattr(checks, options['fail_level']),
> File
> "/home/matthijs/docs/src/upstream/django/django/core/management/base.py",
> line 395, in check
> include_deployment_checks=include_deployment_checks,
> File
> "/home/matthijs/docs/src/upstream/django/django/core/management/base.py",
> line 382, in _run_checks
> return checks.run_checks(**kwargs)
> File
> "/home/matthijs/docs/src/upstream/django/django/core/checks/registry.py",
> line 72, in run_checks
> new_errors = check(app_configs=app_configs)
> File
> "/home/matthijs/docs/src/upstream/django/django/core/checks/model_checks.py",
> line 34, in check_all_models
> errors.extend(model.check(**kwargs))
> File
> "/home/matthijs/docs/src/upstream/django/django/db/models/base.py", line
> 1276, in check
> *cls._check_constraints(),
> File
> "/home/matthijs/docs/src/upstream/django/django/db/models/base.py", line
> 1842, in _check_constraints
> connection.features.supports_table_check_constraints or
> File
> "/home/matthijs/docs/src/upstream/django/django/utils/functional.py",
> line 48, in __get__
> res = instance.__dict__[self.name] = self.func(instance)
> File
> "/home/matthijs/docs/src/upstream/django/django/db/backends/mysql/features.py",
> line 97, in supports_column_check_constraints
> if self.connection.mysql_is_mariadb:
> File
> "/home/matthijs/docs/src/upstream/django/django/utils/functional.py",
> line 48, in __get__
> res = instance.__dict__[self.name] = self.func(instance)
> File
> "/home/matthijs/docs/src/upstream/django/django/db/backends/mysql/base.py",
> line 364, in mysql_is_mariadb
> return 'mariadb' in self.mysql_server_info.lower()
> File
> "/home/matthijs/docs/src/upstream/django/django/utils/functional.py",
> line 48, in __get__
> res = instance.__dict__[self.name] = self.func(instance)
> File
> "/home/matthijs/docs/src/upstream/django/django/db/backends/mysql/base.py",
> line 351, in mysql_server_info
> with self.temporary_connection() as cursor:
> File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
> return next(self.gen)
> File
> "/home/matthijs/docs/src/upstream/django/django/db/backends/base/base.py",
> line 604, in temporary_connection
> with self.cursor() as cursor:
> File "/home/matthijs/docs/src/upstream/django/django/utils/asyncio.py",
> line 24, in inner
> return func(*args, **kwargs)
> File
> "/home/matthijs/docs/src/upstream/django/django/db/backends/base/base.py",
> line 260, in cursor
> return self._cursor()
> File
> "/home/matthijs/docs/src/upstream/django/django/db/backends/base/base.py",
> line 236, in _cursor
> self.ensure_connection()
> File "/home/matthijs/docs/src/upstream/django/django/utils/asyncio.py",
> line 24, in inner
> return func(*args, **kwargs)
> File
> "/home/matthijs/docs/src/upstream/django/django/db/backends/base/base.py",
> line 220, in ensure_connection
> self.connect()
> File "/home/matthijs/docs/src/upstream/django/django/db/utils.py", line
> 90, in __exit__
> raise dj_exc_value.with_traceback(traceback) from exc_value
> File
> "/home/matthijs/docs/src/upstream/django/django/db/backends/base/base.py",
> line 220, in ensure_connection
> self.connect()
> File "/home/matthijs/docs/src/upstream/django/django/utils/asyncio.py",
> line 24, in inner
> return func(*args, **kwargs)
> File
> "/home/matthijs/docs/src/upstream/django/django/db/backends/base/base.py",
> line 197, in connect
> self.connection = self.get_new_connection(conn_params)
> File "/home/matthijs/docs/src/upstream/django/django/utils/asyncio.py",
> line 24, in inner
> return func(*args, **kwargs)
> File
> "/home/matthijs/docs/src/upstream/django/django/db/backends/mysql/base.py",
> line 233, in get_new_connection
> return Database.connect(**conn_params)
> File "/home/matthijs/docs/src/upstream/django/venv/lib/python3.7/site-
> packages/MySQLdb/__init__.py", line 84, in Connect
> return Connection(*args, **kwargs)
> File "/home/matthijs/docs/src/upstream/django/venv/lib/python3.7/site-
> packages/MySQLdb/connections.py", line 179, in __init__
> super(Connection, self).__init__(*args, **kwargs2)
> django.db.utils.OperationalError: (1044, "Access denied for user
> 'test_django'@'localhost' to database 'django_other'")
> }}}
>
> I am not quite familiar with this code, and this is already a distraction
> from a distraction from a distraction from the actual project I was
> working on, so I'm going to leave this here for others to fix :-)
New description:
While debugging some test framework issues wrt mysql, I noticed a problem
where the test runner would try to access the test database without
prefixing `test_`, leading to an access denied error (because my
permissions are set up tightly).
What I suspect happens is that this subset of tests only uses the
`default` database, so only that one is set up by
`DisoveryRunner.setup_databases`. This is confirmed by using a debugger,
which shows
[[https://github.com/django/django/blob/845042b3d9faaefef8855c2bab48bd9532cd00ca/django/test/runner.py#L683|databases]]
only contains 'default'. Then, it runs the `check` management command,
which looks at `settings.DATABASES`, which still contains the settings for
`default` and `other`. This in turn causes a connection to the `other`
database to be made, but since the name of that database is not modified
by
[[https://github.com/django/django/blob/845042b3d9faaefef8855c2bab48bd9532cd00ca/django/db/backends/base/creation.py#L61|create_test_db]],
that still refers to the original name, and the connection fails.
To reproduce, I have a clean master
(c33eb6dcd0c211f8f02b2976fe3b3463f0a54498), with the following
`tests/test_mysql.py`:
{{{
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'HOST': 'localhost',
'USER': 'test_django',
'PASSWORD': 'XXX',
# Django prepends test_ to this name...
'NAME': 'django_main',
},
'other': {
'ENGINE': 'django.db.backends.mysql',
'HOST': 'localhost',
'USER': 'test_django',
'PASSWORD': 'XXX',
# Django prepends test_ to this name...
'NAME': 'django_other',
}
}
SECRET_KEY = "django_tests_secret_key"
# Use a fast hasher to speed up tests.
PASSWORD_HASHERS = [
'django.contrib.auth.hashers.MD5PasswordHasher',
]
}}}
Then inside `tests`, I run:
{{{
./runtests.py --settings test_mysql --parallel 1 timezones
}}}
I think the `--parallel 1` is not strictly needed, but might make things
easier to debug. With the above, I get:
{{{
Creating test database for alias 'default'...
Destroying test database for alias 'default'...
Testing against Django installed in
'/home/matthijs/docs/src/upstream/django/django'
Traceback (most recent call last):
File
"/home/matthijs/docs/src/upstream/django/django/db/backends/base/base.py",
line 220, in ensure_connection
self.connect()
File "/home/matthijs/docs/src/upstream/django/django/utils/asyncio.py",
line 24, in inner
return func(*args, **kwargs)
File
"/home/matthijs/docs/src/upstream/django/django/db/backends/base/base.py",
line 197, in connect
self.connection = self.get_new_connection(conn_params)
File "/home/matthijs/docs/src/upstream/django/django/utils/asyncio.py",
line 24, in inner
return func(*args, **kwargs)
File
"/home/matthijs/docs/src/upstream/django/django/db/backends/mysql/base.py",
line 233, in get_new_connection
return Database.connect(**conn_params)
File "/home/matthijs/docs/src/upstream/django/venv/lib/python3.7/site-
packages/MySQLdb/__init__.py", line 84, in Connect
return Connection(*args, **kwargs)
File "/home/matthijs/docs/src/upstream/django/venv/lib/python3.7/site-
packages/MySQLdb/connections.py", line 179, in __init__
super(Connection, self).__init__(*args, **kwargs2)
MySQLdb._exceptions.OperationalError: (1044, "Access denied for user
'test_django'@'localhost' to database 'django_other'")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "./runtests.py", line 566, in <module>
options.start_at, options.start_after, options.pdb,
File "./runtests.py", line 308, in django_tests
extra_tests=extra_tests,
File "/home/matthijs/docs/src/upstream/django/django/test/runner.py",
line 687, in run_tests
self.run_checks()
File "/home/matthijs/docs/src/upstream/django/django/test/runner.py",
line 625, in run_checks
call_command('check', verbosity=self.verbosity)
File
"/home/matthijs/docs/src/upstream/django/django/core/management/__init__.py",
line 168, in call_command
return command.execute(*args, **defaults)
File
"/home/matthijs/docs/src/upstream/django/django/core/management/base.py",
line 369, in execute
output = self.handle(*args, **options)
File
"/home/matthijs/docs/src/upstream/django/django/core/management/commands/check.py",
line 64, in handle
fail_level=getattr(checks, options['fail_level']),
File
"/home/matthijs/docs/src/upstream/django/django/core/management/base.py",
line 395, in check
include_deployment_checks=include_deployment_checks,
File
"/home/matthijs/docs/src/upstream/django/django/core/management/base.py",
line 382, in _run_checks
return checks.run_checks(**kwargs)
File
"/home/matthijs/docs/src/upstream/django/django/core/checks/registry.py",
line 72, in run_checks
new_errors = check(app_configs=app_configs)
File
"/home/matthijs/docs/src/upstream/django/django/core/checks/model_checks.py",
line 34, in check_all_models
errors.extend(model.check(**kwargs))
File "/home/matthijs/docs/src/upstream/django/django/db/models/base.py",
line 1276, in check
*cls._check_constraints(),
File "/home/matthijs/docs/src/upstream/django/django/db/models/base.py",
line 1842, in _check_constraints
connection.features.supports_table_check_constraints or
File
"/home/matthijs/docs/src/upstream/django/django/utils/functional.py", line
48, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File
"/home/matthijs/docs/src/upstream/django/django/db/backends/mysql/features.py",
line 97, in supports_column_check_constraints
if self.connection.mysql_is_mariadb:
File
"/home/matthijs/docs/src/upstream/django/django/utils/functional.py", line
48, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File
"/home/matthijs/docs/src/upstream/django/django/db/backends/mysql/base.py",
line 364, in mysql_is_mariadb
return 'mariadb' in self.mysql_server_info.lower()
File
"/home/matthijs/docs/src/upstream/django/django/utils/functional.py", line
48, in __get__
res = instance.__dict__[self.name] = self.func(instance)
File
"/home/matthijs/docs/src/upstream/django/django/db/backends/mysql/base.py",
line 351, in mysql_server_info
with self.temporary_connection() as cursor:
File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
return next(self.gen)
File
"/home/matthijs/docs/src/upstream/django/django/db/backends/base/base.py",
line 604, in temporary_connection
with self.cursor() as cursor:
File "/home/matthijs/docs/src/upstream/django/django/utils/asyncio.py",
line 24, in inner
return func(*args, **kwargs)
File
"/home/matthijs/docs/src/upstream/django/django/db/backends/base/base.py",
line 260, in cursor
return self._cursor()
File
"/home/matthijs/docs/src/upstream/django/django/db/backends/base/base.py",
line 236, in _cursor
self.ensure_connection()
File "/home/matthijs/docs/src/upstream/django/django/utils/asyncio.py",
line 24, in inner
return func(*args, **kwargs)
File
"/home/matthijs/docs/src/upstream/django/django/db/backends/base/base.py",
line 220, in ensure_connection
self.connect()
File "/home/matthijs/docs/src/upstream/django/django/db/utils.py", line
90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File
"/home/matthijs/docs/src/upstream/django/django/db/backends/base/base.py",
line 220, in ensure_connection
self.connect()
File "/home/matthijs/docs/src/upstream/django/django/utils/asyncio.py",
line 24, in inner
return func(*args, **kwargs)
File
"/home/matthijs/docs/src/upstream/django/django/db/backends/base/base.py",
line 197, in connect
self.connection = self.get_new_connection(conn_params)
File "/home/matthijs/docs/src/upstream/django/django/utils/asyncio.py",
line 24, in inner
return func(*args, **kwargs)
File
"/home/matthijs/docs/src/upstream/django/django/db/backends/mysql/base.py",
line 233, in get_new_connection
return Database.connect(**conn_params)
File "/home/matthijs/docs/src/upstream/django/venv/lib/python3.7/site-
packages/MySQLdb/__init__.py", line 84, in Connect
return Connection(*args, **kwargs)
File "/home/matthijs/docs/src/upstream/django/venv/lib/python3.7/site-
packages/MySQLdb/connections.py", line 179, in __init__
super(Connection, self).__init__(*args, **kwargs2)
django.db.utils.OperationalError: (1044, "Access denied for user
'test_django'@'localhost' to database 'django_other'")
}}}
I am not quite familiar with this code, and this is already a distraction
from a distraction from a distraction from the actual project I was
working on, so I'm going to leave this here for others to fix :-)
--
--
Ticket URL: <https://code.djangoproject.com/ticket/31055#comment:1>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/074.ce12b3677843102db2ef8cb7bedad65f%40djangoproject.com.
