#32351: AlterField migration on ForeignKey column fails on fresh DB when
referenced
model has been subsequently modified
-------------------------------------+-------------------------------------
Reporter: | Owner: nobody
djangobugreport |
Type: Bug | Status: new
Component: | Version: 3.1
Migrations | Keywords: migration
Severity: Release | AlterField ForeignKey default
blocker | on_delete
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
I have run into a specific sequence of migrations that work one by one but
fail when applied in sequence. The sequence is as follows (generate and
apply migrations after each below step to reproduce the bug):
1. Create two models with one referencing the other like so:
{{{
class Foo(models.Model):
hoge = models.IntegerField(unique = True)
class Bar(models.Model):
foo = models.ForeignKey(Foo, on_delete = models.SET_NULL, null = True)
}}}
2. Change the ForeignKey field on the referencing model to pull a default
instance from a python function:
{{{
class Foo(models.Model):
hoge = models.IntegerField(unique = True)
def get_default_foo():
foo, created = Foo.objects.get_or_create(
hoge = 0
);
return foo.id
class Bar(models.Model):
#foo = models.ForeignKey(Foo, on_delete = models.SET_NULL, null =
True)
foo = models.ForeignKey(Foo, on_delete = models.SET_DEFAULT, default =
get_default_foo)
}}}
3. Add a new field to the referenced model:
{{{
class Foo(models.Model):
hoge = models.IntegerField(unique = True)
piyo = models.IntegerField(default = 0)
def get_default_foo():
foo, created = Foo.objects.get_or_create(
hoge = 0
);
return foo.id
class Bar(models.Model):
#foo = models.ForeignKey(Foo, on_delete = models.SET_NULL, null =
True)
foo = models.ForeignKey(Foo, on_delete = models.SET_DEFAULT, default =
get_default_foo)
}}}
Now, delete the DB and try to recreate from scratch, and the migration
created in the second step above will fail complaining that the column
"piyo" does not exist. Evidently, the migration runtime is not correctly
referencing the old version of Foo without the piyo field, but rather the
current version in the source code. Here is my stack trace from Visual
Studio:
{{{
Operations to perform:
Apply all migrations: TheApp, admin, auth, contenttypes, sessions
Running migrations:
Applying TheApp.0001_initial... OK
Applying TheApp.0002_auto_20210114_1330...Traceback (most recent call
last):
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\backends\utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\backends\sqlite3\base.py", line 413, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: no such column: TheApp_foo.piyo
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "D:\sandbox\django\three\DjangoWebProject1\manage.py", line 17, in
<module>
execute_from_command_line(sys.argv)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\core\management\__init__.py", line 401, in
execute_from_command_line
utility.execute()
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\core\management\__init__.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\core\management\base.py", line 330, in run_from_argv
self.execute(*args, **cmd_options)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\core\management\base.py", line 371, in execute
output = self.handle(*args, **options)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\core\management\base.py", line 85, in wrapped
res = handle_func(*args, **kwargs)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\core\management\commands\migrate.py", line 245, in handle
fake_initial=fake_initial,
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\migrations\executor.py", line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake,
fake_initial=fake_initial)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\migrations\executor.py", line 147, in
_migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake,
fake_initial=fake_initial)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\migrations\executor.py", line 227, in apply_migration
state = migration.apply(state, schema_editor)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\migrations\migration.py", line 124, in apply
operation.database_forwards(self.app_label, schema_editor, old_state,
project_state)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\migrations\operations\fields.py", line 236, in
database_forwards
schema_editor.alter_field(from_model, from_field, to_field)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\backends\sqlite3\schema.py", line 138, in alter_field
super().alter_field(model, old_field, new_field, strict=strict)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\backends\base\schema.py", line 572, in alter_field
old_db_params, new_db_params, strict)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\backends\sqlite3\schema.py", line 360, in _alter_field
self._remake_table(model, alter_field=(old_field, new_field))
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\backends\sqlite3\schema.py", line 200, in _remake_table
'default': self.quote_value(self.effective_default(new_field))
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\backends\base\schema.py", line 303, in
effective_default
return field.get_db_prep_save(self._effective_default(field),
self.connection)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\backends\base\schema.py", line 282, in
_effective_default
default = field.get_default()
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\models\fields\related.py", line 960, in get_default
field_default = super().get_default()
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\models\fields\__init__.py", line 831, in get_default
return self._get_default()
File ".\TheApp\models.py", line 11, in get_default_foo
hoge = 0
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\models\manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\models\query.py", line 573, in get_or_create
return self.get(**kwargs), False
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\models\query.py", line 425, in get
num = len(clone)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\models\query.py", line 269, in __len__
self._fetch_all()
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\models\query.py", line 1308, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\models\query.py", line 53, in __iter__
results = compiler.execute_sql(chunked_fetch=self.chunked_fetch,
chunk_size=self.chunk_size)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\models\sql\compiler.py", line 1156, in execute_sql
cursor.execute(sql, params)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\backends\utils.py", line 98, in execute
return super().execute(sql, params)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\backends\utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False,
executor=self._execute)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\backends\utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\backends\utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\backends\utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-
packages\django\db\backends\sqlite3\base.py", line 413, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such column: TheApp_foo.piyo
}}}
I cannot see any obvious way to work around this problem without diving
into the internals of the migration engine. Similar migrations work fine,
it's only this combination of changing "on_delete" and "default" in this
way that I've seen cause the problem. Although this isn't personally
blocking a release for me, I've marked this bug as "Release Blocker" as
IMO it is fairly severe because it could easily block the update of a
production database if the above migrations had to be applied in sequence.
(It would be possible to rewind the commit history and apply the
migrations one by one along with the accompanying source code, but this
could potentially be a huge amount of work, especially if dealing with an
automated deployment system, etc.).
I have attached a zip containing a test Visual Studio Django project with
the code above. I simply created an empty project with the Visual Studio
Django wizard, updated requirements.txt to the most recent version of
Django (3.1.5), changed the parts of the default project that caused
errors on Django 3, and then wrote the test case above. There's nothing
Visual Studio centric about this problem so it should reproduce on any
platform as far as I can tell.
If there is a workaround for this bug, please advise. Thank you.
--
Ticket URL: <https://code.djangoproject.com/ticket/32351>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/058.dbfd1aca497a9326a8221d4b3c18391f%40djangoproject.com.
