Hey Marcin, I hope you had a good week-end. After looking through the codebase to get more familiar with how `pre_migrate` and `post_migrate` are currently used, I thought we could simply have the same sort of signal for `post_makemigrations`, where a third party could get a list of the changes and the plan and append operations to the generated file (probably adding itself as dependency too). However, it seems to be far from trivial given the related threads I found related to the subject: https://groups.google.com/d/msg/django-developers/qRNkReCZiCk/I8dIxxhoBwAJ and https://groups.google.com/d/msg/django-developers/iClIpvwlbAU/4uX7q_7aAQAJ I was surprised to see that the `RenameModel` was handled by `ContentType` using the `pre_migrate` hook rather than happening in the `makemigrations` phase, maybe someone can share some context?
I really think this is a real problem and something that could get improved but to do so it would require fixing the bigger issue. To quote Simon Charette (https://groups.google.com/d/msg/django-developers/qRNkReCZiCk/Ah90crNFAAAJ): "As for the makemigrations hooks I think it would require a lot of thought and efforts to get right. Right now the auto-detector is a black box that deals with dependencies and model state deltas resolution." If now is the time to take a stab at it, I'd be happy to help as much as I can. Regards -- Arthur On Friday, October 5, 2018 at 10:28:06 AM UTC-7, Arthur Rio wrote: > > For some reason the text is white… Here it is in black: > > Hey Marcin, > > The problem is that data migration based on app layer (python objects, ie. > Models and Managers here) will cause troubles after some time (when app is > changing). > In the other words - you cannot rely on your app layer when doing database > changes. You should never do that, especially for projects requiring LTS. > > In theory I understand the idea, but in practice, migrations, the ORM and > the content type model are all part of Django and I don’t really see how > the app changing could cause troubles. Do you have an example in mind? > > Maybe "automatic" is not a proper word. They should be created > automatically, but should be applied "at the right time". > > Ok, that we agree on! > > CT opt-in should be connected to a signal related to the creation of > migration files (Autodetector?), not to a signal related to their > execution. I.e. pre/post_autodection signals should be introduced. > > I think we agree that the solution would be some sort of signal triggered > when a model creation/deletion is detected (in the `makemigrations` phase > as opposed to `migrate`) so that some code is added to the generated > migration. The use of a signal is important to keep things decoupled and > flexible. > > The “some code is added to the migration” part still needs to be > determined, either in the shape of insert/delete statements or a RunPython > leveraging the ORM. In both cases, the values to insert (Adding a model) or > to lookup for delete (Removing a model) are just 2 strings, the app label > and the model class name. > > After adding contrib.contenttypes, Django should check existence of > django_content_type table. If exists, Django should only check for data > changes and generate series of inserts/deletes. If not, Django should > generate inserts for all models. If CT is removed later, Django should > remove all CT data . > > It’s a good idea but I don’t know offhand how we can keep migrations and > content type decoupled to do that (especially the removal). > > Finally, I also think the concept could be extended to the permission > model which faces similar issues. > > > Regards > > -- > > Arthur > > On October 5, 2018 at 9:30:58 AM, Arthur Rio (arthur.ri...@gmail.com) > wrote: > > Hey Marcin, > > The problem is that data migration based on app layer (python objects, ie. > Models and Managers here) will cause troubles after some time (when app is > changing). > In the other words - you cannot rely on your app layer when doing database > changes. You should never do that, especially for projects requiring LTS. > > In theory I understand the idea, but in practice, migrations, the ORM and > the content type model are all part of Django and I don’t really see how > the app changing could cause troubles. Do you have an example in mind? > > Maybe "automatic" is not a proper word. They should be created > automatically, but should be applied "at the right time". > > Ok, that we agree on! > > CT opt-in should be connected to a signal related to the creation of > migration files (Autodetector?), not to a signal related to their > execution. I.e. pre/post_autodection signals should be introduced. > > I think we agree that the solution would be some sort of signal triggered > when a model creation/deletion is detected (in the `makemigrations` phase > as opposed to `migrate`) so that some code is added to the generated > migration. The use of a signal is important to keep things decoupled and > flexible. > > The “some code is added to the migration” part still needs to be > determined, either in the shape of insert/delete statements or a RunPython > leveraging the ORM. In both cases, the values to insert (Adding a model) or > to lookup for delete (Removing a model) are just 2 strings, the app label > and the model class name. > > After adding contrib.contenttypes, Django should check existence of > django_content_type table. If exists, Django should only check for data > changes and generate series of inserts/deletes. If not, Django should > generate inserts for all models. If CT is removed later, Django should > remove all CT data . > > It’s a good idea but I don’t know offhand how we can keep migrations and > content type decoupled to do that (especially the removal). > > Finally, I also think the concept could be extended to the permission > model which faces similar issues. > > > Regards > > -- > > Arthur > > > > On October 4, 2018 at 4:47:19 PM, Marcin Nowak (marcin.j.no...@gmail.com) > wrote: > > > Hi Arthur. > > BTW: RunPython() is another thing, which can break your migrations, and > should not be used (especially not by Django internally), because it relies > on the application layer. > > How else can you do a data migration? There is no >> `migrations.InsertIntoTable`, >> > > You're right. That's why "Insert" should be (in my opinion) introduced. > Together with "migrations.Delete". > > The problem is that data migration based on app layer (python objects, ie. > Models and Managers here) will cause troubles after some time (when app is > changing). > In the other words - you cannot rely on your app layer when doing database > changes. You should never do that, especially for projects requiring LTS. > > RunPython() should be used only when you cannot do anything else. In such > case you must accept all consequences. I'm not against RunPython(), but > against doing data migrations using it. > > The problem with hypothetical "Insert" operation is with mapping field > types. Insert cannot use Models directly (app layer is changing over a > time), but should "know" how to map arguments (python types, values) to a > database literals. It can be achieved by introducing field mapping for a > every insert or per migration file (something like "model freeze", but only > for used fields). > > Also Insert should not accept variables calculated in the app layer > (within a migration) - it should contain only plain/direct data. But using > Python as a language for migrations, will be hard to avoid such > possibility. This is important, because database management should not rely > on app layer. Using variables (i.e. from settings) would result in > inconsistent data changes between different environments. > > the only other way currently would be to run a `migrations.RunSql` query >> which may look different based on the database used. >> > > RunSQL is not the solution for db agnostic operations. It may be only used > within a project, because db engine used changes rarely (due to the nature > and importance of the data and relational databases, and systems dependent > on the db). > But RunSQL is a handful operation, because SQL is a natural language for > db management. I'm doing many raw sqls in migrations. > > >> Two things are wrong: >> >> - automatic creation of content types >> >> Why is it wrong to automatically create a content type? >> > > Maybe "automatic" is not a proper word. They should be created > automatically, but should be applied "at the right time". > > >> Content type is an opt-in feature since you can remove it from >> `INSTALLED_APPS`. >> > > I know, but it is required by contrib.auth. I saw no project depending on > something else, so CT is optional.. but theoretically ;) > > >> >> - creation of content types after running migrations >> >> That’s the only real problem for me. Maybe using a signal for >> `migrations.CreateModel` (e.g. post_create_model), instead of using >> `post_migrate` would fix it, but there may be other scenarios I’m not >> thinking about. >> > > Because signals are related to the app layer and mixing app and db layers > together is generally wrong (sorry for wording, please read as "not so > good"), changing one signal to another is not a good solution. > > Before I wrote about the issue, I was doing diagnosis and I was thinking > about the issue for a while. So I think that the best place for applying > data changes is a migration. > Django can detect model changes (adding and removals of models, too), so > it can generate series of inserts or deletes in the right place, to run > them *at the right time*. > > Or if `django.contrib.contenttypes` is only added later on to >> `INSTALLED_APPS`? > > > After adding contrib.contenttypes, Django should check existence of > django_content_type table. If exists, Django should only check for data > changes and generate series of inserts/deletes. If not, Django should > generate inserts for all models. If CT is removed later, Django should > remove all CT data . > > > CT opt-in should be connected to a signal related to the creation of > migration files (Autodetector?), not to a signal related to their > execution. I.e. pre/post_autodection signals should be introduced. > > > Kind Regards, > > Marcin > -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at https://groups.google.com/group/django-developers. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/af22034d-3983-4b19-87b2-256adc1cb11a%40googlegroups.com > > <https://groups.google.com/d/msgid/django-developers/af22034d-3983-4b19-87b2-256adc1cb11a%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/464a7049-87bb-436b-bab6-b8eeecc7ef60%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
