Re: Migrations with multiple databases
Ah nice thanks Markus, however - different subsets of INSTALLED_APPS sounds a bit hacky, how would I do it anyway? checking in the settings which database is being migrated and providing a different list? Or any other way? - MIGRATION_MODULES sounds a bit safer, but I'm not sure it gives the same advantage, since from my understanding django still has to go through all the apps for each db anyway right? >From my understanding the INSTALLED_APPS would be a much bigger performance gain right? thanks On Tuesday, October 25, 2016 at 4:29:45 PM UTC+1, Markus Holtermann wrote: > > Hi Andrea, > > If you have models that are entirely unrelated in different databases > you could use different values for INSTALLED_APPS to limit the apps > Django sees or by settings the values in the MIGRATION_MODULES dict to > None to tell Django that those apps don't have migrations. > > Cheers, > > /Markus > > On Tue, Oct 25, 2016 at 08:05:46AM -0700, andrea crotti wrote: > >We have a couple of databases with a tiny number of tables, but > >django-migrations has still go through all the migrations anyway. > >So even if the SQL itself is nothing it still takes a massive amount of > >time and memory for the usual known issues with migrations on big > projects. > > > >I guess it works this way because it only knows at run time in the router > >if something should be done, however it's a bit odd. > > > >If we declared the models per database somehow statically, could we not > >just skip everything else? > > > >-- > >You received this message because you are subscribed to the Google Groups > "Django users" group. > >To unsubscribe from this group and stop receiving emails from it, send an > email to django-users...@googlegroups.com . > >To post to this group, send email to django...@googlegroups.com > . > >Visit this group at https://groups.google.com/group/django-users. > >To view this discussion on the web visit > https://groups.google.com/d/msgid/django-users/f5340bbf-2247-4068-bd37-9f3a0faf23d4%40googlegroups.com. > > > >For more options, visit https://groups.google.com/d/optout. > > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at https://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/27b7a52a-d7d5-47b0-be8f-643af74379ec%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Migrations with multiple databases
We have a couple of databases with a tiny number of tables, but django-migrations has still go through all the migrations anyway. So even if the SQL itself is nothing it still takes a massive amount of time and memory for the usual known issues with migrations on big projects. I guess it works this way because it only knows at run time in the router if something should be done, however it's a bit odd. If we declared the models per database somehow statically, could we not just skip everything else? -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at https://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/f5340bbf-2247-4068-bd37-9f3a0faf23d4%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Sqlmigrate generated SQL
Answering to myself obviously sqlmigrate can't really do anything with code in RunPython. So I suspect the biggest problem would be something like this: - early data migration that uses an initial schema a certain model - a later schema migration that changes that so trying to generate the SQL after would not apply because the schema changed in the meanwhile.. On Tuesday, October 4, 2016 at 10:41:25 AM UTC+1, andrea crotti wrote: > > Related to this other thread: > > > https://groups.google.com/forum/#!msg/django-users/V8Ei2qZJ8VI/bFUeY2wTAQAJ;context-place=forum/django-users > > I'm working on trying to find a workaround for migrations killing our > local/CI servers, until there is a better fix in Django stable.. > > So started this project to do some analysis: > https://github.com/AndreaCrotti/django-migrations-analysis > > > One interesting thing is that it looks like using sqlmigrate (taking some > code from it) to generate the raw SQL and composing ti basically generates > exactly the same final schema for the database. > > > https://github.com/AndreaCrotti/django-migrations-analysis/blob/master/commands/sqlmigrate_all.py > > In general the question is, assuming all the settings are the same, can > they differ under some conditions? > I guess for example that using RunPython would cause issues in this case > right? > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at https://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/a805f787-0cb3-4166-ac01-b00b8dd718ad%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Sqlmigrate generated SQL
Related to this other thread: https://groups.google.com/forum/#!msg/django-users/V8Ei2qZJ8VI/bFUeY2wTAQAJ;context-place=forum/django-users I'm working on trying to find a workaround for migrations killing our local/CI servers, until there is a better fix in Django stable.. So started this project to do some analysis: https://github.com/AndreaCrotti/django-migrations-analysis One interesting thing is that it looks like using sqlmigrate (taking some code from it) to generate the raw SQL and composing ti basically generates exactly the same final schema for the database. https://github.com/AndreaCrotti/django-migrations-analysis/blob/master/commands/sqlmigrate_all.py In general the question is, assuming all the settings are the same, can they differ under some conditions? I guess for example that using RunPython would cause issues in this case right? -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at https://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/acccdd2d-044c-4771-ae79-b99e89f0301b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Django migrations taking all the memory on Django 1.9
Ah ok thanks, well generally it's not a blocker, except that we first have
to upgrade to 1.10.
We have done a lot of migrations in the last few months and given the size
of the project it always takes a long time from one version to the other,
so just not sure when I'll be able to test these changes on our project..
On Tuesday, September 6, 2016 at 5:02:24 PM UTC+1, Tim Graham wrote:
>
> Per our support versions policy, Django 1.9.x is only receiving fixes for
> security and data loss issues. Migrations optimizations are unlikely to
> qualify for a backport to Django 1.10.x either. Hopefully that doesn't
> demotivate you from contributing an improvement.
>
>
> https://docs.djangoproject.com/en/dev/internals/security/#supported-versions
>
> On Monday, September 5, 2016 at 9:55:30 AM UTC-4, andrea crotti wrote:
>>
>> Thanks Marcus and others, this issue is in general getting worse and
>> worse.
>>
>> We did some more investigation and we should probably just help out the
>> work you are doing.
>>
>> Do you think it would be possible to apply these changes also on Django
>> 1.9.9 or it has to be from master/1.10 only?
>>
>> I also tried another approach:
>> - use "./manage.py sqlmigrate " to generate the SQL
>> - add in the generated SQL the INSERT to django_migrations
>> - run the SQL manually in the DB
>>
>> This actually works and could be automated, but I'm quite sure it's not a
>> good idea..
>> But what do you think could go wrong with this approach?
>>
>> I would say that as long as the dependencies are met and things are done
>> in the right order it might be safe enough, but of course we get into muddy
>> waters..
>>
>> On Monday, August 8, 2016 at 2:36:23 PM UTC+1, Tim Graham wrote:
>>>
>>> I think that error suggests that you disabled migrations for an app
>>> (something_else) that's a dependency of another app that doesn't have
>>> migrations disabled (something). That isn't permitted. I guess there's no
>>> good solution for your requirements as of now.
>>>
>>> On Monday, August 8, 2016 at 6:02:48 AM UTC-4, andrea crotti wrote:
>>>>
>>>> Since we also use --keepdb I thought we could simply avoid running all
>>>> the migrations, and maybe dynamically even skip all the apps that were not
>>>> changed at all.
>>>>
>>>> So I just tried something silly like this:
>>>>
>>>>
>>>> NO_MIGRATIONS = os.environ.get('NO_MIGRATIONS', 'false').lower() in
>>>> ('true', '1')
>>>> if NO_MIGRATIONS:
>>>> for app in INSTALLED_APPS:
>>>> if app not in MIGRATION_MODULES:
>>>> MIGRATION_MODULES[app] = None
>>>>
>>>>
>>>> But I get this error:
>>>>
>>>> django.db.migrations.exceptions.NodeNotFoundError: Migration
>>>> something.0001_initial dependencies reference nonexistent parent node
>>>> ('something_else', u'0001_initial')
>>>>
>>>> So I guess it's not as simple (and maybe that ticket is what that's
>>>> about, but sadly it's not even merged in master now from my understanding).
>>>>
>>>> PS. so to clarify our tests do rely on migrations (and there are some
>>>> data migrations too) but since we normally use --keepdb anyway we
>>>> definitively not need to run only a few migrations at a time max (and even
>>>> then sometimes it takes ages and uses all the RAM).
>>>>
>>>> On Saturday, August 6, 2016 at 4:03:50 PM UTC+1, Tim Graham wrote:
>>>>>
>>>>> As long as your tests don't rely on any data migrations, you can set
>>>>> MIGRATIONS_MODULES['app'] = None for all apps in a test settings file as
>>>>> described in
>>>>> https://docs.djangoproject.com/en/stable/ref/settings/#std:setting-MIGRATION_MODULES
>>>>>
>>>>> to disable migrations while testing. There's a ticket to make that a bit
>>>>> easier: https://code.djangoproject.com/ticket/25388
>>>>>
>>>>> On Saturday, August 6, 2016 at 6:07:04 AM UTC-4, andrea crotti wrote:
>>>>>>
>>>>>>
>>>>>> Ok great thanks for the answer Markus.
>>>>>> And yes I can try to help, it's quite a big issue for us..
>>>>>> Anything else we can do to improve the situation in the meanwhile?
>>>>>>
>>>>>> In general I was wondering why do we need to
Re: Django migrations taking all the memory on Django 1.9
Thanks Marcus and others, this issue is in general getting worse and worse.
We did some more investigation and we should probably just help out the
work you are doing.
Do you think it would be possible to apply these changes also on Django
1.9.9 or it has to be from master/1.10 only?
I also tried another approach:
- use "./manage.py sqlmigrate " to generate the SQL
- add in the generated SQL the INSERT to django_migrations
- run the SQL manually in the DB
This actually works and could be automated, but I'm quite sure it's not a
good idea..
But what do you think could go wrong with this approach?
I would say that as long as the dependencies are met and things are done in
the right order it might be safe enough, but of course we get into muddy
waters..
On Monday, August 8, 2016 at 2:36:23 PM UTC+1, Tim Graham wrote:
>
> I think that error suggests that you disabled migrations for an app
> (something_else) that's a dependency of another app that doesn't have
> migrations disabled (something). That isn't permitted. I guess there's no
> good solution for your requirements as of now.
>
> On Monday, August 8, 2016 at 6:02:48 AM UTC-4, andrea crotti wrote:
>>
>> Since we also use --keepdb I thought we could simply avoid running all
>> the migrations, and maybe dynamically even skip all the apps that were not
>> changed at all.
>>
>> So I just tried something silly like this:
>>
>>
>> NO_MIGRATIONS = os.environ.get('NO_MIGRATIONS', 'false').lower() in
>> ('true', '1')
>> if NO_MIGRATIONS:
>> for app in INSTALLED_APPS:
>> if app not in MIGRATION_MODULES:
>> MIGRATION_MODULES[app] = None
>>
>>
>> But I get this error:
>>
>> django.db.migrations.exceptions.NodeNotFoundError: Migration
>> something.0001_initial dependencies reference nonexistent parent node
>> ('something_else', u'0001_initial')
>>
>> So I guess it's not as simple (and maybe that ticket is what that's
>> about, but sadly it's not even merged in master now from my understanding).
>>
>> PS. so to clarify our tests do rely on migrations (and there are some
>> data migrations too) but since we normally use --keepdb anyway we
>> definitively not need to run only a few migrations at a time max (and even
>> then sometimes it takes ages and uses all the RAM).
>>
>> On Saturday, August 6, 2016 at 4:03:50 PM UTC+1, Tim Graham wrote:
>>>
>>> As long as your tests don't rely on any data migrations, you can set
>>> MIGRATIONS_MODULES['app'] = None for all apps in a test settings file as
>>> described in
>>> https://docs.djangoproject.com/en/stable/ref/settings/#std:setting-MIGRATION_MODULES
>>>
>>> to disable migrations while testing. There's a ticket to make that a bit
>>> easier: https://code.djangoproject.com/ticket/25388
>>>
>>> On Saturday, August 6, 2016 at 6:07:04 AM UTC-4, andrea crotti wrote:
>>>>
>>>>
>>>> Ok great thanks for the answer Markus.
>>>> And yes I can try to help, it's quite a big issue for us..
>>>> Anything else we can do to improve the situation in the meanwhile?
>>>>
>>>> In general I was wondering why do we need to handle all the "baggage"
>>>> of supporting backward migrations and doing things 100% properly while
>>>> running migrations just for tests.
>>>>
>>>> I think that in general if there was just a way to render all the SQL
>>>> that needs to be run statically for tests it would be great.
>>>>
>>>> On Friday, August 5, 2016 at 11:07:56 AM UTC+1, andrea crotti wrote:
>>>>>
>>>>> We have a very big Django project with > 100 apps and > 300 models.
>>>>> We had some massive issues with Django 1.8 and migrations, which would
>>>>> take forever and just take all the computer memory after a while.
>>>>>
>>>>> Now with Django 1.9 things improved, however we are again back with
>>>>> some extremely bad performances and massive memory usages (up to 8GB of
>>>>> RAM
>>>>> for example), sometimes just to run ONE single migration.
>>>>>
>>>>> It's not even entirely deterministic though sometimes killing the
>>>>> process and doing it again just works.
>>>>>
>>>>> I'm attaching the profile graph (done wtith gprof2dot) from running
>>>>> "./manage migrate" and one thing that clearly looks bad is that *render*
>>>>> for example is called 44355 times,
Re: Django migrations taking all the memory on Django 1.9
Since we also use --keepdb I thought we could simply avoid running all the
migrations, and maybe dynamically even skip all the apps that were not
changed at all.
So I just tried something silly like this:
NO_MIGRATIONS = os.environ.get('NO_MIGRATIONS', 'false').lower() in
('true', '1')
if NO_MIGRATIONS:
for app in INSTALLED_APPS:
if app not in MIGRATION_MODULES:
MIGRATION_MODULES[app] = None
But I get this error:
django.db.migrations.exceptions.NodeNotFoundError: Migration
something.0001_initial dependencies reference nonexistent parent node
('something_else', u'0001_initial')
So I guess it's not as simple (and maybe that ticket is what that's about,
but sadly it's not even merged in master now from my understanding).
PS. so to clarify our tests do rely on migrations (and there are some data
migrations too) but since we normally use --keepdb anyway we definitively
not need to run only a few migrations at a time max (and even then
sometimes it takes ages and uses all the RAM).
On Saturday, August 6, 2016 at 4:03:50 PM UTC+1, Tim Graham wrote:
>
> As long as your tests don't rely on any data migrations, you can set
> MIGRATIONS_MODULES['app'] = None for all apps in a test settings file as
> described in
> https://docs.djangoproject.com/en/stable/ref/settings/#std:setting-MIGRATION_MODULES
>
> to disable migrations while testing. There's a ticket to make that a bit
> easier: https://code.djangoproject.com/ticket/25388
>
> On Saturday, August 6, 2016 at 6:07:04 AM UTC-4, andrea crotti wrote:
>>
>>
>> Ok great thanks for the answer Markus.
>> And yes I can try to help, it's quite a big issue for us..
>> Anything else we can do to improve the situation in the meanwhile?
>>
>> In general I was wondering why do we need to handle all the "baggage" of
>> supporting backward migrations and doing things 100% properly while running
>> migrations just for tests.
>>
>> I think that in general if there was just a way to render all the SQL
>> that needs to be run statically for tests it would be great.
>>
>> On Friday, August 5, 2016 at 11:07:56 AM UTC+1, andrea crotti wrote:
>>>
>>> We have a very big Django project with > 100 apps and > 300 models.
>>> We had some massive issues with Django 1.8 and migrations, which would
>>> take forever and just take all the computer memory after a while.
>>>
>>> Now with Django 1.9 things improved, however we are again back with some
>>> extremely bad performances and massive memory usages (up to 8GB of RAM for
>>> example), sometimes just to run ONE single migration.
>>>
>>> It's not even entirely deterministic though sometimes killing the
>>> process and doing it again just works.
>>>
>>> I'm attaching the profile graph (done wtith gprof2dot) from running
>>> "./manage migrate" and one thing that clearly looks bad is that *render*
>>> for example is called 44355 times, which is definitively not normal.
>>>
>>> Any idea about about what we can do about this and what could be the
>>> problem?
>>> I have the impression that it's related with the amount of models and
>>> how they are interconnected, and mabye some caching would avoid all this
>>> extra computation.
>>>
>>> Noone else has similar issues?
>>> Thanks
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-users/9d7a4ea9-1683-4bc3-9ab4-54b82e0ae027%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Django migrations taking all the memory on Django 1.9
Ok great thanks for the answer Markus. And yes I can try to help, it's quite a big issue for us.. Anything else we can do to improve the situation in the meanwhile? In general I was wondering why do we need to handle all the "baggage" of supporting backward migrations and doing things 100% properly while running migrations just for tests. I think that in general if there was just a way to render all the SQL that needs to be run statically for tests it would be great. On Friday, August 5, 2016 at 11:07:56 AM UTC+1, andrea crotti wrote: > > We have a very big Django project with > 100 apps and > 300 models. > We had some massive issues with Django 1.8 and migrations, which would > take forever and just take all the computer memory after a while. > > Now with Django 1.9 things improved, however we are again back with some > extremely bad performances and massive memory usages (up to 8GB of RAM for > example), sometimes just to run ONE single migration. > > It's not even entirely deterministic though sometimes killing the process > and doing it again just works. > > I'm attaching the profile graph (done wtith gprof2dot) from running > "./manage migrate" and one thing that clearly looks bad is that *render* > for example is called 44355 times, which is definitively not normal. > > Any idea about about what we can do about this and what could be the > problem? > I have the impression that it's related with the amount of models and how > they are interconnected, and mabye some caching would avoid all this extra > computation. > > Noone else has similar issues? > Thanks > -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at https://groups.google.com/group/django-users. To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/98682ed2-90ae-4c2d-9e91-557fe01cb73b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Templates error on partial rendering
I have a case where we use django templates, but it really makes no sense to render the templates if a value is not passed in the contentx, and it should just fail.. I don't find anywhere how to force that though, I understand the default of failing silently but there should be a way also to fail making some noise, right? -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
