Re: Thoughts on diff-based migrations with django?

[Florian Apolloner]

Just for one migration; but there is the sqlmigrate management command 
which you can use as building base.

Cheers,
Florian

On Wednesday, May 2, 2018 at 7:07:27 AM UTC+2, djrobstep wrote:
>
> Bump! Is there a documented way to generate the from-scratch creation SQL 
> for the currently defined models? If not, is it possible to add something 
> that spits this out? Happy to take a look at implementing this.
>
>
> On Sunday, 25 February 2018 13:48:27 UTC+11, djrobstep wrote:
>>
>> Yes, I don't think it would make sense to include in django directly. 
>> Because the diffing approach works at the database directly, below the 
>> layer of the models, the two parts don't really need to be integrated. All 
>> that is really required for django to support this as an option is to 
>> provide a documented way to emit the from-scratch creation SQL for the 
>> currently defined models.
>>
>> As far as I can tell there used to be an admin command available for this 
>> in version 1.x but it was removed? Is there already another supported way 
>> to do this?
>>
>> Incidentally, I've used this approach on some larger projects with 
>> diverging branches and it worked pretty well. You do have to set some 
>> things up differently tho - keep meaning to write a blog post or something 
>> about this. The need to have access to a production schema is an additional 
>> hurdle, but well worth it for the additional testability and robustness, in 
>> my experience.
>>
>> On Wednesday, 21 February 2018 18:44:30 UTC+11, Andrew Godwin wrote:
>>>
>>> Hi,
>>>
>>> This was an approach we deliberately avoided during development of the 
>>> current migration system - it works poorly with diverging feature branches 
>>> and requires that your developers have access to production schema at all 
>>> times (and additionally that you don't have divergent production/staging/QA 
>>> deploys, like many environments do).
>>>
>>> I think it would be a fine addition as an optional third-party tool, but 
>>> it's not something that really makes sense to include in Django at this 
>>> point I think.
>>>
>>> Andrew
>>>
>>> On Mon, Feb 19, 2018 at 10:05 PM, djrobstep  wrote:
>>>
 Hi all,

 Some thoughts on schema migrations that may interest you. By way of 
 background, I'm the author of migra (https://github.com/djrobstep/migra), 
 a schema comparison/diff tool for Postgres.

 I initially wrote this tool because I wanted to be able to generate 
 migration scripts automatically, without needing historical migration 
 files.

 It means that you can do things like sync your database to your models 
 mostly automatically, and also explicitly test for a matching schema. 
 Instead of a chain of migration files you only ever need to keep track of 
 one, containing any pending changes).

 I've used this approach with success on a number of projects now, and 
 it seems to work pretty well. I also talked about this approach to 
 migrations at the most recent PostgresOpen (
 https://www.youtube.com/watch?v=xr498W8oMRo).

 The downside of this approach is that it's slightly more operationally 
 complex (you need have a temporary copy of your production schema 
 available 
 for comparison purposes, typically a --schema-only dump file) and also 
 that 
 it's postgres-only at this point. It's also conceptually quite different 
 to 
 the typical Django/Rails style of migrations which represents a barrier to 
 entry.

 In spite of this, I think doing migrations this way has some compelling 
 advantages, that are relevant to users of Django.

 I'm interested in your opinions on this topic as Django developers. 
 What do you think? Is it something worth offering as an option for 
 managing 
 Django migrations? If your feelings are positive, I'm happy to take a look 
 at extending the code accordingly.

 -- 
 You received this message because you are subscribed to the Google 
 Groups "Django developers (Contributions to Django itself)" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to django-develop...@googlegroups.com.
 To post to this group, send email to django-d...@googlegroups.com.
 Visit this group at https://groups.google.com/group/django-developers.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com
  
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop 

Re: Thoughts on diff-based migrations with django?

[djrobstep]

Bump! Is there a documented way to generate the from-scratch creation SQL 
for the currently defined models? If not, is it possible to add something 
that spits this out? Happy to take a look at implementing this.


On Sunday, 25 February 2018 13:48:27 UTC+11, djrobstep wrote:
>
> Yes, I don't think it would make sense to include in django directly. 
> Because the diffing approach works at the database directly, below the 
> layer of the models, the two parts don't really need to be integrated. All 
> that is really required for django to support this as an option is to 
> provide a documented way to emit the from-scratch creation SQL for the 
> currently defined models.
>
> As far as I can tell there used to be an admin command available for this 
> in version 1.x but it was removed? Is there already another supported way 
> to do this?
>
> Incidentally, I've used this approach on some larger projects with 
> diverging branches and it worked pretty well. You do have to set some 
> things up differently tho - keep meaning to write a blog post or something 
> about this. The need to have access to a production schema is an additional 
> hurdle, but well worth it for the additional testability and robustness, in 
> my experience.
>
> On Wednesday, 21 February 2018 18:44:30 UTC+11, Andrew Godwin wrote:
>>
>> Hi,
>>
>> This was an approach we deliberately avoided during development of the 
>> current migration system - it works poorly with diverging feature branches 
>> and requires that your developers have access to production schema at all 
>> times (and additionally that you don't have divergent production/staging/QA 
>> deploys, like many environments do).
>>
>> I think it would be a fine addition as an optional third-party tool, but 
>> it's not something that really makes sense to include in Django at this 
>> point I think.
>>
>> Andrew
>>
>> On Mon, Feb 19, 2018 at 10:05 PM, djrobstep  wrote:
>>
>>> Hi all,
>>>
>>> Some thoughts on schema migrations that may interest you. By way of 
>>> background, I'm the author of migra (https://github.com/djrobstep/migra), 
>>> a schema comparison/diff tool for Postgres.
>>>
>>> I initially wrote this tool because I wanted to be able to generate 
>>> migration scripts automatically, without needing historical migration files.
>>>
>>> It means that you can do things like sync your database to your models 
>>> mostly automatically, and also explicitly test for a matching schema. 
>>> Instead of a chain of migration files you only ever need to keep track of 
>>> one, containing any pending changes).
>>>
>>> I've used this approach with success on a number of projects now, and it 
>>> seems to work pretty well. I also talked about this approach to migrations 
>>> at the most recent PostgresOpen (
>>> https://www.youtube.com/watch?v=xr498W8oMRo).
>>>
>>> The downside of this approach is that it's slightly more operationally 
>>> complex (you need have a temporary copy of your production schema available 
>>> for comparison purposes, typically a --schema-only dump file) and also that 
>>> it's postgres-only at this point. It's also conceptually quite different to 
>>> the typical Django/Rails style of migrations which represents a barrier to 
>>> entry.
>>>
>>> In spite of this, I think doing migrations this way has some compelling 
>>> advantages, that are relevant to users of Django.
>>>
>>> I'm interested in your opinions on this topic as Django developers. What 
>>> do you think? Is it something worth offering as an option for managing 
>>> Django migrations? If your feelings are positive, I'm happy to take a look 
>>> at extending the code accordingly.
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com
>>>  
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 

Re: Thoughts on diff-based migrations with django?

[djrobstep]

Yes, I don't think it would make sense to include in django directly. 
Because the diffing approach works at the database directly, below the 
layer of the models, the two parts don't really need to be integrated. All 
that is really required for django to support this as an option is to 
provide a documented way to emit the from-scratch creation SQL for the 
currently defined models.

As far as I can tell there used to be an admin command available for this 
in version 1.x but it was removed? Is there already another supported way 
to do this?

Incidentally, I've used this approach on some larger projects with 
diverging branches and it worked pretty well. You do have to set some 
things up differently tho - keep meaning to write a blog post or something 
about this. The need to have access to a production schema is an additional 
hurdle, but well worth it for the additional testability and robustness, in 
my experience.

On Wednesday, 21 February 2018 18:44:30 UTC+11, Andrew Godwin wrote:
>
> Hi,
>
> This was an approach we deliberately avoided during development of the 
> current migration system - it works poorly with diverging feature branches 
> and requires that your developers have access to production schema at all 
> times (and additionally that you don't have divergent production/staging/QA 
> deploys, like many environments do).
>
> I think it would be a fine addition as an optional third-party tool, but 
> it's not something that really makes sense to include in Django at this 
> point I think.
>
> Andrew
>
> On Mon, Feb 19, 2018 at 10:05 PM, djrobstep  > wrote:
>
>> Hi all,
>>
>> Some thoughts on schema migrations that may interest you. By way of 
>> background, I'm the author of migra (https://github.com/djrobstep/migra), 
>> a schema comparison/diff tool for Postgres.
>>
>> I initially wrote this tool because I wanted to be able to generate 
>> migration scripts automatically, without needing historical migration files.
>>
>> It means that you can do things like sync your database to your models 
>> mostly automatically, and also explicitly test for a matching schema. 
>> Instead of a chain of migration files you only ever need to keep track of 
>> one, containing any pending changes).
>>
>> I've used this approach with success on a number of projects now, and it 
>> seems to work pretty well. I also talked about this approach to migrations 
>> at the most recent PostgresOpen (
>> https://www.youtube.com/watch?v=xr498W8oMRo).
>>
>> The downside of this approach is that it's slightly more operationally 
>> complex (you need have a temporary copy of your production schema available 
>> for comparison purposes, typically a --schema-only dump file) and also that 
>> it's postgres-only at this point. It's also conceptually quite different to 
>> the typical Django/Rails style of migrations which represents a barrier to 
>> entry.
>>
>> In spite of this, I think doing migrations this way has some compelling 
>> advantages, that are relevant to users of Django.
>>
>> I'm interested in your opinions on this topic as Django developers. What 
>> do you think? Is it something worth offering as an option for managing 
>> Django migrations? If your feelings are positive, I'm happy to take a look 
>> at extending the code accordingly.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f17d9d96-db18-4501-ad10-f174bb1dc468%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Thoughts on diff-based migrations with django?

[Andrew Godwin]

Hi,

This was an approach we deliberately avoided during development of the
current migration system - it works poorly with diverging feature branches
and requires that your developers have access to production schema at all
times (and additionally that you don't have divergent production/staging/QA
deploys, like many environments do).

I think it would be a fine addition as an optional third-party tool, but
it's not something that really makes sense to include in Django at this
point I think.

Andrew

On Mon, Feb 19, 2018 at 10:05 PM, djrobstep  wrote:

> Hi all,
>
> Some thoughts on schema migrations that may interest you. By way of
> background, I'm the author of migra (https://github.com/djrobstep/migra),
> a schema comparison/diff tool for Postgres.
>
> I initially wrote this tool because I wanted to be able to generate
> migration scripts automatically, without needing historical migration files.
>
> It means that you can do things like sync your database to your models
> mostly automatically, and also explicitly test for a matching schema.
> Instead of a chain of migration files you only ever need to keep track of
> one, containing any pending changes).
>
> I've used this approach with success on a number of projects now, and it
> seems to work pretty well. I also talked about this approach to migrations
> at the most recent PostgresOpen (https://www.youtube.com/
> watch?v=xr498W8oMRo).
>
> The downside of this approach is that it's slightly more operationally
> complex (you need have a temporary copy of your production schema available
> for comparison purposes, typically a --schema-only dump file) and also that
> it's postgres-only at this point. It's also conceptually quite different to
> the typical Django/Rails style of migrations which represents a barrier to
> entry.
>
> In spite of this, I think doing migrations this way has some compelling
> advantages, that are relevant to users of Django.
>
> I'm interested in your opinions on this topic as Django developers. What
> do you think? Is it something worth offering as an option for managing
> Django migrations? If your feelings are positive, I'm happy to take a look
> at extending the code accordingly.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/5f84157b-071e-42c3-90dd-
> 6f9e3df48ab3%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFwN1urzK%3DwWT0SfS32sg_YhcdJnG_2h9Cbp8u2Uouzs6LaWFg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Thoughts on diff-based migrations with django?

[djrobstep]

Yes, this approach certainly doesn't magically solve data migration issues, 
however the data migrations question has been a common one.

The nature of data migrations is that they are very one-off and context 
dependent and so still something you'd need to script up manually. You 
might use the diff tool to generate the migration script required but then 
edit it to add any queries needed for carrying out the data migration.

And doing things this way, there's nothing stopping you continuing to chain 
multiple migration scripts together to support multiple versions in the 
context of a reusable app.

On Tuesday, 20 February 2018 23:27:31 UTC+11, Curtis Maloney wrote:
>
> Hi, 
>
> I must say your idea intrigues me... and I'd certainly like to discuss 
> it further. 
>
> The first thought that comes to mind is how would such a system handle 
> data migrations? 
>
> Being able to leap from one schema state to another is great, buy 
> typically in the life-cycle of a reusable app it's required to also 
> migrate old data into new schema. 
>
> This also happens within project-specific apps. 
>
> -- 
> Curtis 
>
>
> On 02/20/2018 05:05 PM, djrobstep wrote: 
> > Hi all, 
> > 
> > Some thoughts on schema migrations that may interest you. By way of 
> > background, I'm the author of migra 
> > (https://github.com/djrobstep/migra), a schema comparison/diff tool for 
> > Postgres. 
> > 
> > I initially wrote this tool because I wanted to be able to generate 
> > migration scripts automatically, without needing historical migration 
> files. 
> > 
> > It means that you can do things like sync your database to your models 
> > mostly automatically, and also explicitly test for a matching schema. 
> > Instead of a chain of migration files you only ever need to keep track 
> > of one, containing any pending changes). 
> > 
> > I've used this approach with success on a number of projects now, and it 
> > seems to work pretty well. I also talked about this approach to 
> > migrations at the most recent PostgresOpen 
> > (https://www.youtube.com/watch?v=xr498W8oMRo). 
> > 
> > The downside of this approach is that it's slightly more operationally 
> > complex (you need have a temporary copy of your production schema 
> > available for comparison purposes, typically a --schema-only dump file) 
> > and also that it's postgres-only at this point. It's also conceptually 
> > quite different to the typical Django/Rails style of migrations which 
> > represents a barrier to entry. 
> > 
> > In spite of this, I think doing migrations this way has some compelling 
> > advantages, that are relevant to users of Django. 
> > 
> > I'm interested in your opinions on this topic as Django developers. What 
> > do you think? Is it something worth offering as an option for managing 
> > Django migrations? If your feelings are positive, I'm happy to take a 
> > look at extending the code accordingly. 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> > Groups "Django developers (Contributions to Django itself)" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> > an email to django-develop...@googlegroups.com  
> > . 
> > To post to this group, send email to django-d...@googlegroups.com 
>  
> > . 
> > Visit this group at https://groups.google.com/group/django-developers. 
> > To view this discussion on the web visit 
> > 
> https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com
>  
> > <
> https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com?utm_medium=email_source=footer>.
>  
>
> > For more options, visit https://groups.google.com/d/optout. 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/62c00b16-a854-4c49-a5c9-7e83dc0c9a2a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Thoughts on diff-based migrations with django?

[Curtis Maloney]

Hi,

I must say your idea intrigues me... and I'd certainly like to discuss 
it further.


The first thought that comes to mind is how would such a system handle 
data migrations?


Being able to leap from one schema state to another is great, buy 
typically in the life-cycle of a reusable app it's required to also 
migrate old data into new schema.


This also happens within project-specific apps.

--
Curtis


On 02/20/2018 05:05 PM, djrobstep wrote:

Hi all,

Some thoughts on schema migrations that may interest you. By way of 
background, I'm the author of migra 
(https://github.com/djrobstep/migra), a schema comparison/diff tool for 
Postgres.


I initially wrote this tool because I wanted to be able to generate 
migration scripts automatically, without needing historical migration files.


It means that you can do things like sync your database to your models 
mostly automatically, and also explicitly test for a matching schema. 
Instead of a chain of migration files you only ever need to keep track 
of one, containing any pending changes).


I've used this approach with success on a number of projects now, and it 
seems to work pretty well. I also talked about this approach to 
migrations at the most recent PostgresOpen 
(https://www.youtube.com/watch?v=xr498W8oMRo).


The downside of this approach is that it's slightly more operationally 
complex (you need have a temporary copy of your production schema 
available for comparison purposes, typically a --schema-only dump file) 
and also that it's postgres-only at this point. It's also conceptually 
quite different to the typical Django/Rails style of migrations which 
represents a barrier to entry.


In spite of this, I think doing migrations this way has some compelling 
advantages, that are relevant to users of Django.


I'm interested in your opinions on this topic as Django developers. What 
do you think? Is it something worth offering as an option for managing 
Django migrations? If your feelings are positive, I'm happy to take a 
look at extending the code accordingly.


--
You received this message because you are subscribed to the Google 
Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to django-developers+unsubscr...@googlegroups.com 
.
To post to this group, send email to django-developers@googlegroups.com 
.

Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com 
.

For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4ef38de9-780c-e2f9-9ce2-5d9621d6bf64%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.

Thoughts on diff-based migrations with django?

[djrobstep]

Hi all,

Some thoughts on schema migrations that may interest you. By way of 
background, I'm the author of migra (https://github.com/djrobstep/migra), a 
schema comparison/diff tool for Postgres.

I initially wrote this tool because I wanted to be able to generate 
migration scripts automatically, without needing historical migration files.

It means that you can do things like sync your database to your models 
mostly automatically, and also explicitly test for a matching schema. 
Instead of a chain of migration files you only ever need to keep track of 
one, containing any pending changes).

I've used this approach with success on a number of projects now, and it 
seems to work pretty well. I also talked about this approach to migrations 
at the most recent PostgresOpen (https://www.youtube.com/watch?v=xr498W8oMRo
).

The downside of this approach is that it's slightly more operationally 
complex (you need have a temporary copy of your production schema available 
for comparison purposes, typically a --schema-only dump file) and also that 
it's postgres-only at this point. It's also conceptually quite different to 
the typical Django/Rails style of migrations which represents a barrier to 
entry.

In spite of this, I think doing migrations this way has some compelling 
advantages, that are relevant to users of Django.

I'm interested in your opinions on this topic as Django developers. What do 
you think? Is it something worth offering as an option for managing Django 
migrations? If your feelings are positive, I'm happy to take a look at 
extending the code accordingly.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5f84157b-071e-42c3-90dd-6f9e3df48ab3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.