Re: Non-atomic migrations in Django?
Thanks for your feedback and support. I've opened a ticket and pull request: https://code.djangoproject.com/ticket/25833 Am Dienstag, 24. November 2015 21:05:04 UTC+1 schrieb Markus Holtermann: > > Hi Ludwig, > > the API looks clean and sensible to me. +1 for getting that into 1.10 > > /Markus > > On November 25, 2015 12:00:59 AM GMT+10:00, "Ludwig Hähne" < > lha...@gmail.com > wrote: > >Hi all, > > > >I'd like to get your feedback on supporting non-atomic migrations in > >Django. > > > >Database transactions are always wrapped in transactions (on DBs that > >support transactional DDL like Postgres). Generally this is a good > >idea. > > > >However, one can't do batched updates in data migrations which is > >essential > >for performing changes on big tables on a live DB [1]. It's also not > >possible to create indexes concurrently on Postgres from inside a > >transaction. > > > >Therefore, I'd like to have support for non-atomic migrations in Django > > > >because it's pretty messy to work around not having proper support for > >that > >[2]. > > > >Here's a proof-of-concept implementation [3] of exempting specific > >migrations from being wrapped in a transaction by setting `atomic = > >False` > >on the migration: > > > > > https://github.com/django/django/compare/master...Ableton:non-atomic-migrations > > > > >Do you agree that non-atomic migrations should be supported by Django? > > > >Is setting an `atomic` property on a migration a good API for that? > > > >If there's a chance this will be merged, I'd add documentation, > >incorporate > >your feedback, and open a ticket or PR. > > > >Thanks, > >Ludwig > > > >[1] http://pankrat.github.io/2015/django-migrations-without-downtimes/ > >[2] > > > http://stackoverflow.com/questions/31247810/commit-manually-in-django-data-migration > > >[3] > > > https://github.com/django/django/compare/master...Ableton:non-atomic-migrations > > > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/d3668bf1-1783-47f8-80f8-c6a33752eace%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Non-atomic migrations in Django?
Hi Ludwig, the API looks clean and sensible to me. +1 for getting that into 1.10 /Markus On November 25, 2015 12:00:59 AM GMT+10:00, "Ludwig Hähne" <lhae...@gmail.com> wrote: >Hi all, > >I'd like to get your feedback on supporting non-atomic migrations in >Django. > >Database transactions are always wrapped in transactions (on DBs that >support transactional DDL like Postgres). Generally this is a good >idea. > >However, one can't do batched updates in data migrations which is >essential >for performing changes on big tables on a live DB [1]. It's also not >possible to create indexes concurrently on Postgres from inside a >transaction. > >Therefore, I'd like to have support for non-atomic migrations in Django > >because it's pretty messy to work around not having proper support for >that >[2]. > >Here's a proof-of-concept implementation [3] of exempting specific >migrations from being wrapped in a transaction by setting `atomic = >False` >on the migration: > >https://github.com/django/django/compare/master...Ableton:non-atomic-migrations > >Do you agree that non-atomic migrations should be supported by Django? > >Is setting an `atomic` property on a migration a good API for that? > >If there's a chance this will be merged, I'd add documentation, >incorporate >your feedback, and open a ticket or PR. > >Thanks, >Ludwig > >[1] http://pankrat.github.io/2015/django-migrations-without-downtimes/ >[2] >http://stackoverflow.com/questions/31247810/commit-manually-in-django-data-migration >[3] >https://github.com/django/django/compare/master...Ableton:non-atomic-migrations -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/704E30F0-A249-45CD-B084-FCECCF711E88%40markusholtermann.eu. For more options, visit https://groups.google.com/d/optout.
Re: Non-atomic migrations in Django?
I also agree this looks sensible - I think South even had an attribute like this on the migration class, it just never got ported over. +1 On Tue, Nov 24, 2015 at 8:19 AM, Marc Tamlyn <marc.tam...@gmail.com> wrote: > +1 to idea and API. I've wished I had this recently - even if it's just so > I can check up on the progress of a slow running data generation migration > so it flushes the data every few 100 records. > > On 24 November 2015 at 16:07, Aymeric Augustin < > aymeric.augus...@polytechnique.org> wrote: > >> I like the API as well. Surprisingly, I couldn't find a Trac ticket about >> this. >> >> -- >> Aymeric. >> >> 2015-11-24 16:39 GMT+01:00 Anssi Kääriäinen <akaar...@gmail.com>: >> >>> I don't see any problem with optional non-transactional migrations. So, >>> +1 for the idea and API. I haven't looked at the implementation, so no >>> comments about that. >>> >>> - Anssi >>> >>> >>> On Tuesday, November 24, 2015, Ludwig Hähne <lhae...@gmail.com> wrote: >>> >>>> Hi all, >>>> >>>> I'd like to get your feedback on supporting non-atomic migrations in >>>> Django. >>>> >>>> Database transactions are always wrapped in transactions (on DBs that >>>> support transactional DDL like Postgres). Generally this is a good idea. >>>> >>>> However, one can't do batched updates in data migrations which is >>>> essential for performing changes on big tables on a live DB [1]. It's also >>>> not possible to create indexes concurrently on Postgres from inside a >>>> transaction. >>>> >>>> Therefore, I'd like to have support for non-atomic migrations in Django >>>> because it's pretty messy to work around not having proper support for that >>>> [2]. >>>> >>>> Here's a proof-of-concept implementation [3] of exempting specific >>>> migrations from being wrapped in a transaction by setting `atomic = False` >>>> on the migration: >>>> >>>> >>>> https://github.com/django/django/compare/master...Ableton:non-atomic-migrations >>>> >>>> Do you agree that non-atomic migrations should be supported by Django? >>>> >>>> Is setting an `atomic` property on a migration a good API for that? >>>> >>>> If there's a chance this will be merged, I'd add documentation, >>>> incorporate your feedback, and open a ticket or PR. >>>> >>>> Thanks, >>>> Ludwig >>>> >>>> [1] http://pankrat.github.io/2015/django-migrations-without-downtimes/ >>>> [2] >>>> http://stackoverflow.com/questions/31247810/commit-manually-in-django-data-migration >>>> [3] >>>> https://github.com/django/django/compare/master...Ableton:non-atomic-migrations >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Django developers (Contributions to Django itself)" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to django-developers+unsubscr...@googlegroups.com. >>>> To post to this group, send email to django-developers@googlegroups.com >>>> . >>>> Visit this group at http://groups.google.com/group/django-developers. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/django-developers/c062736d-2ebc-42cd-83a5-fe4d064cb24e%40googlegroups.com >>>> <https://groups.google.com/d/msgid/django-developers/c062736d-2ebc-42cd-83a5-fe4d064cb24e%40googlegroups.com?utm_medium=email_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Django developers (Contributions to Django itself)" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to django-developers+unsubscr...@googlegroups.com. >>> To post to this group, send email to django-developers@googlegroups.com. >>> Visit this group at http://groups.google.com/group/django-developers. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/django-developers/CALMtK1GDj0_wheW5dHEdp1ZLYcXa-6OpxfHgcnHqWtMcT9YG0g%40mail.gmail.com >>> <https://groups.google.com/d/msgid/django-developers/CALMtK1
Re: Non-atomic migrations in Django?
+1 to idea and API. I've wished I had this recently - even if it's just so I can check up on the progress of a slow running data generation migration so it flushes the data every few 100 records. On 24 November 2015 at 16:07, Aymeric Augustin < aymeric.augus...@polytechnique.org> wrote: > I like the API as well. Surprisingly, I couldn't find a Trac ticket about > this. > > -- > Aymeric. > > 2015-11-24 16:39 GMT+01:00 Anssi Kääriäinen <akaar...@gmail.com>: > >> I don't see any problem with optional non-transactional migrations. So, >> +1 for the idea and API. I haven't looked at the implementation, so no >> comments about that. >> >> - Anssi >> >> >> On Tuesday, November 24, 2015, Ludwig Hähne <lhae...@gmail.com> wrote: >> >>> Hi all, >>> >>> I'd like to get your feedback on supporting non-atomic migrations in >>> Django. >>> >>> Database transactions are always wrapped in transactions (on DBs that >>> support transactional DDL like Postgres). Generally this is a good idea. >>> >>> However, one can't do batched updates in data migrations which is >>> essential for performing changes on big tables on a live DB [1]. It's also >>> not possible to create indexes concurrently on Postgres from inside a >>> transaction. >>> >>> Therefore, I'd like to have support for non-atomic migrations in Django >>> because it's pretty messy to work around not having proper support for that >>> [2]. >>> >>> Here's a proof-of-concept implementation [3] of exempting specific >>> migrations from being wrapped in a transaction by setting `atomic = False` >>> on the migration: >>> >>> >>> https://github.com/django/django/compare/master...Ableton:non-atomic-migrations >>> >>> Do you agree that non-atomic migrations should be supported by Django? >>> >>> Is setting an `atomic` property on a migration a good API for that? >>> >>> If there's a chance this will be merged, I'd add documentation, >>> incorporate your feedback, and open a ticket or PR. >>> >>> Thanks, >>> Ludwig >>> >>> [1] http://pankrat.github.io/2015/django-migrations-without-downtimes/ >>> [2] >>> http://stackoverflow.com/questions/31247810/commit-manually-in-django-data-migration >>> [3] >>> https://github.com/django/django/compare/master...Ableton:non-atomic-migrations >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Django developers (Contributions to Django itself)" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to django-developers+unsubscr...@googlegroups.com. >>> To post to this group, send email to django-developers@googlegroups.com. >>> Visit this group at http://groups.google.com/group/django-developers. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/django-developers/c062736d-2ebc-42cd-83a5-fe4d064cb24e%40googlegroups.com >>> <https://groups.google.com/d/msgid/django-developers/c062736d-2ebc-42cd-83a5-fe4d064cb24e%40googlegroups.com?utm_medium=email_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django developers (Contributions to Django itself)" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to django-developers+unsubscr...@googlegroups.com. >> To post to this group, send email to django-developers@googlegroups.com. >> Visit this group at http://groups.google.com/group/django-developers. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-developers/CALMtK1GDj0_wheW5dHEdp1ZLYcXa-6OpxfHgcnHqWtMcT9YG0g%40mail.gmail.com >> <https://groups.google.com/d/msgid/django-developers/CALMtK1GDj0_wheW5dHEdp1ZLYcXa-6OpxfHgcnHqWtMcT9YG0g%40mail.gmail.com?utm_medium=email_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Aymeric. > > -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@go
Re: Non-atomic migrations in Django?
I like the API as well. Surprisingly, I couldn't find a Trac ticket about this. -- Aymeric. 2015-11-24 16:39 GMT+01:00 Anssi Kääriäinen <akaar...@gmail.com>: > I don't see any problem with optional non-transactional migrations. So, +1 > for the idea and API. I haven't looked at the implementation, so no > comments about that. > > - Anssi > > > On Tuesday, November 24, 2015, Ludwig Hähne <lhae...@gmail.com> wrote: > >> Hi all, >> >> I'd like to get your feedback on supporting non-atomic migrations in >> Django. >> >> Database transactions are always wrapped in transactions (on DBs that >> support transactional DDL like Postgres). Generally this is a good idea. >> >> However, one can't do batched updates in data migrations which is >> essential for performing changes on big tables on a live DB [1]. It's also >> not possible to create indexes concurrently on Postgres from inside a >> transaction. >> >> Therefore, I'd like to have support for non-atomic migrations in Django >> because it's pretty messy to work around not having proper support for that >> [2]. >> >> Here's a proof-of-concept implementation [3] of exempting specific >> migrations from being wrapped in a transaction by setting `atomic = False` >> on the migration: >> >> >> https://github.com/django/django/compare/master...Ableton:non-atomic-migrations >> >> Do you agree that non-atomic migrations should be supported by Django? >> >> Is setting an `atomic` property on a migration a good API for that? >> >> If there's a chance this will be merged, I'd add documentation, >> incorporate your feedback, and open a ticket or PR. >> >> Thanks, >> Ludwig >> >> [1] http://pankrat.github.io/2015/django-migrations-without-downtimes/ >> [2] >> http://stackoverflow.com/questions/31247810/commit-manually-in-django-data-migration >> [3] >> https://github.com/django/django/compare/master...Ableton:non-atomic-migrations >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Django developers (Contributions to Django itself)" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to django-developers+unsubscr...@googlegroups.com. >> To post to this group, send email to django-developers@googlegroups.com. >> Visit this group at http://groups.google.com/group/django-developers. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/django-developers/c062736d-2ebc-42cd-83a5-fe4d064cb24e%40googlegroups.com >> <https://groups.google.com/d/msgid/django-developers/c062736d-2ebc-42cd-83a5-fe4d064cb24e%40googlegroups.com?utm_medium=email_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at http://groups.google.com/group/django-developers. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/CALMtK1GDj0_wheW5dHEdp1ZLYcXa-6OpxfHgcnHqWtMcT9YG0g%40mail.gmail.com > <https://groups.google.com/d/msgid/django-developers/CALMtK1GDj0_wheW5dHEdp1ZLYcXa-6OpxfHgcnHqWtMcT9YG0g%40mail.gmail.com?utm_medium=email_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Aymeric. -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CANE-7mUiytuj26np%3DzP-kX4fHmPv3i0u7Hn-a8GTGuHk4sF3qw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Non-atomic migrations in Django?
I don't see any problem with optional non-transactional migrations. So, +1 for the idea and API. I haven't looked at the implementation, so no comments about that. - Anssi On Tuesday, November 24, 2015, Ludwig Hähne <lhae...@gmail.com> wrote: > Hi all, > > I'd like to get your feedback on supporting non-atomic migrations in > Django. > > Database transactions are always wrapped in transactions (on DBs that > support transactional DDL like Postgres). Generally this is a good idea. > > However, one can't do batched updates in data migrations which is > essential for performing changes on big tables on a live DB [1]. It's also > not possible to create indexes concurrently on Postgres from inside a > transaction. > > Therefore, I'd like to have support for non-atomic migrations in Django > because it's pretty messy to work around not having proper support for that > [2]. > > Here's a proof-of-concept implementation [3] of exempting specific > migrations from being wrapped in a transaction by setting `atomic = False` > on the migration: > > > https://github.com/django/django/compare/master...Ableton:non-atomic-migrations > > Do you agree that non-atomic migrations should be supported by Django? > > Is setting an `atomic` property on a migration a good API for that? > > If there's a chance this will be merged, I'd add documentation, > incorporate your feedback, and open a ticket or PR. > > Thanks, > Ludwig > > [1] http://pankrat.github.io/2015/django-migrations-without-downtimes/ > [2] > http://stackoverflow.com/questions/31247810/commit-manually-in-django-data-migration > [3] > https://github.com/django/django/compare/master...Ableton:non-atomic-migrations > > -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com > <javascript:_e(%7B%7D,'cvml','django-developers%2bunsubscr...@googlegroups.com');> > . > To post to this group, send email to django-developers@googlegroups.com > <javascript:_e(%7B%7D,'cvml','django-developers@googlegroups.com');>. > Visit this group at http://groups.google.com/group/django-developers. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/c062736d-2ebc-42cd-83a5-fe4d064cb24e%40googlegroups.com > <https://groups.google.com/d/msgid/django-developers/c062736d-2ebc-42cd-83a5-fe4d064cb24e%40googlegroups.com?utm_medium=email_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CALMtK1GDj0_wheW5dHEdp1ZLYcXa-6OpxfHgcnHqWtMcT9YG0g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Non-atomic migrations in Django?
Hi all, I'd like to get your feedback on supporting non-atomic migrations in Django. Database transactions are always wrapped in transactions (on DBs that support transactional DDL like Postgres). Generally this is a good idea. However, one can't do batched updates in data migrations which is essential for performing changes on big tables on a live DB [1]. It's also not possible to create indexes concurrently on Postgres from inside a transaction. Therefore, I'd like to have support for non-atomic migrations in Django because it's pretty messy to work around not having proper support for that [2]. Here's a proof-of-concept implementation [3] of exempting specific migrations from being wrapped in a transaction by setting `atomic = False` on the migration: https://github.com/django/django/compare/master...Ableton:non-atomic-migrations Do you agree that non-atomic migrations should be supported by Django? Is setting an `atomic` property on a migration a good API for that? If there's a chance this will be merged, I'd add documentation, incorporate your feedback, and open a ticket or PR. Thanks, Ludwig [1] http://pankrat.github.io/2015/django-migrations-without-downtimes/ [2] http://stackoverflow.com/questions/31247810/commit-manually-in-django-data-migration [3] https://github.com/django/django/compare/master...Ableton:non-atomic-migrations -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/c062736d-2ebc-42cd-83a5-fe4d064cb24e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
