Re: Migrate a many-to-many field to an explicit intermediate ("through") model?

[Carsten Fuchs]

Hi all,

just wanted to let you know that I was successful!

I followed Gergely's final recipe, using RunPython for the data migrations and separate 
migration files for each of the steps.


For step 3 (the deletion of the ManyToManyField) I temporarily had to remove the 
django.contrib.admin from the INSTALLED_APPS, because I was referring to the old 
ManyToManyField in my app's admin.py setup, which in turn was processed while running 
`./manage.py makemigrations`.


Overall, this worked very well.

Many thanks to you both for your help!
:-)

Am 23.07.2015 um 22:37 schrieb Carl Meyer:

Meanwhile it might worth a question to the devs (or a more thorough
search on the topic) to understand why you can't switch from a simple
m2m field to one with a through option.


Just because it's a bit complex to implement, and nobody has implemented
it yet.


Well, if I understand this correctly, https://code.djangoproject.com/ticket/23034 has a 
reviewed PR that implements this, but unfortunately the original author has not updated 
the PR for several months...


Best regards,
Carsten

--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/55B2814E.1090604%40cafu.de.
For more options, visit https://groups.google.com/d/optout.

Re: Migrate a many-to-many field to an explicit intermediate ("through") model?

[Carl Meyer]

Hi Gergely,

On 07/23/2015 02:27 PM, Gergely Polonkai wrote:
> Yes, you are right, my attempt is not the solution to your problem; it
> seems that this m2m field really cannot be modified like this. With some
> slight modifications, though, it may be.
> 
> 1) create the through table
> 2) migrate data with RunPython — if you want to be portable, stay with
> RunPython instead of RunSQL

Yes, this is true. Don't use RunSQL if you want your code to be portable
between database backends. (For most of my non-reusable-apps code, I
take great advantage of Postgres-specific features and don't care at all
about cross-database portability.)

> 3) delete the ManyToManyField
> 4) recreate the ManyToManyField with the through option
> 
> All this can (and I think, should) go into the same migration file.

Hmm. I'm almost sure I've had problems in the past with trying to do
schema alterations and data migration in the same migration file (thus
same transaction), but with a simple test just now it seems to work OK
on Postgres. I'll have to start trying to combine migrations like this
more often and see how it goes.

> Meanwhile it might worth a question to the devs (or a more thorough
> search on the topic) to understand why you can't switch from a simple
> m2m field to one with a through option.

Just because it's a bit complex to implement, and nobody has implemented
it yet.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/55B15070.8000707%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature

Re: Migrate a many-to-many field to an explicit intermediate ("through") model?

[Gergely Polonkai]

Hello,

I'm sorry, I was not around my mailbox lately.

Yes, you are right, my attempt is not the solution to your problem; it
seems that this m2m field really cannot be modified like this. With some
slight modifications, though, it may be.

1) create the through table
2) migrate data with RunPython — if you want to be portable, stay with
RunPython instead of RunSQL
3) delete the ManyToManyField
4) recreate the ManyToManyField with the through option

All this can (and I think, should) go into the same migration file.
Meanwhile it might worth a question to the devs (or a more thorough search
on the topic) to understand why you can't switch from a simple m2m field to
one with a through option.

Best,
Gergely
On 23 Jul 2015 19:08, "Carsten Fuchs"  wrote:

> Hi Carl,
>
> Am 23.07.2015 um 18:28 schrieb Carl Meyer:
>
>> Overall I think it might be simpler to go with your initial approach. I'd
>> first rename
>> the old field to some other temporary name (you don't have to update any
>> other code to
>> use this name, as you'll commit all of these migrations in one commit, so
>> nothing but
>> the following data migration ever needs to know anything about this
>> temporary name),
>>
>
> Thanks for clarifying the details regarding renaming! With this, this is
> the approach that I feel the most comfortable with and will try now.
>
> Overall, many thanks for your thoughts and your detailed reply, it's very
> much appreciated and helps a lot!
>
> Best regards,
> Carsten
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/55B11F7B.1010805%40cafu.de.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CACczBU%2BYcmaxdqr7q-i%3DY1Xe3U%2BE0BqUF%2BqbBWFhthQoWFOg7w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Migrate a many-to-many field to an explicit intermediate ("through") model?

[Carsten Fuchs]

Hi Carl,

Am 23.07.2015 um 18:28 schrieb Carl Meyer:

Overall I think it might be simpler to go with your initial approach. I'd first 
rename
the old field to some other temporary name (you don't have to update any other 
code to
use this name, as you'll commit all of these migrations in one commit, so 
nothing but
the following data migration ever needs to know anything about this temporary 
name),


Thanks for clarifying the details regarding renaming! With this, this is the approach 
that I feel the most comfortable with and will try now.


Overall, many thanks for your thoughts and your detailed reply, it's very much 
appreciated and helps a lot!


Best regards,
Carsten

--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/55B11F7B.1010805%40cafu.de.
For more options, visit https://groups.google.com/d/optout.

Re: Migrate a many-to-many field to an explicit intermediate ("through") model?

[Carl Meyer]

Hi Carsten,

On Thursday, July 16, 2015 at 12:44:55 PM UTC-6, Carsten Fuchs wrote:
>
> Am 2015-07-16 um 18:15 schrieb Gergely Polonkai: 
> > 1) create the "through" model, add it to the migration file 
> > 2) migrate all m2m instances (e.g. iterate over all Bereich objects then 
> > iterate through its users, creating a UserBereichAssignment object for 
> > each (all this done in a migrations.RunPython call) 
>
> Ok. I've never used migrations.RunPython before, but I'm quite confident 
> that I can manage it. 
>

You could also use RunSQL instead of RunPython for this data migration step 
- personally I often find that easier (and almost always more efficient) 
for a simple data migration, since all you need is a SQL query to copy some 
data from one table to another, you don't need any Python conveniences or 
model instances along the way.

But either can work, it really just depends how comfortable you are with 
raw SQL.
 

> Would the migration for step 2) be a separate migration file from step 
> 1), or is the migration file of step 1) edited to cover step 2) as well? 
>

Separate migration. In general you can't put schema alterations and data 
migrations in the same migration file, because then they'll try to run in 
the same transaction, and PostgreSQL at least doesn't like that.

In general, for complex changes, this three-step dance (one migration to 
add the new field/table, a second migration to copy the data, and a third 
migration to remove the old field/table) is very common and the right way 
to do things.
 

> > 3) change the m2m field by adding the through option, and add this 
> > change to the migrations file. 
>
> Same question: Is this supposed to go into a separate migration file or 
> into the common migrations file from above? 
>
> And won't this last step trigger the same error as before? ("... you 
> cannot alter to or from M2M fields, or add or remove through= on M2M 
> fields") ? 
>

This part I'm not sure about without trying it. I'm honestly not sure what 
exactly Gergely was recommending. Based on my reading of the code, the 
error is raised by the schema-editor backend, meaning if you try an 
`AlterField` with this change, you'd hit the error.

Another possibility might be to use the `SeparateDatabaseAndState` 
operation to create a migration that has no effect on the schema, but just 
updates the Python state for the field. Since you've already made the 
necessary db schema changes simply by adding the through model itself, this 
should work fine.

You could also go back and edit the field definition in whichever migration 
initially created it. This would probably work, but it would cause problems 
for any `RunPython` migrations since then that used that field (because now 
they'd try to use the through table instead), including your own data 
migration that you just created prior to this. So probably not a good 
option.
 

> (Not so important, but out of curiosity: This won't get us rid of the 
> old, implicit intermediate table, will it?) 
>

No. You could add another RunSQL migration to remove this table using raw 
SQL.

Overall I think it might be simpler to go with your initial approach. I'd 
first rename the old field to some other temporary name (you don't have to 
update any other code to use this name, as you'll commit all of these 
migrations in one commit, so nothing but the following data migration ever 
needs to know anything about this temporary name), then add the new field 
with through table (using the original field name), then migrate data 
(using either RunPython or RunSQL), then remove the old field. I think this 
will work fine, and it should also remove the old auto-created through 
table.

Good luck! I think this is definitely an area where the migrations system 
could help a bit more.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/e987cf79-358d-4fdc-bac5-5131a19a25c1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Migrate a many-to-many field to an explicit intermediate ("through") model?

[Carsten Fuchs]

Am 16.07.2015 um 16:05 schrieb Carsten Fuchs:

Alas... are there any viable alternatives to this?
I'd be very grateful for any hint!   :-)



Anyone please?

Best regards,
Carsten

--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/55B10E2A.7060809%40cafu.de.
For more options, visit https://groups.google.com/d/optout.

Re: Migrate a many-to-many field to an explicit intermediate ("through") model?

[Carsten Fuchs]

Hi Gergely,

Am 16.07.2015 um 20:44 schrieb Carsten Fuchs:

3) change the m2m field by adding the through option, and add this
change to the migrations file.


[...]
And won't this last step trigger the same error as before? ("... you cannot 
alter to or
from M2M fields, or add or remove through= on M2M fields") ?


Would you mind clarifying and elaborating on your reply please?

Most importantly, why would step 3) not result in the same error as I originally 
experienced?


Many thanks and best regards,
Carsten

--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/55ACE8C2.7000508%40cafu.de.
For more options, visit https://groups.google.com/d/optout.

Re: Migrate a many-to-many field to an explicit intermediate ("through") model?

[BHAGYARAJ POLA]

>
> hey hi,

I want to create web based visualization by using django framework in 
python.

 i need to have a front end interface which will communicate with the 
database.

The front end should look like similar to this.

http://specobs.ee.washington.edu/


For Map you can use Google Maps API, See how to use that and you know most 
of the information.

>From my view:

Google Maps should take the location data from your input
Input will be latitude, longitude and channel number

Output will be the heat map with the PSD value from the database.


Can u help me in this task?

thanx,

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/c3a8c6c8-f704-46b0-8d74-c555f1e0ca63%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Migrate a many-to-many field to an explicit intermediate ("through") model?

[Carsten Fuchs]

Hi Gergely,

many thanks for your quick reply!

Am 2015-07-16 um 18:15 schrieb Gergely Polonkai:

1) create the "through" model, add it to the migration file
2) migrate all m2m instances (e.g. iterate over all Bereich objects then
iterate through its users, creating a UserBereichAssignment object for
each (all this done in a migrations.RunPython call)


Ok. I've never used migrations.RunPython before, but I'm quite confident 
that I can manage it.


Would the migration for step 2) be a separate migration file from step 
1), or is the migration file of step 1) edited to cover step 2) as well?



3) change the m2m field by adding the through option, and add this
change to the migrations file.


Same question: Is this supposed to go into a separate migration file or 
into the common migrations file from above?


And won't this last step trigger the same error as before? ("... you 
cannot alter to or from M2M fields, or add or remove through= on M2M 
fields") ?


(Not so important, but out of curiosity: This won't get us rid of the 
old, implicit intermediate table, will it?)



However, you may want to go with django-guardian and thus, with
object-level permissions instead ;)


Thanks for mentioning this!

I've started looking into it, but the only user-code example that I've 
been able to find so far is in the middle of 
https://github.com/djangoadvent/djangoadvent-articles/blob/master/1.2/06_object-permissions.rst
Are there any other examples that demonstrate the purpose and usage of 
this package?



Many thanks and best regards,
Carsten

--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/55A7FB88.1040108%40cafu.de.
For more options, visit https://groups.google.com/d/optout.

Re: Migrate a many-to-many field to an explicit intermediate ("through") model?

[Gergely Polonkai]

You don't need the temporary field.

1) create the "through" model, add it to the migration file
2) migrate all m2m instances (e.g. iterate over all Bereich objects then
iterate through its users, creating a UserBereichAssignment object for each
(all this done in a migrations.RunPython call)
3) change the m2m field by adding the through option, and add this change
to the migrations file.

However, you may want to go with django-guardian and thus, with
object-level permissions instead ;)
On 16 Jul 2015 16:05, "Carsten Fuchs"  wrote:

> Dear Django fellows,
>
> using Django 1.8.3, in a fully migrated app I have a model with a
> many-to-many relationship like this:
>
> from django.db import models
> from django.contrib.auth.models import User
>
> class Bereich(models.Model):
> benutzer = models.ManyToManyField(User)
>
> Now I would like to store some extra information with the relationship
> (e.g. some Booleans that describe what the user is permitted to do with the
> related Bereich object).
> That is, the intended result is planned to look like this:
>
> class Bereich(models.Model):
> benutzer = models.ManyToManyField(User,
> through='UserBereichAssignment')
>
> class UserBereichAssignment(models.Model):
> bereich  = models.ForeignKey(Bereich)
> user = models.ForeignKey(User)
> can_edit = models.BooleanField(verbose_name="can edit?",
> default=True)
>
> However:
>
> $ ./manage.py makemigration # ok, creates a new migrations file
> $ ./manage.py migrate
>
> # ...
> ValueError: Cannot alter field Lori.Bereich.benutzer into
> Lori.Bereich.benutzer - they are not compatible types (you cannot alter to
> or from M2M fields, or add or remove through= on M2M fields)
>
>
> Well, I understand that and also seem to see some of the involved
> problems: For example, how would it make sure that the same database table
> is used for the previous implicit intermediate model and the new
> UserBereichAssignment?
>
> I'm also aware of https://code.djangoproject.com/ticket/23034, but I'm
> not sure what to make of it, or if it is even applicable here.
>
>
> Thus, what would be a good way to proceed?
>
> To the best of my understanding, I could do this manually, that is: create
> an entirely new many-to-many field, e.g.
>
> class Bereich(models.Model):
> benutzer  = models.ManyToManyField(User)# old,
> unchanged
> benutzer_ = models.ManyToManyField(User,
> through='UserBereichAssignment')
>
> then manually migrate the data from "benutzer" to "benutzer_" (possibly
> using a data migration?), and finally remove the old field "benutzer". The
> only downside seems that all dependent code had to be updated from using
> the old name "benutzer" to the new name "benutzer_". (I'm not sure if
> making a migration for renaming the new field name to the old field name is
> possible?)
>
> Alas... are there any viable alternatives to this?
> I'd be very grateful for any hint!   :-)
>
> Best regards,
> Carsten
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/55A7BA3E.5030105%40cafu.de.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CACczBUKez8kCfMFJgT94P_3bueNrdYck7fCqWQWF6hO_1AtEFw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Migrate a many-to-many field to an explicit intermediate ("through") model?

[Carsten Fuchs]

Dear Django fellows,

using Django 1.8.3, in a fully migrated app I have a model with a many-to-many 
relationship like this:


from django.db import models
from django.contrib.auth.models import User

class Bereich(models.Model):
benutzer = models.ManyToManyField(User)

Now I would like to store some extra information with the relationship (e.g. some 
Booleans that describe what the user is permitted to do with the related Bereich object).

That is, the intended result is planned to look like this:

class Bereich(models.Model):
benutzer = models.ManyToManyField(User, through='UserBereichAssignment')

class UserBereichAssignment(models.Model):
bereich  = models.ForeignKey(Bereich)
user = models.ForeignKey(User)
can_edit = models.BooleanField(verbose_name="can edit?", default=True)

However:

$ ./manage.py makemigration # ok, creates a new migrations file
$ ./manage.py migrate

# ...
ValueError: Cannot alter field Lori.Bereich.benutzer into Lori.Bereich.benutzer - 
they are not compatible types (you cannot alter to or from M2M fields, or add or remove 
through= on M2M fields)



Well, I understand that and also seem to see some of the involved problems: For example, 
how would it make sure that the same database table is used for the previous implicit 
intermediate model and the new UserBereichAssignment?


I'm also aware of https://code.djangoproject.com/ticket/23034, but I'm not sure what to 
make of it, or if it is even applicable here.



Thus, what would be a good way to proceed?

To the best of my understanding, I could do this manually, that is: create an entirely 
new many-to-many field, e.g.


class Bereich(models.Model):
benutzer  = models.ManyToManyField(User)# old, unchanged
benutzer_ = models.ManyToManyField(User, 
through='UserBereichAssignment')

then manually migrate the data from "benutzer" to "benutzer_" (possibly using a data 
migration?), and finally remove the old field "benutzer". The only downside seems that 
all dependent code had to be updated from using the old name "benutzer" to the new name 
"benutzer_". (I'm not sure if making a migration for renaming the new field name to the 
old field name is possible?)


Alas... are there any viable alternatives to this?
I'd be very grateful for any hint!   :-)

Best regards,
Carsten

--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/55A7BA3E.5030105%40cafu.de.
For more options, visit https://groups.google.com/d/optout.