I'm curious what consensus looks like. In what forum among which 
stakeholders. Clearly among developers who have some knowledge of Djangos 
innards, and so I suspect here. But I find conversation here on this thread 
so neatly short I can digest it in a short read, and 

https://code.djangoproject.com/ticket/24731
https://code.djangoproject.com/ticket/12938

both touch on the same issue with similarly delectably short comment 
streams that I could read and digest them in a jiffy. Read another way, not 
a lot of consensus generation discussion or activity visible.

So I'll play the newb (because I am I guess, well been coding with Django 
for almost 4 years very very casually part time, and have had this 
particular issue on my "to solve" list for a long time already. But for me 
to make progress on it I need to invest some time into learning and some of 
that empirically through experimentation but I'll try and stimulate a 
little conversation here by doing some of it through the posing of a 
possibly naive question or two.

Before that let me state that in my application (and I suspect this is not 
unusual, I am trying to validate m2m relations and one2m as well in what 
I'd describe as a rich model object, I use the term rich to describe a 
handful of models related to one another, all of which are updated with one 
form submission. In my simplest example I am modelling a game session which 
has a one2many relationship to ranks, which have a one2one relationship to 
teams which have an m2m relationship to players. A single game session is 
logged as the ranked list of teams and their players ... but I wouldn't get 
lost in the one example, the point is such relationships can't be wildly 
uncommon and we would all like to keep the validation logic in the 
models. In my example a session would like to check that there are the 
right number of ranks, not too few, not too many, as the game allows. 
Validating this in the form is frustrating as it's a) somewhat more 
complicated and b) not as secure (allows erroneous saves through means 
other than this form). And the logic belongs in the model. The session 
knows how many ranks it's expecting. I've seen other examples just as clear.

And so onto the learning through possibly naive questioning.

As I see it there are two versions of this rich object (objects of several 
models all interrelated:

1) The database version 
2) the ORM version 

The problem I see is that the ORM versions lack primary keys and the 
relationships they create until they are saved. So aside from inviting 
correction of any misunderstandings I may have tabled I will ask the 
salient question: Is this not a classic application for transactions. 
Namely we save all of the objects without committing then do the validation 
of the relationships in the ORM. This seems naive to me as it presupposes a 
few things that may or may not be true (and I fear are not):

1) The an uncommitted transaction delivers primary keys
2) That an uncommitted transaction can easily be reflected back in the ORM

If these are possible, is it not a good chance then to validate the 
relations int he respective model's Clean() methods and through an 
exception on failure, that results in a complete roll back and if it 
succeeds results in a commit. 

I invite commentary, and discussion in the hope of achieving the elusive 
beast of consensus that Frederico alludes to.

The main need I see is to create the relationships in the ORM, and it may 
be possible to do this pre-save too, with place-holder PKs, I do exactly 
this sort of thing at the form level, managing the relations between the 
various form elements, with placeholder IDs on elements. And so I imagine 
loosely if the strategy above is impossible that another might implement 
something along these lines, a way to create all the objects and have them 
as ORM objects but with placeholder PKs where needed managing the relations 
and (a central difficult) a modelled through object for M2Ms - which I 
imagine can be modelled as a django model with a ForeignKey to each of the 
related models one such object capturing the relationshiips.

Where real PKs exist, they can be used, where they do not exist a 
placeholder could be used to model relationships in the ORM only. On save 
these placeholders are ignored and the database creates PKs as usual. 

I can imagine a number of ways of modelling such placeholders. Either 
simply encoding otherwise illegal values (-ve numbers), or reserving on ID 
to flag a placheholder (PK-0 or PK=Maxvalue) and the placeholder PK (PH) is 
another attribute on model with references to PK returning that if the 
reserved value is in place etc. Alternately just use legal values in the 
high range (max value down) for the PKs and use a state flag to indicate 
they are faux, and to be ignored on save (it could be the simple existing 
flag that tells us it's unsaved can serve that role).

Anyhow a pile of naive questions and speculations and I hope it stimulates 
a response or two or three and something that heads towards consensus.  
Even if it's something completely different and all this is dismissed ;-).

Kind regards,

Bernd.

On Saturday, 5 December 2015 04:59:05 UTC+11, Federico Capoano wrote:
>
> It could be a potential ticket to work on my next django dev sprint.
>
> But first it would be nice to have some basic consensus on how to proceed.
>
> Was it ever discussed in any older thread or ticket?
>
>
>
> On Thursday, December 3, 2015 at 5:21:06 PM UTC+1, Tim Graham wrote:
>>
>> Here's an open ticket about adding model level validation for 
>> many-to-many: https://code.djangoproject.com/ticket/12938
>>
>> On Thursday, December 3, 2015 at 11:04:22 AM UTC-5, Federico Capoano 
>> wrote:
>>>
>>> Thanks Aymeric,
>>>
>>> I decided to post this problem on django-developers because I've read 
>>> this ticket:
>>> https://code.djangoproject.com/ticket/24731
>>> Sorry for omitting this information.
>>>
>>> Has there been a discussion about this topic already?
>>>
>>> Would it be hard to implement an easier solution into django?
>>>
>>> I spent a few hours working on this issue, I consider myself fluent with 
>>> django and it's quite shocking I had to put such an amount of effort just 
>>> to validate many2many relationships before they are saved to the database.
>>>
>>> IMHO it would be better if we could do one of these two (or even both) 
>>> things:
>>>
>>> 1. make this process easier in future django versions
>>> 2. document the current best practice to solve this problem in current 
>>> django to save people's time
>>>
>>> What do people you think?
>>>
>>> Here's my solution working with django 1.9:
>>> https://github.com/openwisp/django-netjsonconfig/compare/4082988...master
>>>
>>> What do you think of it? Can it be improved in some way?
>>>
>>> Federico
>>>
>>>
>>> On Thursday, December 3, 2015 at 1:43:21 PM UTC+1, Aymeric Augustin 
>>> wrote:
>>>>
>>>> Hello Frederico,
>>>>
>>>> It appears that you're hitting the problem described in the "Avoid 
>>>> catching exceptions inside atomic!" warning on this page:
>>>>
>>>> https://docs.djangoproject.com/en/1.8/topics/db/transactions/#handling-exceptions-within-postgresql-transactions
>>>>
>>>> To obtain that sort of result, I suppose you must be catching an 
>>>> IntegrityError, re-raising a ValidationError, which Django in turn 
>>>> catches, 
>>>> and then you hit that traceback.
>>>>
>>>> Adding an atomic block inside your try/catch block that catches the 
>>>> IntegrityError will resolve that particular problem — putting that part of 
>>>> the discussion into django-users territory.
>>>>
>>>> If this isn't what happens, please post your code and ask for help on 
>>>> django-users.
>>>>
>>>> -- 
>>>> Aymeric
>>>>
>>>> 2015-12-03 13:28 GMT+01:00 Federico Capoano <federico...@gmail.com>:
>>>>
>>>>> Hi everybody,
>>>>>
>>>>> I am sure it has happened to many of you.
>>>>>
>>>>> Validating m2m BEFORE saving the relationships is very hard and time 
>>>>> consuming.
>>>>>
>>>>> Now this solution:
>>>>> http://schinckel.net/2012/02/06/pre-validating-many-to-many-fields./
>>>>>
>>>>> Proposes to solve it with a ModelForm in the admin.
>>>>> Cool, that works.
>>>>>
>>>>> But, if I want to enforce validation on the model, to avoid corrupted 
>>>>> data, I notice the signal is executed in a transaction block, in which if 
>>>>> I 
>>>>> raise a ValidationError I get the following:
>>>>>
>>>>> Traceback (most recent call last):
>>>>>   File 
>>>>> "/var/www/django-netjsonconfig/django_netjsonconfig/tests/test_device.py",
>>>>>  
>>>>> line 106, in test_m2m_validation
>>>>>     d.templates.add(t)
>>>>>   File 
>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/fields/related_descriptors.py",
>>>>>  
>>>>> line 843, in add
>>>>>     self._add_items(self.source_field_name, self.target_field_name, 
>>>>> *objs)
>>>>>   File 
>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/sortedm2m/fields.py",
>>>>>  
>>>>> line 138, in _add_items
>>>>>     for val in vals:
>>>>>   File 
>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/query.py",
>>>>>  
>>>>> line 258, in __iter__
>>>>>     self._fetch_all()
>>>>>   File 
>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/query.py",
>>>>>  
>>>>> line 1074, in _fetch_all
>>>>>     self._result_cache = list(self.iterator())
>>>>>   File 
>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/query.py",
>>>>>  
>>>>> line 158, in __iter__
>>>>>     for row in compiler.results_iter():
>>>>>   File 
>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/sql/compiler.py",
>>>>>  
>>>>> line 806, in results_iter
>>>>>     results = self.execute_sql(MULTI)
>>>>>   File 
>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/sql/compiler.py",
>>>>>  
>>>>> line 852, in execute_sql
>>>>>     cursor.execute(sql, params)
>>>>>   File 
>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/backends/utils.py",
>>>>>  
>>>>> line 59, in execute
>>>>>     self.db.validate_no_broken_transaction()
>>>>>   File 
>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/backends/base/base.py",
>>>>>  
>>>>> line 429, in validate_no_broken_transaction
>>>>>     "An error occurred in the current transaction. You can't "
>>>>> django.db.transaction.TransactionManagementError: An error occurred in 
>>>>> the current transaction. You can't execute queries until the end of the 
>>>>> 'atomic' block.
>>>>>
>>>>> This is surely an area that needs improvement in django.
>>>>>
>>>>> Why is it so hard?
>>>>>
>>>>> Best regards
>>>>> Federico
>>>>>
>>>>> -- 
>>>>> You received this message because you are subscribed to the Google 
>>>>> Groups "Django developers (Contributions to Django itself)" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send 
>>>>> an email to django-develop...@googlegroups.com.
>>>>> To post to this group, send email to django-d...@googlegroups.com.
>>>>> Visit this group at http://groups.google.com/group/django-developers.
>>>>> To view this discussion on the web visit 
>>>>> https://groups.google.com/d/msgid/django-developers/2e6e82d0-0645-4fd7-8905-d327c99b6352%40googlegroups.com
>>>>>  
>>>>> <https://groups.google.com/d/msgid/django-developers/2e6e82d0-0645-4fd7-8905-d327c99b6352%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>>
>>>>
>>>> -- 
>>>> Aymeric.
>>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/89c2bda2-d7a8-45bf-b215-69f01b442b45%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to