Considering https://code.djangoproject.com/ticket/12938 is still marked as 
"open" and https://code.djangoproject.com/ticket/24731 is more or less a 
duplicate (is there a reason it isn't closed as a dupe specifically?), I 
don't think any more consensus is really required. The problem seems 
difficult without diving in and getting your hands dirty, which is probably 
why you're seeing such light discussion on this thread.

What I'd suggest is: put together a proposal and let discussion occur 
around that. Propose an API, and speak to how it won't impact existing 
users, and I think you'll find lots of people will have something to say.

FWIW, I'd be interested in seeing some kind of solution. Our ($work$) 
current response to feature requests that involve validation of m2m fields 
is "too hard for the effort".

Cheers

On Friday, 15 February 2019 22:44:22 UTC+11, Bernd Wechner wrote:
>
> I'm curious what consensus looks like. In what forum among which 
> stakeholders. Clearly among developers who have some knowledge of Djangos 
> innards, and so I suspect here. But I find conversation here on this thread 
> so neatly short I can digest it in a short read, and 
>
> https://code.djangoproject.com/ticket/24731
> https://code.djangoproject.com/ticket/12938
>
> both touch on the same issue with similarly delectably short comment 
> streams that I could read and digest them in a jiffy. Read another way, not 
> a lot of consensus generation discussion or activity visible.
>
> So I'll play the newb (because I am I guess, well been coding with Django 
> for almost 4 years very very casually part time, and have had this 
> particular issue on my "to solve" list for a long time already. But for me 
> to make progress on it I need to invest some time into learning and some of 
> that empirically through experimentation but I'll try and stimulate a 
> little conversation here by doing some of it through the posing of a 
> possibly naive question or two.
>
> Before that let me state that in my application (and I suspect this is not 
> unusual, I am trying to validate m2m relations and one2m as well in what 
> I'd describe as a rich model object, I use the term rich to describe a 
> handful of models related to one another, all of which are updated with one 
> form submission. In my simplest example I am modelling a game session which 
> has a one2many relationship to ranks, which have a one2one relationship to 
> teams which have an m2m relationship to players. A single game session is 
> logged as the ranked list of teams and their players ... but I wouldn't get 
> lost in the one example, the point is such relationships can't be wildly 
> uncommon and we would all like to keep the validation logic in the 
> models. In my example a session would like to check that there are the 
> right number of ranks, not too few, not too many, as the game allows. 
> Validating this in the form is frustrating as it's a) somewhat more 
> complicated and b) not as secure (allows erroneous saves through means 
> other than this form). And the logic belongs in the model. The session 
> knows how many ranks it's expecting. I've seen other examples just as clear.
>
> And so onto the learning through possibly naive questioning.
>
> As I see it there are two versions of this rich object (objects of several 
> models all interrelated:
>
> 1) The database version 
> 2) the ORM version 
>
> The problem I see is that the ORM versions lack primary keys and the 
> relationships they create until they are saved. So aside from inviting 
> correction of any misunderstandings I may have tabled I will ask the 
> salient question: Is this not a classic application for transactions. 
> Namely we save all of the objects without committing then do the validation 
> of the relationships in the ORM. This seems naive to me as it presupposes a 
> few things that may or may not be true (and I fear are not):
>
> 1) The an uncommitted transaction delivers primary keys
> 2) That an uncommitted transaction can easily be reflected back in the ORM
>
> If these are possible, is it not a good chance then to validate the 
> relations int he respective model's Clean() methods and through an 
> exception on failure, that results in a complete roll back and if it 
> succeeds results in a commit. 
>
> I invite commentary, and discussion in the hope of achieving the elusive 
> beast of consensus that Frederico alludes to.
>
> The main need I see is to create the relationships in the ORM, and it may 
> be possible to do this pre-save too, with place-holder PKs, I do exactly 
> this sort of thing at the form level, managing the relations between the 
> various form elements, with placeholder IDs on elements. And so I imagine 
> loosely if the strategy above is impossible that another might implement 
> something along these lines, a way to create all the objects and have them 
> as ORM objects but with placeholder PKs where needed managing the relations 
> and (a central difficult) a modelled through object for M2Ms - which I 
> imagine can be modelled as a django model with a ForeignKey to each of the 
> related models one such object capturing the relationshiips.
>
> Where real PKs exist, they can be used, where they do not exist a 
> placeholder could be used to model relationships in the ORM only. On save 
> these placeholders are ignored and the database creates PKs as usual. 
>
> I can imagine a number of ways of modelling such placeholders. Either 
> simply encoding otherwise illegal values (-ve numbers), or reserving on ID 
> to flag a placheholder (PK-0 or PK=Maxvalue) and the placeholder PK (PH) is 
> another attribute on model with references to PK returning that if the 
> reserved value is in place etc. Alternately just use legal values in the 
> high range (max value down) for the PKs and use a state flag to indicate 
> they are faux, and to be ignored on save (it could be the simple existing 
> flag that tells us it's unsaved can serve that role).
>
> Anyhow a pile of naive questions and speculations and I hope it stimulates 
> a response or two or three and something that heads towards consensus.  
> Even if it's something completely different and all this is dismissed ;-).
>
> Kind regards,
>
> Bernd.
>
> On Saturday, 5 December 2015 04:59:05 UTC+11, Federico Capoano wrote:
>>
>> It could be a potential ticket to work on my next django dev sprint.
>>
>> But first it would be nice to have some basic consensus on how to proceed.
>>
>> Was it ever discussed in any older thread or ticket?
>>
>>
>>
>> On Thursday, December 3, 2015 at 5:21:06 PM UTC+1, Tim Graham wrote:
>>>
>>> Here's an open ticket about adding model level validation for 
>>> many-to-many: https://code.djangoproject.com/ticket/12938
>>>
>>> On Thursday, December 3, 2015 at 11:04:22 AM UTC-5, Federico Capoano 
>>> wrote:
>>>>
>>>> Thanks Aymeric,
>>>>
>>>> I decided to post this problem on django-developers because I've read 
>>>> this ticket:
>>>> https://code.djangoproject.com/ticket/24731
>>>> Sorry for omitting this information.
>>>>
>>>> Has there been a discussion about this topic already?
>>>>
>>>> Would it be hard to implement an easier solution into django?
>>>>
>>>> I spent a few hours working on this issue, I consider myself fluent 
>>>> with django and it's quite shocking I had to put such an amount of effort 
>>>> just to validate many2many relationships before they are saved to the 
>>>> database.
>>>>
>>>> IMHO it would be better if we could do one of these two (or even both) 
>>>> things:
>>>>
>>>> 1. make this process easier in future django versions
>>>> 2. document the current best practice to solve this problem in current 
>>>> django to save people's time
>>>>
>>>> What do people you think?
>>>>
>>>> Here's my solution working with django 1.9:
>>>>
>>>> https://github.com/openwisp/django-netjsonconfig/compare/4082988...master
>>>>
>>>> What do you think of it? Can it be improved in some way?
>>>>
>>>> Federico
>>>>
>>>>
>>>> On Thursday, December 3, 2015 at 1:43:21 PM UTC+1, Aymeric Augustin 
>>>> wrote:
>>>>>
>>>>> Hello Frederico,
>>>>>
>>>>> It appears that you're hitting the problem described in the "Avoid 
>>>>> catching exceptions inside atomic!" warning on this page:
>>>>>
>>>>> https://docs.djangoproject.com/en/1.8/topics/db/transactions/#handling-exceptions-within-postgresql-transactions
>>>>>
>>>>> To obtain that sort of result, I suppose you must be catching an 
>>>>> IntegrityError, re-raising a ValidationError, which Django in turn 
>>>>> catches, 
>>>>> and then you hit that traceback.
>>>>>
>>>>> Adding an atomic block inside your try/catch block that catches the 
>>>>> IntegrityError will resolve that particular problem — putting that part 
>>>>> of 
>>>>> the discussion into django-users territory.
>>>>>
>>>>> If this isn't what happens, please post your code and ask for help on 
>>>>> django-users.
>>>>>
>>>>> -- 
>>>>> Aymeric
>>>>>
>>>>> 2015-12-03 13:28 GMT+01:00 Federico Capoano <federico...@gmail.com>:
>>>>>
>>>>>> Hi everybody,
>>>>>>
>>>>>> I am sure it has happened to many of you.
>>>>>>
>>>>>> Validating m2m BEFORE saving the relationships is very hard and time 
>>>>>> consuming.
>>>>>>
>>>>>> Now this solution:
>>>>>> http://schinckel.net/2012/02/06/pre-validating-many-to-many-fields./
>>>>>>
>>>>>> Proposes to solve it with a ModelForm in the admin.
>>>>>> Cool, that works.
>>>>>>
>>>>>> But, if I want to enforce validation on the model, to avoid corrupted 
>>>>>> data, I notice the signal is executed in a transaction block, in which 
>>>>>> if I 
>>>>>> raise a ValidationError I get the following:
>>>>>>
>>>>>> Traceback (most recent call last):
>>>>>>   File 
>>>>>> "/var/www/django-netjsonconfig/django_netjsonconfig/tests/test_device.py",
>>>>>>  
>>>>>> line 106, in test_m2m_validation
>>>>>>     d.templates.add(t)
>>>>>>   File 
>>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/fields/related_descriptors.py",
>>>>>>  
>>>>>> line 843, in add
>>>>>>     self._add_items(self.source_field_name, self.target_field_name, 
>>>>>> *objs)
>>>>>>   File 
>>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/sortedm2m/fields.py",
>>>>>>  
>>>>>> line 138, in _add_items
>>>>>>     for val in vals:
>>>>>>   File 
>>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/query.py",
>>>>>>  
>>>>>> line 258, in __iter__
>>>>>>     self._fetch_all()
>>>>>>   File 
>>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/query.py",
>>>>>>  
>>>>>> line 1074, in _fetch_all
>>>>>>     self._result_cache = list(self.iterator())
>>>>>>   File 
>>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/query.py",
>>>>>>  
>>>>>> line 158, in __iter__
>>>>>>     for row in compiler.results_iter():
>>>>>>   File 
>>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/sql/compiler.py",
>>>>>>  
>>>>>> line 806, in results_iter
>>>>>>     results = self.execute_sql(MULTI)
>>>>>>   File 
>>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/sql/compiler.py",
>>>>>>  
>>>>>> line 852, in execute_sql
>>>>>>     cursor.execute(sql, params)
>>>>>>   File 
>>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/backends/utils.py",
>>>>>>  
>>>>>> line 59, in execute
>>>>>>     self.db.validate_no_broken_transaction()
>>>>>>   File 
>>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/backends/base/base.py",
>>>>>>  
>>>>>> line 429, in validate_no_broken_transaction
>>>>>>     "An error occurred in the current transaction. You can't "
>>>>>> django.db.transaction.TransactionManagementError: An error occurred 
>>>>>> in the current transaction. You can't execute queries until the end of 
>>>>>> the 
>>>>>> 'atomic' block.
>>>>>>
>>>>>> This is surely an area that needs improvement in django.
>>>>>>
>>>>>> Why is it so hard?
>>>>>>
>>>>>> Best regards
>>>>>> Federico
>>>>>>
>>>>>> -- 
>>>>>> You received this message because you are subscribed to the Google 
>>>>>> Groups "Django developers (Contributions to Django itself)" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it, 
>>>>>> send an email to django-develop...@googlegroups.com.
>>>>>> To post to this group, send email to django-d...@googlegroups.com.
>>>>>> Visit this group at http://groups.google.com/group/django-developers.
>>>>>> To view this discussion on the web visit 
>>>>>> https://groups.google.com/d/msgid/django-developers/2e6e82d0-0645-4fd7-8905-d327c99b6352%40googlegroups.com
>>>>>>  
>>>>>> <https://groups.google.com/d/msgid/django-developers/2e6e82d0-0645-4fd7-8905-d327c99b6352%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -- 
>>>>> Aymeric.
>>>>>
>>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b108d0f6-a2d7-42df-bfd3-8435a34e3179%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to