Thanks Josh,

   1. A agree, they seem to be dupes and one should close, but not sure how 
   that gets done, by whom (am a learner not a pro).
   2. I'm happy to take a stab at an API proposal but was throwing some 
   naive questions out there int he hope of seeing some thought shared. In the 
   mean time it prompted me to work on some experiments with the Tutorial 
   models which I'm doing and it's proving interesting to say the least, but 
   as noted by many, it's kind of costly as I spent an evening just learning 
   one little thing about the difference between a CreateView and UpdateView 
   (kicking myself on that but could not find, for the life of me any docs on 
   it or alrady asked and answered enlightening questions on the stackoverflow 
   front or elsewhere. And to write an API worth critiquing I'd want a little 
   more chance of it being a tenable one (i.e. understand the innards a little 
   better, which is one reason I single stepped a lot of Django code last 
   night to learn what I have, and am still not 100% clear on it all).
   3. What I did learn is that it's a red herring to think of this as an 
   m2m issue! It seems universally filed as that but empirically I see that 
   one2m relationships suffer exactly the same problem - which is that the 
   submitted object as no PK until saved, and so the many related objects (be 
   it one to many or many to many, applies to all the "to many" contexts) are 
   not yet known to it during the model validation (clean) calls. Which leads 
   me suspect the true way forward might be indeed to define a form of 
   placeholder PK that is 100% backward compatibl, so looks like PK=None to 
   the rest of Django, excepting for a select view on those objects which 
   respects the placeholder PKs and so makes "to many" relations visible and 
   available for validation at the model not form level (which is what 
   everyone seems to agree is what we want - because it's a) easier, in the 
   ORM not in the form processing code, and b) more robust as it catches 
   submissions from any form or avenue that have to pass model validation. 
   
I totally get your work's response that m2m validation is "too hard for the 
effort" as I'm finding it that way (have an alpha site running without it 
for said reason), but seems to me numerous voices agree that for this very 
reason it's worth there being a Django internal solution as many of us 
benefit from on and can validate these relations at model level (where it's 
easier and better).

Cheers,

Bernd.

On Sunday, 17 February 2019 00:01:47 UTC+11, Josh Smeaton wrote:
>
> Considering https://code.djangoproject.com/ticket/12938 is still marked 
> as "open" and https://code.djangoproject.com/ticket/24731 is more or less 
> a duplicate (is there a reason it isn't closed as a dupe specifically?), I 
> don't think any more consensus is really required. The problem seems 
> difficult without diving in and getting your hands dirty, which is probably 
> why you're seeing such light discussion on this thread.
>
> What I'd suggest is: put together a proposal and let discussion occur 
> around that. Propose an API, and speak to how it won't impact existing 
> users, and I think you'll find lots of people will have something to say.
>
> FWIW, I'd be interested in seeing some kind of solution. Our ($work$) 
> current response to feature requests that involve validation of m2m fields 
> is "too hard for the effort".
>
> Cheers
>
> On Friday, 15 February 2019 22:44:22 UTC+11, Bernd Wechner wrote:
>>
>> I'm curious what consensus looks like. In what forum among which 
>> stakeholders. Clearly among developers who have some knowledge of Djangos 
>> innards, and so I suspect here. But I find conversation here on this thread 
>> so neatly short I can digest it in a short read, and 
>>
>> https://code.djangoproject.com/ticket/24731
>> https://code.djangoproject.com/ticket/12938
>>
>> both touch on the same issue with similarly delectably short comment 
>> streams that I could read and digest them in a jiffy. Read another way, not 
>> a lot of consensus generation discussion or activity visible.
>>
>> So I'll play the newb (because I am I guess, well been coding with Django 
>> for almost 4 years very very casually part time, and have had this 
>> particular issue on my "to solve" list for a long time already. But for me 
>> to make progress on it I need to invest some time into learning and some of 
>> that empirically through experimentation but I'll try and stimulate a 
>> little conversation here by doing some of it through the posing of a 
>> possibly naive question or two.
>>
>> Before that let me state that in my application (and I suspect this is 
>> not unusual, I am trying to validate m2m relations and one2m as well in 
>> what I'd describe as a rich model object, I use the term rich to describe a 
>> handful of models related to one another, all of which are updated with one 
>> form submission. In my simplest example I am modelling a game session which 
>> has a one2many relationship to ranks, which have a one2one relationship to 
>> teams which have an m2m relationship to players. A single game session is 
>> logged as the ranked list of teams and their players ... but I wouldn't get 
>> lost in the one example, the point is such relationships can't be wildly 
>> uncommon and we would all like to keep the validation logic in the 
>> models. In my example a session would like to check that there are the 
>> right number of ranks, not too few, not too many, as the game allows. 
>> Validating this in the form is frustrating as it's a) somewhat more 
>> complicated and b) not as secure (allows erroneous saves through means 
>> other than this form). And the logic belongs in the model. The session 
>> knows how many ranks it's expecting. I've seen other examples just as clear.
>>
>> And so onto the learning through possibly naive questioning.
>>
>> As I see it there are two versions of this rich object (objects of 
>> several models all interrelated:
>>
>> 1) The database version 
>> 2) the ORM version 
>>
>> The problem I see is that the ORM versions lack primary keys and the 
>> relationships they create until they are saved. So aside from inviting 
>> correction of any misunderstandings I may have tabled I will ask the 
>> salient question: Is this not a classic application for transactions. 
>> Namely we save all of the objects without committing then do the validation 
>> of the relationships in the ORM. This seems naive to me as it presupposes a 
>> few things that may or may not be true (and I fear are not):
>>
>> 1) The an uncommitted transaction delivers primary keys
>> 2) That an uncommitted transaction can easily be reflected back in the ORM
>>
>> If these are possible, is it not a good chance then to validate the 
>> relations int he respective model's Clean() methods and through an 
>> exception on failure, that results in a complete roll back and if it 
>> succeeds results in a commit. 
>>
>> I invite commentary, and discussion in the hope of achieving the elusive 
>> beast of consensus that Frederico alludes to.
>>
>> The main need I see is to create the relationships in the ORM, and it may 
>> be possible to do this pre-save too, with place-holder PKs, I do exactly 
>> this sort of thing at the form level, managing the relations between the 
>> various form elements, with placeholder IDs on elements. And so I imagine 
>> loosely if the strategy above is impossible that another might implement 
>> something along these lines, a way to create all the objects and have them 
>> as ORM objects but with placeholder PKs where needed managing the relations 
>> and (a central difficult) a modelled through object for M2Ms - which I 
>> imagine can be modelled as a django model with a ForeignKey to each of the 
>> related models one such object capturing the relationshiips.
>>
>> Where real PKs exist, they can be used, where they do not exist a 
>> placeholder could be used to model relationships in the ORM only. On save 
>> these placeholders are ignored and the database creates PKs as usual. 
>>
>> I can imagine a number of ways of modelling such placeholders. Either 
>> simply encoding otherwise illegal values (-ve numbers), or reserving on ID 
>> to flag a placheholder (PK-0 or PK=Maxvalue) and the placeholder PK (PH) is 
>> another attribute on model with references to PK returning that if the 
>> reserved value is in place etc. Alternately just use legal values in the 
>> high range (max value down) for the PKs and use a state flag to indicate 
>> they are faux, and to be ignored on save (it could be the simple existing 
>> flag that tells us it's unsaved can serve that role).
>>
>> Anyhow a pile of naive questions and speculations and I hope it 
>> stimulates a response or two or three and something that heads towards 
>> consensus.  Even if it's something completely different and all this is 
>> dismissed ;-).
>>
>> Kind regards,
>>
>> Bernd.
>>
>> On Saturday, 5 December 2015 04:59:05 UTC+11, Federico Capoano wrote:
>>>
>>> It could be a potential ticket to work on my next django dev sprint.
>>>
>>> But first it would be nice to have some basic consensus on how to 
>>> proceed.
>>>
>>> Was it ever discussed in any older thread or ticket?
>>>
>>>
>>>
>>> On Thursday, December 3, 2015 at 5:21:06 PM UTC+1, Tim Graham wrote:
>>>>
>>>> Here's an open ticket about adding model level validation for 
>>>> many-to-many: https://code.djangoproject.com/ticket/12938
>>>>
>>>> On Thursday, December 3, 2015 at 11:04:22 AM UTC-5, Federico Capoano 
>>>> wrote:
>>>>>
>>>>> Thanks Aymeric,
>>>>>
>>>>> I decided to post this problem on django-developers because I've read 
>>>>> this ticket:
>>>>> https://code.djangoproject.com/ticket/24731
>>>>> Sorry for omitting this information.
>>>>>
>>>>> Has there been a discussion about this topic already?
>>>>>
>>>>> Would it be hard to implement an easier solution into django?
>>>>>
>>>>> I spent a few hours working on this issue, I consider myself fluent 
>>>>> with django and it's quite shocking I had to put such an amount of effort 
>>>>> just to validate many2many relationships before they are saved to the 
>>>>> database.
>>>>>
>>>>> IMHO it would be better if we could do one of these two (or even both) 
>>>>> things:
>>>>>
>>>>> 1. make this process easier in future django versions
>>>>> 2. document the current best practice to solve this problem in current 
>>>>> django to save people's time
>>>>>
>>>>> What do people you think?
>>>>>
>>>>> Here's my solution working with django 1.9:
>>>>>
>>>>> https://github.com/openwisp/django-netjsonconfig/compare/4082988...master
>>>>>
>>>>> What do you think of it? Can it be improved in some way?
>>>>>
>>>>> Federico
>>>>>
>>>>>
>>>>> On Thursday, December 3, 2015 at 1:43:21 PM UTC+1, Aymeric Augustin 
>>>>> wrote:
>>>>>>
>>>>>> Hello Frederico,
>>>>>>
>>>>>> It appears that you're hitting the problem described in the "Avoid 
>>>>>> catching exceptions inside atomic!" warning on this page:
>>>>>>
>>>>>> https://docs.djangoproject.com/en/1.8/topics/db/transactions/#handling-exceptions-within-postgresql-transactions
>>>>>>
>>>>>> To obtain that sort of result, I suppose you must be catching an 
>>>>>> IntegrityError, re-raising a ValidationError, which Django in turn 
>>>>>> catches, 
>>>>>> and then you hit that traceback.
>>>>>>
>>>>>> Adding an atomic block inside your try/catch block that catches the 
>>>>>> IntegrityError will resolve that particular problem — putting that part 
>>>>>> of 
>>>>>> the discussion into django-users territory.
>>>>>>
>>>>>> If this isn't what happens, please post your code and ask for help on 
>>>>>> django-users.
>>>>>>
>>>>>> -- 
>>>>>> Aymeric
>>>>>>
>>>>>> 2015-12-03 13:28 GMT+01:00 Federico Capoano <federico...@gmail.com>:
>>>>>>
>>>>>>> Hi everybody,
>>>>>>>
>>>>>>> I am sure it has happened to many of you.
>>>>>>>
>>>>>>> Validating m2m BEFORE saving the relationships is very hard and time 
>>>>>>> consuming.
>>>>>>>
>>>>>>> Now this solution:
>>>>>>> http://schinckel.net/2012/02/06/pre-validating-many-to-many-fields./
>>>>>>>
>>>>>>> Proposes to solve it with a ModelForm in the admin.
>>>>>>> Cool, that works.
>>>>>>>
>>>>>>> But, if I want to enforce validation on the model, to avoid 
>>>>>>> corrupted data, I notice the signal is executed in a transaction block, 
>>>>>>> in 
>>>>>>> which if I raise a ValidationError I get the following:
>>>>>>>
>>>>>>> Traceback (most recent call last):
>>>>>>>   File 
>>>>>>> "/var/www/django-netjsonconfig/django_netjsonconfig/tests/test_device.py",
>>>>>>>  
>>>>>>> line 106, in test_m2m_validation
>>>>>>>     d.templates.add(t)
>>>>>>>   File 
>>>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/fields/related_descriptors.py",
>>>>>>>  
>>>>>>> line 843, in add
>>>>>>>     self._add_items(self.source_field_name, self.target_field_name, 
>>>>>>> *objs)
>>>>>>>   File 
>>>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/sortedm2m/fields.py",
>>>>>>>  
>>>>>>> line 138, in _add_items
>>>>>>>     for val in vals:
>>>>>>>   File 
>>>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/query.py",
>>>>>>>  
>>>>>>> line 258, in __iter__
>>>>>>>     self._fetch_all()
>>>>>>>   File 
>>>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/query.py",
>>>>>>>  
>>>>>>> line 1074, in _fetch_all
>>>>>>>     self._result_cache = list(self.iterator())
>>>>>>>   File 
>>>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/query.py",
>>>>>>>  
>>>>>>> line 158, in __iter__
>>>>>>>     for row in compiler.results_iter():
>>>>>>>   File 
>>>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/sql/compiler.py",
>>>>>>>  
>>>>>>> line 806, in results_iter
>>>>>>>     results = self.execute_sql(MULTI)
>>>>>>>   File 
>>>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/models/sql/compiler.py",
>>>>>>>  
>>>>>>> line 852, in execute_sql
>>>>>>>     cursor.execute(sql, params)
>>>>>>>   File 
>>>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/backends/utils.py",
>>>>>>>  
>>>>>>> line 59, in execute
>>>>>>>     self.db.validate_no_broken_transaction()
>>>>>>>   File 
>>>>>>> "/home/nemesis/.virtualenvs/djnetconfig3/lib/python3.4/site-packages/django/db/backends/base/base.py",
>>>>>>>  
>>>>>>> line 429, in validate_no_broken_transaction
>>>>>>>     "An error occurred in the current transaction. You can't "
>>>>>>> django.db.transaction.TransactionManagementError: An error occurred 
>>>>>>> in the current transaction. You can't execute queries until the end of 
>>>>>>> the 
>>>>>>> 'atomic' block.
>>>>>>>
>>>>>>> This is surely an area that needs improvement in django.
>>>>>>>
>>>>>>> Why is it so hard?
>>>>>>>
>>>>>>> Best regards
>>>>>>> Federico
>>>>>>>
>>>>>>> -- 
>>>>>>> You received this message because you are subscribed to the Google 
>>>>>>> Groups "Django developers (Contributions to Django itself)" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it, 
>>>>>>> send an email to django-develop...@googlegroups.com.
>>>>>>> To post to this group, send email to django-d...@googlegroups.com.
>>>>>>> Visit this group at http://groups.google.com/group/django-developers
>>>>>>> .
>>>>>>> To view this discussion on the web visit 
>>>>>>> https://groups.google.com/d/msgid/django-developers/2e6e82d0-0645-4fd7-8905-d327c99b6352%40googlegroups.com
>>>>>>>  
>>>>>>> <https://groups.google.com/d/msgid/django-developers/2e6e82d0-0645-4fd7-8905-d327c99b6352%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> Aymeric.
>>>>>>
>>>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/badbf72c-0c50-451c-aa48-a3a9bb695903%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to