Re: Model-level validation

[Adrian Torres]

Hi,

Regardless of what you consider ModelForms to be, the fact that validation 
doesn't happen at the model level is very jarring if you've ever used any 
other MVC framework, it was and still is one of the major pet peeves of 
Django for me, to the point where we do something similar to what Uri does, 
and I'm sure many other people do.

bulk_update is not an excuse, as Aaron mentioned many other ORMs / 
frameworks come with features that forego certain safeties in favor of 
performance / convenience, heck you can write SQL directly if you so desire 
but you need to be ready to face the consequences.

I like the `UnvalidatedModel` vs `Model` idea proposed by Aaron.

Cheers,
Adrian

On Friday, September 30, 2022 at 3:39:20 AM UTC+2 aa...@aaronsmith.co wrote:

> I would also like everyone to know, my objective in starting this thread 
> is to get the go-ahead to open a PR for this. I would like to contribute 
> back.
>
> On Thursday, September 29, 2022 at 6:30:32 PM UTC-7 Aaron Smith wrote:
>
>> How about a new class, `ValidatedModel`, that subclasses `Model` and does 
>> nothing more than call `full_clean()` on `save()`?
>>
>> This would be completely backwards compatible, would clearly communicate 
>> what it does, and when documented right next to `Model` make it fairly 
>> obvious that Model is something other than validated, hopefully preventing 
>> many footguns.
>>
>> Or, and I think this would be better, if the current Model were renamed 
>> `UnvalidatedModel`, the new validated implementation above were `Model`. 
>> This upgrade path is a simple string replacement for those legacy codebases 
>> (Model->UnvalidatedModel), making it abundantly clear they are not 
>> validated, and new apps following the most naive path (Model) are as safe 
>> as possible. The new, validated, `Model.save()` could accept the kwarg 
>> `validate=False` as an opt-out, which as much as I hate to admit it is an 
>> important option for some codebases.
>>
>> On Thursday, September 29, 2022 at 5:19:07 PM UTC-7 cur...@tinbrain.net 
>> wrote:
>>
>>> On Thu, 29 Sep 2022, at 14:29, Aaron Smith wrote:
>>>
>>> Why doesn't Django validate Models on save()?
>>>
>>>
>>> The short answer is: backwards compatibility.
>>>
>>> Model level validation hasn't always been a thing, with Django initially 
>>> depending primarily on Form validation.
>>>
>>> Since it hadn't _always_ been there, it wasn't possible to introduce it, 
>>> enforce it, and not break most apps out there.
>>>
>>> There was so much code written that generally assumed it could call 
>>> `save()` and not have to catch validation errors.
>>>
>>> For what it's worth, I'm all in favor of making it run on `save()` ... 
>>> updating the documentation and preparing the community is going to be a 
>>> mammoth task, however. A safe course through this will take some very 
>>> careful planning.
>>>
>>> --
>>> C
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f51db24d-7c69-49f2-ad83-c8fd2418bfbdn%40googlegroups.com.

Re: Model-level validation

[Aaron Smith]

I would also like everyone to know, my objective in starting this thread is 
to get the go-ahead to open a PR for this. I would like to contribute back.

On Thursday, September 29, 2022 at 6:30:32 PM UTC-7 Aaron Smith wrote:

> How about a new class, `ValidatedModel`, that subclasses `Model` and does 
> nothing more than call `full_clean()` on `save()`?
>
> This would be completely backwards compatible, would clearly communicate 
> what it does, and when documented right next to `Model` make it fairly 
> obvious that Model is something other than validated, hopefully preventing 
> many footguns.
>
> Or, and I think this would be better, if the current Model were renamed 
> `UnvalidatedModel`, the new validated implementation above were `Model`. 
> This upgrade path is a simple string replacement for those legacy codebases 
> (Model->UnvalidatedModel), making it abundantly clear they are not 
> validated, and new apps following the most naive path (Model) are as safe 
> as possible. The new, validated, `Model.save()` could accept the kwarg 
> `validate=False` as an opt-out, which as much as I hate to admit it is an 
> important option for some codebases.
>
> On Thursday, September 29, 2022 at 5:19:07 PM UTC-7 cur...@tinbrain.net 
> wrote:
>
>> On Thu, 29 Sep 2022, at 14:29, Aaron Smith wrote:
>>
>> Why doesn't Django validate Models on save()?
>>
>>
>> The short answer is: backwards compatibility.
>>
>> Model level validation hasn't always been a thing, with Django initially 
>> depending primarily on Form validation.
>>
>> Since it hadn't _always_ been there, it wasn't possible to introduce it, 
>> enforce it, and not break most apps out there.
>>
>> There was so much code written that generally assumed it could call 
>> `save()` and not have to catch validation errors.
>>
>> For what it's worth, I'm all in favor of making it run on `save()` ... 
>> updating the documentation and preparing the community is going to be a 
>> mammoth task, however. A safe course through this will take some very 
>> careful planning.
>>
>> --
>> C
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ce26943e-6755-4f87-b6ee-0ea6325367dcn%40googlegroups.com.

Re: Model-level validation

[Aaron Smith]

Carl,

All ORMs I have worked with allow you to bypass validations when necessary. 
Sometimes you have to. But the path of greatest naivety should be as safe 
as possible. I cannot even imagine how many lost hours and economic damages 
have occurred because the easy path is the dangerous path. I have been 
there for some of it - data consistency problems are horrible.

On Thursday, September 29, 2022 at 5:36:29 PM UTC-7 carl.j...@gmail.com 
wrote:

> On Thu, Sep 29, 2022 at 6:19 PM Curtis Maloney  
> wrote:
>
>> On Thu, 29 Sep 2022, at 14:29, Aaron Smith wrote:
>>
>> Why doesn't Django validate Models on save()?
>>
>>
>> The short answer is: backwards compatibility.
>>
>> Model level validation hasn't always been a thing, with Django initially 
>> depending primarily on Form validation.
>>
>> Since it hadn't _always_ been there, it wasn't possible to introduce it, 
>> enforce it, and not break most apps out there.
>>
>> There was so much code written that generally assumed it could call 
>> `save()` and not have to catch validation errors.
>>
>> For what it's worth, I'm all in favor of making it run on `save()` ... 
>> updating the documentation and preparing the community is going to be a 
>> mammoth task, however. A safe course through this will take some very 
>> careful planning.
>>
>
> Another factor that should be considered is that the Django ORM gives you 
> plenty of ways to update your database (eg bulk updates on a queryset) that 
> clearly cannot materialize and validate every object in the queryset. So is 
> it better to consistently say “the ORM doesn’t validate, forms do,” or 
> better to say “the ORM will validate on Model.save() but not in various 
> other cases, so you still can’t really rely on the ORM to enforce model 
> validation invariants consistently.”
>
> Carl
>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b3fe241c-914c-47d4-8bd1-ef5b646b8c2bn%40googlegroups.com.

Re: Model-level validation

[Aaron Smith]

How about a new class, `ValidatedModel`, that subclasses `Model` and does 
nothing more than call `full_clean()` on `save()`?

This would be completely backwards compatible, would clearly communicate 
what it does, and when documented right next to `Model` make it fairly 
obvious that Model is something other than validated, hopefully preventing 
many footguns.

Or, and I think this would be better, if the current Model were renamed 
`UnvalidatedModel`, the new validated implementation above were `Model`. 
This upgrade path is a simple string replacement for those legacy codebases 
(Model->UnvalidatedModel), making it abundantly clear they are not 
validated, and new apps following the most naive path (Model) are as safe 
as possible. The new, validated, `Model.save()` could accept the kwarg 
`validate=False` as an opt-out, which as much as I hate to admit it is an 
important option for some codebases.

On Thursday, September 29, 2022 at 5:19:07 PM UTC-7 cur...@tinbrain.net 
wrote:

> On Thu, 29 Sep 2022, at 14:29, Aaron Smith wrote:
>
> Why doesn't Django validate Models on save()?
>
>
> The short answer is: backwards compatibility.
>
> Model level validation hasn't always been a thing, with Django initially 
> depending primarily on Form validation.
>
> Since it hadn't _always_ been there, it wasn't possible to introduce it, 
> enforce it, and not break most apps out there.
>
> There was so much code written that generally assumed it could call 
> `save()` and not have to catch validation errors.
>
> For what it's worth, I'm all in favor of making it run on `save()` ... 
> updating the documentation and preparing the community is going to be a 
> mammoth task, however. A safe course through this will take some very 
> careful planning.
>
> --
> C
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f4224b99-6207-4916-a203-d8596f7f9211n%40googlegroups.com.

Re: Model-level validation

[Carl Meyer]

On Thu, Sep 29, 2022 at 6:19 PM Curtis Maloney  wrote:

> On Thu, 29 Sep 2022, at 14:29, Aaron Smith wrote:
>
> Why doesn't Django validate Models on save()?
>
>
> The short answer is: backwards compatibility.
>
> Model level validation hasn't always been a thing, with Django initially
> depending primarily on Form validation.
>
> Since it hadn't _always_ been there, it wasn't possible to introduce it,
> enforce it, and not break most apps out there.
>
> There was so much code written that generally assumed it could call
> `save()` and not have to catch validation errors.
>
> For what it's worth, I'm all in favor of making it run on `save()` ...
> updating the documentation and preparing the community is going to be a
> mammoth task, however. A safe course through this will take some very
> careful planning.
>

Another factor that should be considered is that the Django ORM gives you
plenty of ways to update your database (eg bulk updates on a queryset) that
clearly cannot materialize and validate every object in the queryset. So is
it better to consistently say “the ORM doesn’t validate, forms do,” or
better to say “the ORM will validate on Model.save() but not in various
other cases, so you still can’t really rely on the ORM to enforce model
validation invariants consistently.”

Carl

>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAGLapGA7NCkNrBNe7LQmTOzbj%2Bpa3xm7_tEPnBa7eZbUsq1xGg%40mail.gmail.com.

Re: Model-level validation

[Curtis Maloney]

On Thu, 29 Sep 2022, at 14:29, Aaron Smith wrote:
> Why doesn't Django validate Models on save()?

The short answer is: backwards compatibility.

Model level validation hasn't always been a thing, with Django initially 
depending primarily on Form validation.

Since it hadn't _always_ been there, it wasn't possible to introduce it, 
enforce it, and not break most apps out there.

There was so much code written that generally assumed it could call `save()` 
and not have to catch validation errors.

For what it's worth, I'm all in favor of making it run on `save()` ... updating 
the documentation and preparing the community is going to be a mammoth task, 
however. A safe course through this will take some very careful planning.

--
C

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/10d789b9-5f10-4823-98e8-0494c8510bf7%40app.fastmail.com.

Re: Model-level validation

[Aaron Smith]

Yes, I did search, and I did not find an answer to my question.

If one is always supposed to use a ModelForm, why isn't that ModelForm 
functionality part of the Model?
On Thursday, September 29, 2022 at 1:04:17 AM UTC-7 carlton...@gmail.com 
wrote:

> Hi. 
>
> I have to ask, did you search the history at all here? This has been 
> discussed *several times* over the years. 
>
> > Bringing View-level concepts such as forms down into celery tasks and 
> management commands breaks separation of concerns...
>
> I think it's an error to think of forms (please read "or serializers" 
> throughout) as "view-level". 
> A view's job is to turn requests into responses. 
> It may use other layers of Django, such as the ORM to do that, but that 
> doesn't make said layers part of the view. (The ORM is not "view-level".)
>
> Forms are such another level. Their role is to validate (and transform) 
> incoming data, and they provide an opportunity to present user-friendly 
> validation errors back up to the client code. 
> You should definitely be using them in all places that you process 
> incoming data, and that includes celery tasks. 
>
> On Thursday, 29 September 2022 at 06:29:30 UTC+2 aa...@aaronsmith.co 
> wrote:
>
>> Why doesn't Django validate Models on save()?
>>
>> I am aware that full_clean() is called when using ModelForms. But most 
>> web app development these days, and every django app I've ever worked with, 
>> are headless APIs. The default behavior is dangerous for the naive 
>> developer.
>>
>> Bringing View-level concepts such as forms or serializers down into 
>> celery tasks and management commands breaks separation of concerns, and 
>> having multiple validation implementations at different layers in the app 
>> is fraught with divergence and unexpected behavior.
>>
>> It's not right. The data store layer should protect the validity of the 
>> data.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e61463d2-bfd9-497c-af90-c281e7291ce3n%40googlegroups.com.

Model-level validation

[Aaron Smith]

Why doesn't Django validate Models on save()?

I am aware that full_clean() is called when using ModelForms. But most web 
app development these days, and every django app I've ever worked with, are 
headless APIs. The default behavior is dangerous for the naive developer.

Bringing View-level concepts such as forms or serializers down into celery 
tasks and management commands breaks separation of concerns, and having 
multiple validation implementations at different layers in the app is 
fraught with divergence and unexpected behavior.

It's not right. The data store layer should protect the validity of the 
data.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/37ec0c58-2561-4300-9ead-05160410c389n%40googlegroups.com.

Re: Model-level validation

[Aaron Smith]

All I was able to find was that it was for "performance reasons", and I 
refuse to believe that a mature web framework like Django would prioritize 
performance (let's face it - even my millions-of-requests-per-day API 
service doesn't care about a few extra milliseconds here and there) over 
the most basic data safety concerns by default. My expectation would be 
that validation would be opt-out, not opt-in. There must be another reason.
On Thursday, September 29, 2022 at 1:04:17 AM UTC-7 carlton...@gmail.com 
wrote:

> Hi. 
>
> I have to ask, did you search the history at all here? This has been 
> discussed *several times* over the years. 
>
> > Bringing View-level concepts such as forms down into celery tasks and 
> management commands breaks separation of concerns...
>
> I think it's an error to think of forms (please read "or serializers" 
> throughout) as "view-level". 
> A view's job is to turn requests into responses. 
> It may use other layers of Django, such as the ORM to do that, but that 
> doesn't make said layers part of the view. (The ORM is not "view-level".)
>
> Forms are such another level. Their role is to validate (and transform) 
> incoming data, and they provide an opportunity to present user-friendly 
> validation errors back up to the client code. 
> You should definitely be using them in all places that you process 
> incoming data, and that includes celery tasks. 
>
> On Thursday, 29 September 2022 at 06:29:30 UTC+2 aa...@aaronsmith.co 
> wrote:
>
>> Why doesn't Django validate Models on save()?
>>
>> I am aware that full_clean() is called when using ModelForms. But most 
>> web app development these days, and every django app I've ever worked with, 
>> are headless APIs. The default behavior is dangerous for the naive 
>> developer.
>>
>> Bringing View-level concepts such as forms or serializers down into 
>> celery tasks and management commands breaks separation of concerns, and 
>> having multiple validation implementations at different layers in the app 
>> is fraught with divergence and unexpected behavior.
>>
>> It's not right. The data store layer should protect the validity of the 
>> data.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/46a799fe-a9c7-4d8b-aff3-29e86be25b57n%40googlegroups.com.

Re: git tag missing for django 4.1.1?

[Mariusz Felisiak]

Thanks for noticing, I added a missing tag.

Best,
Mariusz

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/920200a0-3d32-49ac-ba7f-f392f1b25758n%40googlegroups.com.

git tag missing for django 4.1.1?

[John Speno]

Hi folks,

According to the Django's release process document[1], there should be a
signed tag for 4.1.1 yet I do not see it listed:
https://github.com/django/django/tags

Thanks.

[1] https://docs.djangoproject.com/en/dev/internals/release-process/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMfGPjSgzX8AtxQqBBg5sfmhwe8b3ikRDW%3DD14CsMK9v0rQGjA%40mail.gmail.com.

Re: Model-level validation

[אורי]

אורי
u...@speedy.net


On Thu, Sep 29, 2022 at 11:04 AM Carlton Gibson 
wrote:

>
> On Thursday, 29 September 2022 at 06:29:30 UTC+2 aa...@aaronsmith.co
> wrote:
>
>> Why doesn't Django validate Models on save()?
>>
>> I am aware that full_clean() is called when using ModelForms. But most
>> web app development these days, and every django app I've ever worked with,
>> are headless APIs. The default behavior is dangerous for the naive
>> developer.
>>
>> Bringing View-level concepts such as forms or serializers down into
>> celery tasks and management commands breaks separation of concerns, and
>> having multiple validation implementations at different layers in the app
>> is fraught with divergence and unexpected behavior.
>>
>> It's not right. The data store layer should protect the validity of the
>> data.
>>
>
I haven't received the original message but only the reply from Carlton
Gibson.

In Speedy Net, all the models inherit from a BaseModel which inherits
from ValidateModelMixin which runs self.full_clean() on save().
https://github.com/speedy-net/speedy-net/blob/master/speedy/core/base/models.py#L7-L16

I was surprised that this is not a default in Django and I think it should
be - models should validate their values before saving to the database.

I also disabled bulk actions such as bulk_create() to avoid saving to the
database without calling save().

Personally, I think this is a must in Django - calling save() for each
instance and calling self.full_clean() to validate data in save().

Uri Rodberg, Speedy Net.



> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/566b4706-4075-485b-9122-eca24d3b67fcn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CABD5YeH3jb_7soEcMKM5qdHyvqWxXDCA3%3D%3DZ%2BPBC47YMtTAggg%40mail.gmail.com.

Re: Proposal: Add utility class "ClassList"

[Carlton Gibson]

Hey Jacob. Thanks for this. 

Can I ask you to give a few examples of potential usages in Django, and 
showing the gain over use a set in these cases? 

I'm trying to imagine exactly what you have in mind, but I'm not entirely 
clear. 

Thanks again. 
Carlton

On Friday, 23 September 2022 at 11:14:27 UTC+2 Jacob Rief wrote:

> In JavaScript each HTMLElement has a property named classList 
> . 
> This actually is a set allowing to *add* a single CSS class string, 
> *remove* it
> and/or *toggle* it.
>
> If we would reimplement this as a Python class, methods such as
> css_classes 
> 
>  
> could be implemented as a one-liner. This would also be beneficial
> for future uses of similar methods in Django and 3rd party libraries, 
> because it is a quite
> common use case that one has to change the list of CSS classes as an 
> element
> attribute.
>
> – Jacob
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d63a27a1-6a67-403e-9ecd-f03b19e8660dn%40googlegroups.com.

Re: Model-level validation

[Carlton Gibson]

Hi. 

I have to ask, did you search the history at all here? This has been 
discussed *several times* over the years. 

> Bringing View-level concepts such as forms down into celery tasks and 
management commands breaks separation of concerns...

I think it's an error to think of forms (please read "or serializers" 
throughout) as "view-level". 
A view's job is to turn requests into responses. 
It may use other layers of Django, such as the ORM to do that, but that 
doesn't make said layers part of the view. (The ORM is not "view-level".)

Forms are such another level. Their role is to validate (and transform) 
incoming data, and they provide an opportunity to present user-friendly 
validation errors back up to the client code. 
You should definitely be using them in all places that you process incoming 
data, and that includes celery tasks. 

On Thursday, 29 September 2022 at 06:29:30 UTC+2 aa...@aaronsmith.co wrote:

> Why doesn't Django validate Models on save()?
>
> I am aware that full_clean() is called when using ModelForms. But most web 
> app development these days, and every django app I've ever worked with, are 
> headless APIs. The default behavior is dangerous for the naive developer.
>
> Bringing View-level concepts such as forms or serializers down into celery 
> tasks and management commands breaks separation of concerns, and having 
> multiple validation implementations at different layers in the app is 
> fraught with divergence and unexpected behavior.
>
> It's not right. The data store layer should protect the validity of the 
> data.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/566b4706-4075-485b-9122-eca24d3b67fcn%40googlegroups.com.