Re: Model-level validation

[אורי]

אורי
u...@speedy.net


On Thu, Oct 6, 2022 at 6:11 AM Aaron Smith  wrote:

> It sounds like there is little support for this being the default. But I'd
> like to propose something that might satisfy the different concerns:
>
> 1) A `validate` kwarg for `save()`, defaulted to `False`. This maintains
> backwards compatibility and also moves the validation behavior users coming
> to Django from other frameworks likely expect, in a more user friendly way
> than overriding save to call `full_clean()`.
>
> And/or...
>
> 2) An optional Django setting (`VALIDATE_MODELS_DEFAULT`?) to change the
> default behavior to `True`. The `validate` kwarg above would override this
> per call, allowing unvalidated saves when necessary.
>
> These changes would be simple, backwards compatible, and give individual
> projects the choice to make Django behave like other ORMs with regard to
> validation. This being the Django developers mailing list I should not be
> surprised that most people here support the status quo, but in my personal
> experience, having had this conversation with dozens of coworkers over the
> years - 100% of them expressed a strong desire for Django to do this
> differently.
>

+1

I would suggest having a setting "VALIDATE_MODELS_BY_DEFAULT", which is
true or false (true by default), whether to call full_clean() on save(),
with an option to call it with "validate=True" or "validate=False" to
override this default. Maybe also allow changing the default for specific
models.

This is similar to forms that have `def save(self, commit=True):`, and you
can call them with "commit=True" or "commit=False" to save or not save the
results to the database. I also suggest that VALIDATE_MODELS_BY_DEFAULT
will be true by default from some specific future version of Django, so
that if users don't want it, they will have to manually set it to false.

We should still remember that there are bulk actions such as bulk_create()
or update(), that bypass save() completely, so we have to decide how to
handle them if we want our data to be always validated.

Uri Rodberg, Speedy Net.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CABD5YeGHLfzA%3DV%3DosAFM1wnitJ6R20sqc5ddVCCWarBv%2Bk1j3A%40mail.gmail.com.

Re: Model-level validation

[Curtis Maloney]

FWIW +1 from me!

--
C


On Thu, 6 Oct 2022, at 14:11, Aaron Smith wrote:
> It sounds like there is little support for this being the default. But I'd 
> like to propose something that might satisfy the different concerns:
> 
> 1) A `validate` kwarg for `save()`, defaulted to `False`. This maintains 
> backwards compatibility and also moves the validation behavior users coming 
> to Django from other frameworks likely expect, in a more user friendly way 
> than overriding save to call `full_clean()`.
> 
> And/or...
> 
> 2) An optional Django setting (`VALIDATE_MODELS_DEFAULT`?) to change the 
> default behavior to `True`. The `validate` kwarg above would override this 
> per call, allowing unvalidated saves when necessary.
> 
> These changes would be simple, backwards compatible, and give individual 
> projects the choice to make Django behave like other ORMs with regard to 
> validation. This being the Django developers mailing list I should not be 
> surprised that most people here support the status quo, but in my personal 
> experience, having had this conversation with dozens of coworkers over the 
> years - 100% of them expressed a strong desire for Django to do this 
> differently.
> On Wednesday, September 28, 2022 at 9:29:30 PM UTC-7 Aaron Smith wrote:
>> Why doesn't Django validate Models on save()?
>> 
>> I am aware that full_clean() is called when using ModelForms. But most web 
>> app development these days, and every django app I've ever worked with, are 
>> headless APIs. The default behavior is dangerous for the naive developer.
>> 
>> Bringing View-level concepts such as forms or serializers down into celery 
>> tasks and management commands breaks separation of concerns, and having 
>> multiple validation implementations at different layers in the app is 
>> fraught with divergence and unexpected behavior.
>> 
>> It's not right. The data store layer should protect the validity of the data.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/004c2b9b-6e36-4061-b860-17eb7ebee15en%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/0b25292a-4197-451f-926f-f35a527b5d6d%40app.fastmail.com.

Re: Model-level validation

[Aaron Smith]

It sounds like there is little support for this being the default. But I'd 
like to propose something that might satisfy the different concerns:

1) A `validate` kwarg for `save()`, defaulted to `False`. This maintains 
backwards compatibility and also moves the validation behavior users coming 
to Django from other frameworks likely expect, in a more user friendly way 
than overriding save to call `full_clean()`.

And/or...

2) An optional Django setting (`VALIDATE_MODELS_DEFAULT`?) to change the 
default behavior to `True`. The `validate` kwarg above would override this 
per call, allowing unvalidated saves when necessary.

These changes would be simple, backwards compatible, and give individual 
projects the choice to make Django behave like other ORMs with regard to 
validation. This being the Django developers mailing list I should not be 
surprised that most people here support the status quo, but in my personal 
experience, having had this conversation with dozens of coworkers over the 
years - 100% of them expressed a strong desire for Django to do this 
differently.
On Wednesday, September 28, 2022 at 9:29:30 PM UTC-7 Aaron Smith wrote:

> Why doesn't Django validate Models on save()?
>
> I am aware that full_clean() is called when using ModelForms. But most web 
> app development these days, and every django app I've ever worked with, are 
> headless APIs. The default behavior is dangerous for the naive developer.
>
> Bringing View-level concepts such as forms or serializers down into celery 
> tasks and management commands breaks separation of concerns, and having 
> multiple validation implementations at different layers in the app is 
> fraught with divergence and unexpected behavior.
>
> It's not right. The data store layer should protect the validity of the 
> data.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/004c2b9b-6e36-4061-b860-17eb7ebee15en%40googlegroups.com.

Re: Why using django.contrib.sessions as the salt to encode session data? why not secret key?

['John Whitlock' via Django developers (Contributions to Django itself)]

Looking at the code, the hard-coded salt seems OK to me. The encoding is
done by SessionBase.encode()
,
which calls dumps() from django.core.signing
.
The docstring says:






*Return URL-safe, hmac signed base64 compressed JSON string. If key is
None, use settings.SECRET_KEY instead. The hmac algorithm is the default
Signer algorithm.If compress is True (not the default), check if
compressing using zlib can save some space. Prepend a '.' to signify
compression. This is included in the signature, to protect against zip
bombs.Salt can be used to namespace the hash, so that a signed string is
only valid for a given namespace. Leaving this at the default value or
re-using a salt value across different parts of your application without
good cause is a security risk.*

*The serializer is expected to return a bytestring.*

So, settings.SECRET_KEY is used to sign the session, and salt acts like a
namespace to distinguish sessions from other uses of the SECRET_KEY.

Trying it out in `./manage.py shell`:

>>> from django.contrib.sessions.backends.db import SessionStore
>>> from django.core.signing import b64_decode
>>> session = SessionStore().encode({"foo": "bar"})
>>> print(session)
eyJmb28iOiJiYXIifQ:1ogD1v:OIpRWKZdxbhox3d7XaS7A0bZouEXQV6jx03FlAaDGZo
>>> val, ts, sign = session.split(':')
SessionStore().encode({"foo": "bar"})
>>> b64_decode(val.encode())
b'{"foo":"bar"}'
>>> b64_decode(ts.encode())
b'\xd6\x88\x03\xd6'
>>> b64_decode(sign.encode())
b'8\x8aQX\xa6]\xc5\xb8h\xc7w{]\xa4\xbb\x03F\xd9\xa2\xe1\x17A^\xa3\xc7M\xc5\x94\x06\x83\x19\x9a'
>>> len(b64_decode(z.encode()))
32

The first section of the session value is the encoded value, base64
encoded, and potentially compressed.
The second section is the encoded timestamp, used to determine if it was
created too long ago on decode
The third section is the HMAC signature, base64 encoded.

- John

On Wed, Oct 5, 2022 at 4:45 PM 'Adam Johnson' via Django developers
(Contributions to Django itself)  wrote:

> Looking through blame, it looks like this hardcoded salt was added in
> 2010:
> https://github.com/django/django/commit/45c7f427ce830dd1b2f636fb9c244fda9201cadb#diff-3a6c11bbe36a0e6927f71ad8d669f0021897ba73768ee41073a318a12e11c3d1L85-L90
>
> This actually changed from using the secret key as the salt, to the fixed
> string, whilst also changing the algorithm from MAC to HMAC. But I cannot
> see any discussion on why in the ticket:
> https://code.djangoproject.com/ticket/14445 .
>
> 路‍♂️
>
> On Mon, Oct 3, 2022 at 8:43 AM Lokesh Sanapalli 
> wrote:
>
>> Hi,
>>
>> I was going through the code and got a question. I saw that we are using
>> hard-coded string `django.contrib.sessions` as the key salt to encode
>> session data
>> .
>> Why not using the secret key? as the secret key is specific to environment
>> and project it serves as a good candidate. Is it because the session data
>> does not contain any sensitive info (it only contains user id and other
>> info) so that's why this decision is made?
>>
>> Thanks & Regards,
>> Lokesh Sanpalli
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/6c6544b7-a190-4198-9108-6c66fac213ebn%40googlegroups.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAMyDDM1C6HxOVYrVL2XuBOhjWUb0EnThvzSAzocdKXrr%2BjkNEg%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 

Re: Why using django.contrib.sessions as the salt to encode session data? why not secret key?

['Adam Johnson' via Django developers (Contributions to Django itself)]

Looking through blame, it looks like this hardcoded salt was added in 2010:
https://github.com/django/django/commit/45c7f427ce830dd1b2f636fb9c244fda9201cadb#diff-3a6c11bbe36a0e6927f71ad8d669f0021897ba73768ee41073a318a12e11c3d1L85-L90

This actually changed from using the secret key as the salt, to the fixed
string, whilst also changing the algorithm from MAC to HMAC. But I cannot
see any discussion on why in the ticket:
https://code.djangoproject.com/ticket/14445 .

路‍♂️

On Mon, Oct 3, 2022 at 8:43 AM Lokesh Sanapalli 
wrote:

> Hi,
>
> I was going through the code and got a question. I saw that we are using
> hard-coded string `django.contrib.sessions` as the key salt to encode
> session data
> .
> Why not using the secret key? as the secret key is specific to environment
> and project it serves as a good candidate. Is it because the session data
> does not contain any sensitive info (it only contains user id and other
> info) so that's why this decision is made?
>
> Thanks & Regards,
> Lokesh Sanpalli
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/6c6544b7-a190-4198-9108-6c66fac213ebn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM1C6HxOVYrVL2XuBOhjWUb0EnThvzSAzocdKXrr%2BjkNEg%40mail.gmail.com.

Re: Making max_length argument optional

[Carlton Gibson]

Hi Adrian.

Nothing has been done, no.

There seem to be a few competing options:

1. Allow max_length=None on Postgres (using a
supports_unlimited_charfields, name!?, DB backend feature flag maybe 樂)
2. Add an unlimited flag to CharField.
3. Have a subclass in contrib.postgres, similar to JKM's version Tom linked
before


My guess would be 1 or 3. (I'd rather 3 than 2.) So proof-of-concepts
implementing those would be the way forward I think.

Kind Regards,

Carlton


On Wed, 5 Oct 2022 at 12:47, Adrian Torres  wrote:

> Sorry for reviving the thread but just ran into this issue again and
> wanted to ask, has there anything been done to address this?
>
> For me having max_length=None mean "unlimited varchar" for Postgres and
> being an error for any other database (or those that don't support
> unlimited varchar) is the best solution, but I'm also ok with Carlton's
> idea of having a subclass in contrib.postgres is an acceptable compromise.
>
> Cheers,
> Adrian
>
> On Monday, August 17, 2020 at 11:26:39 AM UTC+2 t...@carrick.eu wrote:
>
>> It would work for my use-cases. It was mentioned that it's maybe not the
>> best as a lot of fields subclass CharField, but I don't see a compelling
>> argument for an unlimited email or slug field. The one problem I see is
>> CICharField, you might want to make that unlimited - but the CI fields are
>> more or less redundant on pg >= 12 once we have column collations in. So I
>> think it's okay for now. Only problem I can see is if we add more CharField
>> subclasses in the future...
>>
>> On Mon, 17 Aug 2020 at 11:02, Carlton Gibson 
>> wrote:
>>
>>> Would the subclass in contrib.postgres suggestion be acceptable?
>>>
>>> On Mon, 17 Aug 2020 at 10:31, Tom Carrick  wrote:
>>>
 I'm not a fan of any solution either, really, even the one I suggested.

 I think adding a new kwarg, "unlimited" seems okay to me, though it
 feels a little redundant.

 I don't like the idea of using TextField, I find them semantically
 different. An unlimited varchar says to me "one possibly very long thing",
 whereas TextField feels more like it's free text, or a document, especially
 as the form fields are different. Subclassing CharField is an option, but
 the methods I've seen to do this makes it annoying. I need this so often
 that I do it all the time, but the code is so short that I don't want to
 bring in a new package to do it. Also, the popular ways to do this are not
 great. One way is to just set the max_length extremely high, which is not
 what I want ending up in the database, the other is something like this
 , which
 works well, but will stop working well once column collations are in as
 that PR adds more stuff to CharField.__init__().

 I think it's time we had something in Django, whatever that ends up
 being.

 On Sun, 16 Aug 2020 at 20:28, Carlton Gibson 
 wrote:

> Reading the history, I see an awful lot of -1s to the idea of a
> default.
>
> I see “use a TextField” and “use a subclass” a few times, which were
> my immediate thoughts just on the recent emails.
>
> On Sun, 16 Aug 2020 at 18:47, Tom Forbes  wrote:
>
>> I’m not a fan of implicit max_lengths. Is having to add a keyword
>> argument to a model field really that much of a burden? And we also would
>> likely never be able to change the default without headaches.
>>
>> On 12 Aug 2020, at 13:19, t...@carrick.eu  wrote:
>>
>> I'd like to revive this discussion and try to come to a consensus as
>> it's something I find myself wishing for (for Postgres in particular).
>>
>> My suggestion, after reading through everything:
>>
>> Give CharField a default max_length that is consistent across all
>> vendors. It doesn't really matter what the number is other than that it
>> should be large enough to be useful but small enough to work everywhere. 
>> I
>> think 100 or 255 are both fine options.
>>
>> If you set max_length=None explicitly, on Postgres this will use an
>> unlimited varchar, on everything else will raise an exception on migrate.
>>
>> Any thoughts?
>>
>> Cheers,
>> Tom
>>
>> On Saturday, March 5, 2016 at 2:13:14 PM UTC+1 Shai Berger wrote:
>>
>>> On Saturday 05 March 2016 02:24:17 Loïc Bistuer wrote:
>>>
>>>
>>> > I’m not too keen on a contrib.pg field. CharField is the base
>>> class of many
>>>
>>>
>>> > fields, a __init__ kwarg approach such as max_length=None allows
>>> us to
>>>
>>>
>>> > reach those as well.
>>>
>>>
>>> >
>>>
>>>
>>>
>>>
>>>
>>> That's a good point; Can we enable max_length=None in a mixin?
>>>

Re: Making max_length argument optional

[Adrian Torres]

Sorry for reviving the thread but just ran into this issue again and wanted 
to ask, has there anything been done to address this?

For me having max_length=None mean "unlimited varchar" for Postgres and 
being an error for any other database (or those that don't support 
unlimited varchar) is the best solution, but I'm also ok with Carlton's 
idea of having a subclass in contrib.postgres is an acceptable compromise.

Cheers,
Adrian

On Monday, August 17, 2020 at 11:26:39 AM UTC+2 t...@carrick.eu wrote:

> It would work for my use-cases. It was mentioned that it's maybe not the 
> best as a lot of fields subclass CharField, but I don't see a compelling 
> argument for an unlimited email or slug field. The one problem I see is 
> CICharField, you might want to make that unlimited - but the CI fields are 
> more or less redundant on pg >= 12 once we have column collations in. So I 
> think it's okay for now. Only problem I can see is if we add more CharField 
> subclasses in the future...
>
> On Mon, 17 Aug 2020 at 11:02, Carlton Gibson  wrote:
>
>> Would the subclass in contrib.postgres suggestion be acceptable? 
>>
>> On Mon, 17 Aug 2020 at 10:31, Tom Carrick  wrote:
>>
>>> I'm not a fan of any solution either, really, even the one I suggested.
>>>
>>> I think adding a new kwarg, "unlimited" seems okay to me, though it 
>>> feels a little redundant.
>>>
>>> I don't like the idea of using TextField, I find them semantically 
>>> different. An unlimited varchar says to me "one possibly very long thing", 
>>> whereas TextField feels more like it's free text, or a document, especially 
>>> as the form fields are different. Subclassing CharField is an option, but 
>>> the methods I've seen to do this makes it annoying. I need this so often 
>>> that I do it all the time, but the code is so short that I don't want to 
>>> bring in a new package to do it. Also, the popular ways to do this are not 
>>> great. One way is to just set the max_length extremely high, which is not 
>>> what I want ending up in the database, the other is something like this 
>>> , which 
>>> works well, but will stop working well once column collations are in as 
>>> that PR adds more stuff to CharField.__init__().
>>>
>>> I think it's time we had something in Django, whatever that ends up 
>>> being.
>>>
>>> On Sun, 16 Aug 2020 at 20:28, Carlton Gibson  
>>> wrote:
>>>
 Reading the history, I see an awful lot of -1s to the idea of a 
 default. 

 I see “use a TextField” and “use a subclass” a few times, which were my 
 immediate thoughts just on the recent emails. 

 On Sun, 16 Aug 2020 at 18:47, Tom Forbes  wrote:

> I’m not a fan of implicit max_lengths. Is having to add a keyword 
> argument to a model field really that much of a burden? And we also would 
> likely never be able to change the default without headaches.
>
> On 12 Aug 2020, at 13:19, t...@carrick.eu  wrote:
>
> I'd like to revive this discussion and try to come to a consensus as 
> it's something I find myself wishing for (for Postgres in particular).
>
> My suggestion, after reading through everything:
>
> Give CharField a default max_length that is consistent across all 
> vendors. It doesn't really matter what the number is other than that it 
> should be large enough to be useful but small enough to work everywhere. 
> I 
> think 100 or 255 are both fine options.
>
> If you set max_length=None explicitly, on Postgres this will use an 
> unlimited varchar, on everything else will raise an exception on migrate.
>
> Any thoughts?
>
> Cheers,
> Tom
>
> On Saturday, March 5, 2016 at 2:13:14 PM UTC+1 Shai Berger wrote:
>
>> On Saturday 05 March 2016 02:24:17 Loïc Bistuer wrote:
>>
>>
>> > I’m not too keen on a contrib.pg field. CharField is the base 
>> class of many
>>
>>
>> > fields, a __init__ kwarg approach such as max_length=None allows us 
>> to
>>
>>
>> > reach those as well.
>>
>>
>> > 
>>
>>
>>
>>
>>
>> That's a good point; Can we enable max_length=None in a mixin?
>>
>>
>>
>
>
>
>
> -- 
>
>
> You received this message because you are subscribed to the Google 
> Groups "Django developers (Contributions to Django itself)" group.
>
>
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to django-develop...@googlegroups.com.
>
>
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/dfaaa9d3-dff3-46a3-899f-dd7f4eddfe87n%40googlegroups.com
>  
> 
> .
>
>
>
>
>