Re: Why was CommaSeparatedIntegerField Deprecated?

[Aymeric Augustin]

Hello,

Storing a comma-separated list of integers in a text field in a SQL database is 
a really bad anti-pattern. That's a sufficient reason for Django to stop 
condoning it, which it did by providing that functionality out of the box.

It was widely accepted (at least among the core team) that 
CommaSeparatedIntegerField shouldn't be a core Django field because it's both 
weird and specialized. That's why no one found useful to write down the reasons,

There's a comment saying that CSIF doesn't belong to core Django since the 
magic-removal refactor, just over 11 years ago:
https://github.com/django/django/commit/f69cf70ed813a8cd7e1f963a14ae39103e8d5265#diff-bf776a3b8e5dbfac2432015825ef8afeR393

It already existed when Django was open sourced, making it hard to trace why it 
was introduced — likely to support some sort of one-to-many field?
https://github.com/django/django/commit/ed114e1510#diff-5b08569363cc72ee12188500cecddf69R731

I hope this helps,

-- 
Aymeric.



> On 15 May 2017, at 16:14, Donald Morrone  wrote:
> 
> I see that CommaSeparatedIntegerField is being deprecated, but I don't see 
> why. I know I'm apparently late to the discussion, but after it 16 months I 
> don't see any actual reasoning.
> 
> I recognize that a CharField with the validator is functionally equivalent, 
> but it takes a very handy field type and makes it completely inaccessible and 
> invisible to new users.
> 
> I see this discussion: 
> https://groups.google.com/forum/#!topic/django-developers/wfp9qNpNpaQ/discussion
>  which lead to this ticket: https://code.djangoproject.com/ticket/26154
> 
> Both links focus on the how rather than the why. 
> 
> I disagree with the decision overall, but I'm more concerned that it just 
> kind of slipped through without anyone really thinking about it.
> 
> (To be clear, I don't begrudge Tim Graham, he was right to propose it, but 
> then there was no discussion on it at all that I saw)
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to django-developers@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/django-developers 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/3d54fb68-5548-424f-8806-103c026b5922%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/FF0E8034-5BE7-4A47-8548-F0DE501F6940%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: Why was CommaSeparatedIntegerField Deprecated?

[Adam Johnson]

I agree with the removal, it wasn't providing much value and was making it
unclear as to what it actually aws. It could easily be mistaken by a
beginner for a performant alternative to M2M models or something like
django.contrib.postgres' ArrayField. Additionally it was weird that Django
would offer a single model field that's just differs from another by a
single validator function - why not offer CommaSeparatedStringField,
OddIntegerField, etc. ?

On 15 May 2017 at 18:21, Tim Graham  wrote:

> The CharField(validators=[validate_comma_separated_integer_list]) version
> is more explicit about how the database stores the data. In my mind, having
> a separate field could give the impression that the database uses a special
> field type or validation. Also, I think
> CommaSeparatedIntegerField is/was generally something to avoid -- I've
> never seen it used myself.
>
> On Monday, May 15, 2017 at 12:57:59 PM UTC-4, Donald Morrone wrote:
>>
>> I see that CommaSeparatedIntegerField is being deprecated, but I don't
>> see why. I know I'm apparently late to the discussion, but after it 16
>> months I don't see any actual reasoning.
>>
>> I recognize that a CharField with the validator is functionally
>> equivalent, but it takes a very handy field type and makes it completely
>> inaccessible and invisible to new users.
>>
>> I see this discussion: https://groups.google.com/forum/#!topic/django-
>> developers/wfp9qNpNpaQ/discussion which lead to this ticket:
>> https://code.djangoproject.com/ticket/26154
>>
>> Both links focus on the *how* rather than the *why*.
>>
>> I disagree with the decision overall, but I'm more concerned that it just
>> kind of slipped through without anyone really thinking about it.
>>
>> (To be clear, I don't begrudge Tim Graham, he was right to propose it,
>> but then there was no discussion on it at all that I saw)
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/6e661a98-df55-4be5-852e-
> 99788df05945%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM2F5rvBRUhm0k8NEGp8a%3DD%2B5WPtM%3DgV2%3DRz3mK_1rZXdQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Are there use cases for storing null bytes in CharField/TextField?

[Claude Paroz]

I also think that this should be handled at serialization level (form 
fields and (de)serialization framework).

Claude

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a9dde655-4bb1-409f-883e-6f47f742f17a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Are there use cases for storing null bytes in CharField/TextField?

[Tim Chase]

On 2017-05-15 08:54, Tim Graham wrote:
> Does anyone know of a use case for using null bytes in
> CharField/TextField?

Is this not what BinaryField is for?  It would seem to me that
attempting to store binary NULL bytes in a CharField/TextField should
result in an error condition.

-tkc



-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20170515123050.363a2859%40bigbox.christie.dr.
For more options, visit https://groups.google.com/d/optout.

Re: Why was CommaSeparatedIntegerField Deprecated?

[Tim Graham]

The CharField(validators=[validate_comma_separated_integer_list]) version 
is more explicit about how the database stores the data. In my mind, having 
a separate field could give the impression that the database uses a special 
field type or validation. Also, I think 
CommaSeparatedIntegerField is/was generally something to avoid -- I've 
never seen it used myself.

On Monday, May 15, 2017 at 12:57:59 PM UTC-4, Donald Morrone wrote:
>
> I see that CommaSeparatedIntegerField is being deprecated, but I don't see 
> why. I know I'm apparently late to the discussion, but after it 16 months I 
> don't see any actual reasoning.
>
> I recognize that a CharField with the validator is functionally 
> equivalent, but it takes a very handy field type and makes it completely 
> inaccessible and invisible to new users.
>
> I see this discussion: 
> https://groups.google.com/forum/#!topic/django-developers/wfp9qNpNpaQ/discussion
>  
> which lead to this ticket: https://code.djangoproject.com/ticket/26154
>
> Both links focus on the *how* rather than the *why*. 
>
> I disagree with the decision overall, but I'm more concerned that it just 
> kind of slipped through without anyone really thinking about it.
>
> (To be clear, I don't begrudge Tim Graham, he was right to propose it, but 
> then there was no discussion on it at all that I saw)
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6e661a98-df55-4be5-852e-99788df05945%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Why was CommaSeparatedIntegerField Deprecated?

[Donald Morrone]

I see that CommaSeparatedIntegerField is being deprecated, but I don't see 
why. I know I'm apparently late to the discussion, but after it 16 months I 
don't see any actual reasoning.

I recognize that a CharField with the validator is functionally equivalent, 
but it takes a very handy field type and makes it completely inaccessible 
and invisible to new users.

I see this 
discussion: 
https://groups.google.com/forum/#!topic/django-developers/wfp9qNpNpaQ/discussion
 
which lead to this ticket: https://code.djangoproject.com/ticket/26154

Both links focus on the *how* rather than the *why*. 

I disagree with the decision overall, but I'm more concerned that it just 
kind of slipped through without anyone really thinking about it.

(To be clear, I don't begrudge Tim Graham, he was right to propose it, but 
then there was no discussion on it at all that I saw)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3d54fb68-5548-424f-8806-103c026b5922%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Are there use cases for storing null bytes in CharField/TextField?

[Luke Plant]

I agree with Adam, we should never silently change submitted data at the 
model layer. My preference would be c), a form-level validation error 
that prevents saving.


Luke


On 15/05/17 19:11, Adam Johnson wrote:
The problem with (a) - data with null bytes in strings from other 
databases can't be loaded into PG as per #28117 .


The problem with (b) - data currently in databases in the wild will be 
modified upon save 


(b) is incredibly destructive and could break an unknown number of 
applications whilst (a) doesn't affect anyone until they try to 
migrate null-byte-strings into PG. I vote for (a), or (c) add 
form-level validation to (Char/Text)Field that null bytes aren't in 
the submitted string (for all databases) and error when trying to save 
them on PG.



On 15 May 2017 at 16:54, Tim Graham > wrote:


Does anyone know of a use case for using null bytes in
CharField/TextField?

psycopg2 2.7+ raises ValueError("A string literal cannot contain
NUL (0x00) characters.") when trying to save null bytes [0] and
this exception is unhandled in Django which allow malicious form
submissions to crash [1]. With psycopg2 < 2.7, there is no
exception and null bytes are silently truncated by PostgreSQL.
Other databases that I tested (SQLite, MySQL, Oracle) allow saving
null bytes. This creates possible cross-database compatibility
problems when moving data from those databases to PostgreSQL, e.g.[2].

I propose to have CharField and TextField strip null bytes from
the value either a) only on PostgreSQL or b) on all databases.
Please indicate your preference or suggest another solution.

[0] https://github.com/psycopg/psycopg2/issues/420

[1] https://code.djangoproject.com/ticket/28201
 - Saving a
Char/TextField with psycopg2 2.7+ raises ValueError: A string
literal cannot contain NUL (0x00) characters is unhandled
[2] https://code.djangoproject.com/ticket/28117
 - loaddata raises
ValueError with psycopg2 backend when data contains null bytes
-- 
You received this message because you are subscribed to the Google

Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to django-developers+unsubscr...@googlegroups.com
.
To post to this group, send email to
django-developers@googlegroups.com
.
Visit this group at
https://groups.google.com/group/django-developers
.
To view this discussion on the web visit

https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com

.
For more options, visit https://groups.google.com/d/optout
.




--
Adam
--
You received this message because you are subscribed to the Google 
Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to django-developers+unsubscr...@googlegroups.com 
.
To post to this group, send email to 
django-developers@googlegroups.com 
.

Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM1qVc3ovXb9PhzKY3jd__FURYX6Fy9r1WFrBpcpMy%2Bz%2BA%40mail.gmail.com 
.

For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1fbe9f18-f935-93eb-dd90-ffa754ad9c2b%40cantab.net.
For more options, visit https://groups.google.com/d/optout.

Re: Are there use cases for storing null bytes in CharField/TextField?

[Michael Manfre]

I imagine we won't hear of a use case until after the change happens and
I'm some what strongly opposed to stripping potentially valid data from all
databases because of a limitation of one. I'd be in favor of loaddata
checking for null bytes and complaining when the backend doesn't support
that feature.

Regards,
Michael Manfre

On Mon, May 15, 2017 at 11:54 AM Tim Graham  wrote:

> Does anyone know of a use case for using null bytes in CharField/TextField?
>
> psycopg2 2.7+ raises ValueError("A string literal cannot contain NUL
> (0x00) characters.") when trying to save null bytes [0] and this
> exception is unhandled in Django which allow malicious form submissions to
> crash [1]. With psycopg2 < 2.7, there is no exception and null bytes are
> silently truncated by PostgreSQL. Other databases that I tested (SQLite,
> MySQL, Oracle) allow saving null bytes. This creates possible
> cross-database compatibility problems when moving data from those databases
> to PostgreSQL, e.g.[2].
>
> I propose to have CharField and TextField strip null bytes from the value
> either a) only on PostgreSQL or b) on all databases. Please indicate your
> preference or suggest another solution.
>
> [0] https://github.com/psycopg/psycopg2/issues/420
> [1] https://code.djangoproject.com/ticket/28201 - Saving a Char/TextField
> with psycopg2 2.7+ raises ValueError: A string literal cannot contain NUL
> (0x00) characters is unhandled
> [2] https://code.djangoproject.com/ticket/28117 - loaddata raises
> ValueError with psycopg2 backend when data contains null bytes
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAGdCwBsvwtr4F3j1jGo9uGTwBsjvU0ypLc%2B2q0482Peha3ejzw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Are there use cases for storing null bytes in CharField/TextField?

[Adam Johnson]

The problem with (a) - data with null bytes in strings from other databases
can't be loaded into PG as per #28117 .

The problem with (b) - data currently in databases in the wild will be
modified upon save 

(b) is incredibly destructive and could break an unknown number of
applications whilst (a) doesn't affect anyone until they try to migrate
null-byte-strings into PG. I vote for (a), or (c) add form-level validation
to (Char/Text)Field that null bytes aren't in the submitted string (for all
databases) and error when trying to save them on PG.


On 15 May 2017 at 16:54, Tim Graham  wrote:

> Does anyone know of a use case for using null bytes in CharField/TextField?
>
> psycopg2 2.7+ raises ValueError("A string literal cannot contain NUL
> (0x00) characters.") when trying to save null bytes [0] and this
> exception is unhandled in Django which allow malicious form submissions to
> crash [1]. With psycopg2 < 2.7, there is no exception and null bytes are
> silently truncated by PostgreSQL. Other databases that I tested (SQLite,
> MySQL, Oracle) allow saving null bytes. This creates possible
> cross-database compatibility problems when moving data from those databases
> to PostgreSQL, e.g.[2].
>
> I propose to have CharField and TextField strip null bytes from the value
> either a) only on PostgreSQL or b) on all databases. Please indicate your
> preference or suggest another solution.
>
> [0] https://github.com/psycopg/psycopg2/issues/420
> [1] https://code.djangoproject.com/ticket/28201 - Saving a Char/TextField
> with psycopg2 2.7+ raises ValueError: A string literal cannot contain NUL
> (0x00) characters is unhandled
> [2] https://code.djangoproject.com/ticket/28117 - loaddata raises
> ValueError with psycopg2 backend when data contains null bytes
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/9897126d-b6ef-48f1-9f19-
> 96ed98ce10e5%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM1qVc3ovXb9PhzKY3jd__FURYX6Fy9r1WFrBpcpMy%2Bz%2BA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Are there use cases for storing null bytes in CharField/TextField?

[Tim Graham]

Does anyone know of a use case for using null bytes in CharField/TextField?

psycopg2 2.7+ raises ValueError("A string literal cannot contain NUL (0x00) 
characters.") when trying to save null bytes [0] and this exception is 
unhandled in Django which allow malicious form submissions to crash [1]. With 
psycopg2 < 2.7, there is no exception and null bytes are silently truncated 
by PostgreSQL. Other databases that I tested (SQLite, MySQL, Oracle) allow 
saving null bytes. This creates possible cross-database compatibility 
problems when moving data from those databases to PostgreSQL, e.g.[2].

I propose to have CharField and TextField strip null bytes from the value 
either a) only on PostgreSQL or b) on all databases. Please indicate your 
preference or suggest another solution.

[0] https://github.com/psycopg/psycopg2/issues/420
[1] https://code.djangoproject.com/ticket/28201 - Saving a Char/TextField 
with psycopg2 2.7+ raises ValueError: A string literal cannot contain NUL 
(0x00) characters is unhandled
[2] https://code.djangoproject.com/ticket/28117 - loaddata raises 
ValueError with psycopg2 backend when data contains null bytes

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9897126d-b6ef-48f1-9f19-96ed98ce10e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Project report: Test infrastructure and property-based tests for Channels

[Adam Johnson]

Nice one! 

Happy to see more Hypothesis tests, I've had great experience with it too.

On 15 May 2017 at 11:54,  wrote:

> Hi everyone,
>
> I did some work on Channels' tests and testing infrastructure as part of
> the Mozilla grant. This is my report.
>
> Channels consists of five projects under the Django organization. First of
> all, I agreed with Andrew on versions of Django, Python and Twisted that we
> want to support across all of them. Then I updated Travis and tox
> configurations in all projects to be consistent with each other, and to
> test all agreed-upon version combinations. We also settled on specifying
> test dependencies in an extra in setup.py.
>
> This was important but unexciting work. The only useful takeaway might be
> that it confirmed my view that using Travis to run tox is an anti-pattern.
> It tends to slow down the test runs. I also have seen cases where tox
> didn't update the test environment in response to updated requirements -
> and those cases are hard to debug. Instead, I advocate on simplifying the
> test infrastructure so that there's almost no duplication between Travis
> and tox config - Makefiles or scripts can help with that in more complex
> cases.
>
> The main idea of my proposal was to introduce tests that
> are-close-as-possible translations of the ASGI spec into code. I think this
> wouldn't have yielded much benefit if it weren't for using hypothesis to
> generate test data, aka "property based testing". I was previously
> impressed by the potential of it, and really saw it come to fruition here.
> Hypothesis was annoyingly good at finding edge cases. Think of distinction
> between passing in "None" for headers vs [], for using Unicode in query
> strings , or for making it
> clear that certain characters definitely aren't allowed e.g. in header
> names because they break the underlying HTTP protocol. Last but not least,
> I also introduced a "kitchen sink" (e.g. here
> )
> test for each scenario which basically lets hypothesis try all weird
> combinations by itself, which has highlighted at least one more issue.
> In conclusion, hypothesis let me find corner cases that I wouldn't have
> thought of or would've been too lazy to try, and neatly separated test data
> generation from the actual tests. I vow to use it more in the future :)
>
> While there's always more to test, I think we now have high confidence
> that Daphne does what it says, and any vague or wrong behavior are now well
> documented. This means that the spec has real meaning, and an alternative
> implementation should be possible without looking at Dahpne's source code.
>
> Working with Andrew was good - I'm grateful for the opportunity, achieved
> my goals, and he always replied promptly to my messages, even though he has
> a lot of other things on his plate. Thanks again, Andrew!
>
> If you have any questions regarding my work (which you should find in pull
> requests by 'maikhoepfel') or the project itself, please feel free to ask
> here or reach out personally.
>
> Cheers,
>
> Maik
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/3f8a4b27-6a9a-4173-a251-
> f35bc14a648f%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM0cy7yXRmSuN%3DS3-UCQ3hFypcMnzCkDoD6A9LJR8hWMuw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Project report: Test infrastructure and property-based tests for Channels

[m]

Hi everyone,

I did some work on Channels' tests and testing infrastructure as part of 
the Mozilla grant. This is my report. 

Channels consists of five projects under the Django organization. First of 
all, I agreed with Andrew on versions of Django, Python and Twisted that we 
want to support across all of them. Then I updated Travis and tox 
configurations in all projects to be consistent with each other, and to 
test all agreed-upon version combinations. We also settled on specifying 
test dependencies in an extra in setup.py. 

This was important but unexciting work. The only useful takeaway might be 
that it confirmed my view that using Travis to run tox is an anti-pattern. 
It tends to slow down the test runs. I also have seen cases where tox 
didn't update the test environment in response to updated requirements - 
and those cases are hard to debug. Instead, I advocate on simplifying the 
test infrastructure so that there's almost no duplication between Travis 
and tox config - Makefiles or scripts can help with that in more complex 
cases.

The main idea of my proposal was to introduce tests that 
are-close-as-possible translations of the ASGI spec into code. I think this 
wouldn't have yielded much benefit if it weren't for using hypothesis to 
generate test data, aka "property based testing". I was previously 
impressed by the potential of it, and really saw it come to fruition here. 
Hypothesis was annoyingly good at finding edge cases. Think of distinction 
between passing in "None" for headers vs [], for using Unicode in query 
strings , or for making it 
clear that certain characters definitely aren't allowed e.g. in header 
names because they break the underlying HTTP protocol. Last but not least, 
I also introduced a "kitchen sink" (e.g. here 
)
 
test for each scenario which basically lets hypothesis try all weird 
combinations by itself, which has highlighted at least one more issue.
In conclusion, hypothesis let me find corner cases that I wouldn't have 
thought of or would've been too lazy to try, and neatly separated test data 
generation from the actual tests. I vow to use it more in the future :)

While there's always more to test, I think we now have high confidence that 
Daphne does what it says, and any vague or wrong behavior are now well 
documented. This means that the spec has real meaning, and an alternative 
implementation should be possible without looking at Dahpne's source code. 

Working with Andrew was good - I'm grateful for the opportunity, achieved 
my goals, and he always replied promptly to my messages, even though he has 
a lot of other things on his plate. Thanks again, Andrew!

If you have any questions regarding my work (which you should find in pull 
requests by 'maikhoepfel') or the project itself, please feel free to ask 
here or reach out personally.

Cheers,

Maik

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3f8a4b27-6a9a-4173-a251-f35bc14a648f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django and CSP strict-dynamic

[Adam Johnson]

Hi Ran

Given that it's still a W3C draft I am not sure it should be added to
Django core yet. I don't know of any official policy around what Django
implements versus the W3C spec levels though.

However it does seem that without a centralized solution for CSP nonces,
they won't be usable with third party apps as you found with
django-debug-toolbar.

For web security in Django, I'd like to help get strict-dynamic available
in a third party app, django-csp or otherwise. So I'm wondering, is your
code public? And do you think it is worth trying to merge into django-csp,
or would it be better as a standalone app? (I'm not really familiar with
what django-csp does/does not do)

Adam

On 13 May 2017 at 11:16,  wrote:

> Hey,
>
> I'd like to discuss how Django might encourage and ease the use of CSP in
> the frontend. As Django is used more and more to drive complex web
> applications, I think this is of interest. (I tried to keep this short, but
> failed...).
>
> # CSP background
>
> CSP (Content Security Policy) is an elaborate mechanism which allows to
> attach a policy to a webpage, restricting the things it can do and the
> origins it can interact with. It is mainly meant to aid in the prevention
> of various code injection attacks. It is usually sent in the
> `Content-Security-Policy` header in the response to the HTML request.
> Please see here for more details: https://developer.mozilla.org/
> en-US/docs/Web/HTTP/CSP
>
> Currently, in order to have a secure policy you must write big
> white-lists, which does not lend itself well to automation by a framework.
> For example, here is the policy GitHub sends, which is relatively tame as
> far as these go:
>
> default-src 'none';
> base-uri 'self';
> child-src render.githubusercontent.com;
> connect-src 'self' uploads.github.com status.github.com
> collector.githubapp.com api.github.com www.google-analytics.com
> github-cloud.s3.amazonaws.com github-production-repository-f
> ile-5c1aeb.s3.amazonaws.com github-production-user-asset-7
> 9cafe.s3.amazonaws.com wss://live.github.com;
> font-src assets-cdn.github.com;
> form-action 'self' github.com gist.github.com; frame-ancestors 'none';
> img-src 'self' data: assets-cdn.github.com identicons.github.com
> collector.githubapp.com github-cloud.s3.amazonaws.com *.
> githubusercontent.com;
> media-src 'none';
> script-src assets-cdn.github.com;
> style-src 'unsafe-inline' assets-cdn.github.com
>
> The django-csp project (https://github.com/mozilla/django-csp) helps with
> generating these in Django. However, maintaining, deploying and monitoring
> it is quite a major effort, depending on the complexity of the site.
>
> # The strict-dynamic directive
>
> Because of this, the upcoming version of the CSP standard (Level 3,
> https://www.w3.org/TR/CSP/) includes a new `script-src` directive called
> `strict-dynamic`. It has its own website:
> https://csp.withgoogle.com/docs/index.html
>
> The idea here is that you send a policy like this:
>
> script-src 'nonce-{random}' 'strict-dynamic';
>
> where `{random}` is some token that is randomly generated in each request,
> e.g. `hM6RAC4rgxYFk01gbbfcKw==`. Next, each script included in the page
> must be decorated with a `nonce` attribute, using the nonce sent in the
> header:
>
>  src="/path/to/script.js">
>
> foo()
>
> What the `strict-dynamic` means, is that any script trusted with the nonce
> is transitively trusted to load its own scripts dynamically. Thus, most of
> the XSS protection remains, however the maintenance burden drops
> significantly, and it becomes much more amenable to automation in a
> framework.
>
> # Browser support and backward compatibility
>
> `strict-dynamic` is already supported in recent versions of Firefox and
> Chrome, and is "under consideration" by Edge:
> https://blogs.windows.com/msedgedev/2017/01/10/edge-csp-2/
>
> The CSP Level 3 spec is still a draft, but from what I could gather they
> are aiming to finalize it in "Q3 2017":
> https://rawgit.com/w3c/webappsec/master/admin/webappsec-charter-2017.html
>
> What about browsers which does not support `strict-dynamic`, but only
> support older CSP versions? With the policy I gave above, everything will
> be blocked (if nonces are not supported), or transitive scripts will be
> blocked (if nonces are supported). To workaround this, `strict-dynamic`
> has the behavior of disabling/ignoring other directives if it is
> understood. So we can send a policy like this instead:
>
> script-src 'nonce-{random}' 'unsafe-inline' 'unsafe-eval'
> 'strict-dynamic' https: http:;
>
> In browsers that support CSP3, it will be interpreted as
>
> script-src 'nonce-{random}' 'strict-dynamic' 'unsafe-eval'
>
> In browsers that support CSP2:
>
> script-src 'nonce-{random}' 'unsafe-eval' https: http:;
>
> In browsers that support CSP1:
>
> script-src 'unsafe-inline' 'unsafe-eval' https: http:;
>
> In all cases it should work out.
>

Re: Review of DEP 201 - simplified routing syntax

[Sjoerd Job Postmus]

Thinking about it some more: that is not the solution.

Also, there are probably a couple of corner cases to consider (still: the 
same as for normal regular expressions):

* What if the "keys" (named parameters) of a translated path differs from 
that of the original path?
* What if the converters differ?

For now I think the best way forward is to assume the above corner cases do 
not happen, or otherwise fall out-of-scope.

On Saturday, May 13, 2017 at 2:32:00 PM UTC+2, Sjoerd Job Postmus wrote:
>
>
>
> On Saturday, May 13, 2017 at 1:35:37 PM UTC+2, Marten Kenbeek wrote:
>>
>> The regex in `url()` can be translated 
>> 
>>  
>> using `ugettext_lazy()`, in which case the lazy translation happens when 
>> `resolve()` or `reverse()` is called. It seems the current `path()` 
>> implementation would force evaluation when the URLs are first loaded, so 
>> `resolve()` and `reverse()` would use a fixed language (whichever was 
>> active when they were loaded) rather than allowing lazy translations.
>>
>> I'm not sure how often this feature is used, but it seems like something 
>> `path()` should support out of the box. 
>>
>
> I agree. It's an oversight which I think is solvable. The 
> `LocaleRegexProvider` class already handles the caching.
>
> I will check later, but I think all that's needed is to decorate the 
> `path` function with `keep_lazy_str`. Does that seem correct to you?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/37d03083-ce35-4b9d-a881-0f9c45e90db6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.