Re: Tracking/logging bruteforcing, especially on admin accounts?

[Josh Smeaton]

I understand the reasoning for "use the cache", but not every site has 
caching enabled, especially lots of smaller sites. A separate table could 
be used for tracking attempts, and cleared out per user on successful login 
attempt/ip address. This table would not need to be huge if designed with 
performance in mind. Caching could be layered on (check cache for bans but 
not for maintaining counts) or we could have a cache backend in addition to 
a database backend. I'm aware that django has a database cache backend, but 
any database solution in this space should be designed with the database in 
mind and not just essentially blob storage.

For what it's worth, I've used django-axes 
(https://pypi.python.org/pypi/django-axes) with some success in the past 
and will probably use it again for my current project. I don't think the 
username based protection is adequate though:

AXES_LOCK_OUT_BY_COMBINATION_USER_AND_IP: If True prevents to login from IP 
> under particular user if attempts limit exceed, otherwise lock out based on 
> IP.
>

Distributed attack on a particular user would not be caught in this 
configuration. 

Cheers

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/dcb1395e-c93a-410e-839d-3c06f39d8d40%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Organizing the built-in system check framework's hint messages

[Quentin Fulsher]

Here is a super quick proof of concept that I put together. I just branched 
my fork of django and added a little to it. Here is the comparing changes 
page[1].

Quick summary of changes: I created a dictionary that would contain the 
(id: message) pairs. I also modified the CheckMessage.__init__ method so 
that it will attempt to find a hint message for the id that was passed to 
it. If no key was found then it continues as normal. This allows me to 
comment out the hint parameter when it is created but still be able to pass 
when it is run through its normal tests.

[1] https://github.com/django/django/compare/master...inondle:hints

On Thursday, May 19, 2016 at 10:55:15 AM UTC-7, Tim Graham wrote:
>
> The security checks errors are defined as module level constants to avoid 
> some redundancy since these can be imported in tests: 
> https://github.com/django/django/blob/0eac5535f7afd2295b1db978dffb97d79030807e/django/core/checks/security/base.py
> .
>
> If you feel your approach would be an improvement, maybe you could put 
> together a quick proof of concept so it's a bit easier to understand?
>
> On Wednesday, May 18, 2016 at 11:50:47 PM UTC-4, Quentin Fulsher wrote:
>>
>> Hi all, 
>>
>> Recently I was trying to fix a bug that required me to alter the hint 
>> message of the `fields.E306`[1] system check error[2]. The bug in question 
>> added a condition to a check which if failed caused the instantiation of a 
>> `django.core.checks.Error` object. In order to change the `Error`'s hint 
>> properly I would have to change the hint in 3 different places. Once where 
>> the `Error` was instantiated, in the documentation, and in it's tester. 
>> This was a rather tedious task and if done improperly can lead to differing 
>> error messages. Rather than just passing the hint in as an argument to the 
>> constructor I think it would be better if the Error object had the option 
>> to lookup it's error messages rather than just having them given. 
>>
>> It wouldn't take much effort to create a dictionary of `id: message` 
>> pairs which `Error` objects could use to lookup their messages/hints. In 
>> the event that an `Error` is instantiated with no hint/message given it 
>> could perform a quick check to see if it is logged in the dictionary. This 
>> would not add any significant overhead to the process and eliminate some 
>> possibility of human error. 
>>
>> This situation becomes even more tedious if you have more than one place 
>> where the error is instantiated or tested. This could also help with 
>> documentation because all of the builtin error messages would be stored in 
>> a central location.
>>
>> In any case, let me know what you guys think. I know its kind of a 
>> non-issue as the current system works fine. I just thought there was some 
>> room for improvement. Thanks!
>>
>> [1] https://docs.djangoproject.com/en/1.9/ref/checks/#fields
>> [2] https://docs.djangoproject.com/en/1.9/topics/checks/
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/307a5a2f-5494-4a4f-8b55-d1ddbec3d94f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: need not_equal and not_in query lookup types for a natural use case that can't easily be implemented with existing queries

[Yoong Kang Lim]

BTW, if I understand your problem correctly you might be able to do this 
using a subquery:

non_special_items = Item.objects.exclude(id__in=special_items)

special_only_bundles = Bundle.objects.exclude(items__in=non_special_items)



On Thursday, May 19, 2016 at 11:19:42 PM UTC+10, David Xiao wrote:
>
> Hi folks,
>
> Django is missing a not_equal lookup type for database queries.  I know 
> this has been requested before and that the usual response is to use exclude 
> or ~Q.  While that works for a lot of use cases, I have a natural use 
> case that I haven't seen discussed and that seems to escape easy 
> implementation with the existing queryset implementation.  (I already 
> posted to the django-users group and nobody was able to give a satisfactory 
> answer.  If there is an easy answer, one using a single query, please let 
> me know.)
>
> Example: suppose we have the following models:
>
> class Bundle(Model)
>   items = ManyToManyField("Item")
>
> class Item(Model)
>   pass
>
> (So one item can belong to many Bundles and one Bundle can have many 
> Items.)
>
> The query: I want to select all Bundles whose items are contained in an 
> array special_items.  (Think of special_items as being relatively small 
> and the set of all possible items as large.)  
>
> Example: Suppose the database contains Items item1, item2, item3 (possibly 
> among many other items) and that special_items = [item1, item2].  Suppose 
> also the following Bundles exist in the database:
> 1. Bundle containing item1
> 2. Bundle containing item2
> 3. Bundle containing item1, item2
> 4. Bundle containing item2, item3
> 5. Bundle containing item1, item2, item3
>
> So the query should return bundles 1, 2, 3 but exclude bundles 4, 5 
> because they contain item3, which is not a special item.
>
> Queries that don't work:
> - Bundle.exclude(items__in=special_items) which returns no bundles
> - Bundle.filter(items__in=special_items) which returns bundles 1,2,3,4,5
> - Bundle.exclude(~Q(items__in=special_items)) which returns 1,2,3,4,5
>
> One work-around that does work is 
> Bundle.exclude(items__in=not_special_items) where not_special_items contains 
> all possible items that are not in special_items.  But clearly this is 
> impractical/unusable if the set of possible items is infinite or very large.
>
> On the other hand, if we had a not_equal (or even better, a not_in) 
> operator, we could do the query as follows:
> Bundle.exclude(items__not_in=special_items)
> Bundle.filter(~Q(items__not_equal=special_items[0]) & 
> ~Q(items__not_equal=special_items[1]) & ...)
> (The latter should be OK because in typical use cases special_items is a 
> small set.)
>
> Note: I did try to implement this as a custom lookup, but I guess because 
> it's spanning a relationship you can't use the regular custom lookup API. 
> If there is an easy way to add this as a custom lookup I'd appreciate some 
> guidance.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/64749196-f6d6-4cab-94ed-fb19e25638da%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Unicode normalization for username field

[charettes]

Hi David,

I agree with your reasoning but I think you're missing an important detail 
about
unicode username support: they have been mistakenly enabled on Python 3 
since
Django added support for it (1.5-1.6).

If we were to disallow non-ASCII characters silently from Django 1.10 
Python 3
developers would be left with the same problem you mentioned about existing
users with usernames containing unicode characters.

Cheers,
Simon

Le jeudi 19 mai 2016 14:48:39 UTC-4, David Tan a écrit :
>
> - I'm afraid this change may result in boilerplate as most custom user 
>> models will revert to Django's historical (and in my opinion sensible) 
>> username validation rules. 
>>
>
> That's a tough question to estimate. This might be true for most English 
> monolingual web sites, but not necessarily for the majority of Django 
> sites. Hopefully we'll get some more user inputs in this thread.
>  
>
> Hi, just wanted to give my input on this point, I agree with Aymeric 
> Augustin here and my vote is to keep usernames as ASCII by default.
>
> I created a Django ticket 
>  for this, I will 
> copy my reasoning here:
>
> A Django user who is trying to save time and get a product out the door 
> isn't going to focus on finer details such as Unicode usernames, and will 
> be in for a shock when he finds out a bunch of his users have registered 
> themselves with Egyptian hieroglyphics. He may be very frustrated, 
> eventually figuring out that he must subclass the User model and 
> setusername_validator 
> = ASCIIUsernameValidator() to get the functionality he expected. And what 
> is he to do with the existing Unicode users, delete all their accounts?
>
> Whereas a technologically forward user might be friendlier towards Unicode 
> usernames, and would be well-informed on these capabilities within Django. 
> Furthermore, the technologically forward user will be more likely to 
> already have a custom user model, and won't find it cumbersome to 
> explicitly enable Unicode usernames. Enabling Unicode usernames isn't 
> destructive like disabling it would be (no need to figure out what to do 
> with the existing users offending the validation), so users can simply 
> start using it immediately.
>
> On Sunday, April 24, 2016 at 2:58:55 PM UTC-4, Claude Paroz wrote:
>>
>> Hi Aymeric,
>>
>> Le samedi 23 avril 2016 14:33:56 UTC+2, Aymeric Augustin a écrit :
>>>
>>> > https://github.com/django/django/pull/6494 
>>>
>>> This patch looks pretty good. I have a few questions, not necessarily 
>>> because I disagree with your proposal, but to make sure we have considered 
>>> alternatives. Actually I don't think there's exactly one correct solution 
>>> here; it's more a matter of tradeoffs. 
>>>
>>> You added a username_validator attribute instead of documenting how to 
>>> override the whole username field. Can you elaborate on this decision? I 
>>> simplifies the use case targeted by the patch by introducing a one-off API. 
>>> As a matter of principle I'm a bit skeptical of such special cases. But I 
>>> understand the convenience.
>>>
>>
>> My preoccupation here was not to force users to create a custom user 
>> model just to change the username validation, especially as the migration 
>> system doesn't seem to support yet upgrading from the standard auth User to 
>> a custom user. I thought that creating a proxy custom user is easier 
>> migration-wise, as no new table is required. But I may be wrong.
>>  
>>
>>> Normalization happens at the form layer. I'm wondering whether it would 
>>> be safer to do it at the model layer. That would extend the security 
>>> hardening to cases where users aren't created with a form — for example if 
>>> they're created through an API or programmatically. 
>>>
>>
>> Normalization happens both at the form layer and at the model layer in 
>> _create_user. You may have missed the _create_user change.
>>  
>>
>>> I would keep ASCII usernames as the default because: 
>>>
>>> - this has always been the intent; 
>>>
>>
>> Until now! Things are evolving, we see that for example with 
>> internationalized domain names. I think that most if not all technical 
>> reasons requiring pure ASCII usernames have vanished nowadays.
>>  
>>
>>> - allowing non ASCII usernames may result in interoperability problems 
>>> with other software e.g. if a Django project is used as SSO server; 
>>>
>>
>> These are still not the majority of Django use cases. And even then, I 
>> think that LDAPv3 for example should support unicode in attributes. Those 
>> project could still configure the ASCIIUsernameValidator if desired.
>>  
>>
>>> - these interoperability issues might escalate into security 
>>> vulnerabilities — there isn't a straightforward connection but (1) non 
>>> ASCII data can be used for breaking out of parsing routines (2) I'm 
>>> paranoid with anything that manipulates authentication credentials; 
>>>
>>
>> Sure, the more characters, the more attack 

Re: Unicode normalization for username field

[David Tan]

>
> - I'm afraid this change may result in boilerplate as most custom user 
> models will revert to Django's historical (and in my opinion sensible) 
> username validation rules. 
>

That's a tough question to estimate. This might be true for most English 
monolingual web sites, but not necessarily for the majority of Django 
sites. Hopefully we'll get some more user inputs in this thread.
 

Hi, just wanted to give my input on this point, I agree with Aymeric 
Augustin here and my vote is to keep usernames as ASCII by default.

I created a Django ticket 
 for this, I will copy 
my reasoning here:

A Django user who is trying to save time and get a product out the door 
isn't going to focus on finer details such as Unicode usernames, and will 
be in for a shock when he finds out a bunch of his users have registered 
themselves with Egyptian hieroglyphics. He may be very frustrated, 
eventually figuring out that he must subclass the User model and 
setusername_validator 
= ASCIIUsernameValidator() to get the functionality he expected. And what 
is he to do with the existing Unicode users, delete all their accounts?

Whereas a technologically forward user might be friendlier towards Unicode 
usernames, and would be well-informed on these capabilities within Django. 
Furthermore, the technologically forward user will be more likely to 
already have a custom user model, and won't find it cumbersome to 
explicitly enable Unicode usernames. Enabling Unicode usernames isn't 
destructive like disabling it would be (no need to figure out what to do 
with the existing users offending the validation), so users can simply 
start using it immediately.

On Sunday, April 24, 2016 at 2:58:55 PM UTC-4, Claude Paroz wrote:
>
> Hi Aymeric,
>
> Le samedi 23 avril 2016 14:33:56 UTC+2, Aymeric Augustin a écrit :
>>
>> > https://github.com/django/django/pull/6494 
>>
>> This patch looks pretty good. I have a few questions, not necessarily 
>> because I disagree with your proposal, but to make sure we have considered 
>> alternatives. Actually I don't think there's exactly one correct solution 
>> here; it's more a matter of tradeoffs. 
>>
>> You added a username_validator attribute instead of documenting how to 
>> override the whole username field. Can you elaborate on this decision? I 
>> simplifies the use case targeted by the patch by introducing a one-off API. 
>> As a matter of principle I'm a bit skeptical of such special cases. But I 
>> understand the convenience.
>>
>
> My preoccupation here was not to force users to create a custom user model 
> just to change the username validation, especially as the migration system 
> doesn't seem to support yet upgrading from the standard auth User to a 
> custom user. I thought that creating a proxy custom user is easier 
> migration-wise, as no new table is required. But I may be wrong.
>  
>
>> Normalization happens at the form layer. I'm wondering whether it would 
>> be safer to do it at the model layer. That would extend the security 
>> hardening to cases where users aren't created with a form — for example if 
>> they're created through an API or programmatically. 
>>
>
> Normalization happens both at the form layer and at the model layer in 
> _create_user. You may have missed the _create_user change.
>  
>
>> I would keep ASCII usernames as the default because: 
>>
>> - this has always been the intent; 
>>
>
> Until now! Things are evolving, we see that for example with 
> internationalized domain names. I think that most if not all technical 
> reasons requiring pure ASCII usernames have vanished nowadays.
>  
>
>> - allowing non ASCII usernames may result in interoperability problems 
>> with other software e.g. if a Django project is used as SSO server; 
>>
>
> These are still not the majority of Django use cases. And even then, I 
> think that LDAPv3 for example should support unicode in attributes. Those 
> project could still configure the ASCIIUsernameValidator if desired.
>  
>
>> - these interoperability issues might escalate into security 
>> vulnerabilities — there isn't a straightforward connection but (1) non 
>> ASCII data can be used for breaking out of parsing routines (2) I'm 
>> paranoid with anything that manipulates authentication credentials; 
>>
>
> Sure, the more characters, the more attack surface. As you said before, 
> it's a tradeoff. My thinking is that sooner or later, we'll have to cope 
> with unicode in usernames. So let's do our most to not open security holes, 
> based on some passed issues (BTW I think you forgot you references).
>  
>
>> - I'm afraid this change may result in boilerplate as most custom user 
>> models will revert to Django's historical (and in my opinion sensible) 
>> username validation rules. 
>>
>
> That's a tough question to estimate. This might be true for most English 
> monolingual web sites, but not necessarily for the majority of Django 
> sites. Hopefully 

Re: Organizing the built-in system check framework's hint messages

[Tim Graham]

The security checks errors are defined as module level constants to avoid 
some redundancy since these can be imported in tests: 
https://github.com/django/django/blob/0eac5535f7afd2295b1db978dffb97d79030807e/django/core/checks/security/base.py.

If you feel your approach would be an improvement, maybe you could put 
together a quick proof of concept so it's a bit easier to understand?

On Wednesday, May 18, 2016 at 11:50:47 PM UTC-4, Quentin Fulsher wrote:
>
> Hi all, 
>
> Recently I was trying to fix a bug that required me to alter the hint 
> message of the `fields.E306`[1] system check error[2]. The bug in question 
> added a condition to a check which if failed caused the instantiation of a 
> `django.core.checks.Error` object. In order to change the `Error`'s hint 
> properly I would have to change the hint in 3 different places. Once where 
> the `Error` was instantiated, in the documentation, and in it's tester. 
> This was a rather tedious task and if done improperly can lead to differing 
> error messages. Rather than just passing the hint in as an argument to the 
> constructor I think it would be better if the Error object had the option 
> to lookup it's error messages rather than just having them given. 
>
> It wouldn't take much effort to create a dictionary of `id: message` pairs 
> which `Error` objects could use to lookup their messages/hints. In the 
> event that an `Error` is instantiated with no hint/message given it could 
> perform a quick check to see if it is logged in the dictionary. This would 
> not add any significant overhead to the process and eliminate some 
> possibility of human error. 
>
> This situation becomes even more tedious if you have more than one place 
> where the error is instantiated or tested. This could also help with 
> documentation because all of the builtin error messages would be stored in 
> a central location.
>
> In any case, let me know what you guys think. I know its kind of a 
> non-issue as the current system works fine. I just thought there was some 
> room for improvement. Thanks!
>
> [1] https://docs.djangoproject.com/en/1.9/ref/checks/#fields
> [2] https://docs.djangoproject.com/en/1.9/topics/checks/
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/2b2d3640-08c4-4218-9e9b-0a7e89caf52d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: need not_equal and not_in query lookup types for a natural use case that can't easily be implemented with existing queries

[Yoong Kang Lim]

> If there is an easy answer, one using a single query, please let me know.)

Can you post what your desired SQL would look like?

IIRC, foo.exclude(id__in=[bar, baz]) already adds "NOT IN" to the SQL query 
(but I may be wrong), i.e "SELECT foo WHERE id NOT IN (bar, baz)"

So what do you propose foo.exclude(id__not_in=[bar, baz]) would look like 
as a single SQL query? 


On Thursday, May 19, 2016 at 11:19:42 PM UTC+10, David Xiao wrote:
>
> Hi folks,
>
> Django is missing a not_equal lookup type for database queries.  I know 
> this has been requested before and that the usual response is to use exclude 
> or ~Q.  While that works for a lot of use cases, I have a natural use 
> case that I haven't seen discussed and that seems to escape easy 
> implementation with the existing queryset implementation.  (I already 
> posted to the django-users group and nobody was able to give a satisfactory 
> answer.  If there is an easy answer, one using a single query, please let 
> me know.)
>
> Example: suppose we have the following models:
>
> class Bundle(Model)
>   items = ManyToManyField("Item")
>
> class Item(Model)
>   pass
>
> (So one item can belong to many Bundles and one Bundle can have many 
> Items.)
>
> The query: I want to select all Bundles whose items are contained in an 
> array special_items.  (Think of special_items as being relatively small 
> and the set of all possible items as large.)  
>
> Example: Suppose the database contains Items item1, item2, item3 (possibly 
> among many other items) and that special_items = [item1, item2].  Suppose 
> also the following Bundles exist in the database:
> 1. Bundle containing item1
> 2. Bundle containing item2
> 3. Bundle containing item1, item2
> 4. Bundle containing item2, item3
> 5. Bundle containing item1, item2, item3
>
> So the query should return bundles 1, 2, 3 but exclude bundles 4, 5 
> because they contain item3, which is not a special item.
>
> Queries that don't work:
> - Bundle.exclude(items__in=special_items) which returns no bundles
> - Bundle.filter(items__in=special_items) which returns bundles 1,2,3,4,5
> - Bundle.exclude(~Q(items__in=special_items)) which returns 1,2,3,4,5
>
> One work-around that does work is 
> Bundle.exclude(items__in=not_special_items) where not_special_items contains 
> all possible items that are not in special_items.  But clearly this is 
> impractical/unusable if the set of possible items is infinite or very large.
>
> On the other hand, if we had a not_equal (or even better, a not_in) 
> operator, we could do the query as follows:
> Bundle.exclude(items__not_in=special_items)
> Bundle.filter(~Q(items__not_equal=special_items[0]) & 
> ~Q(items__not_equal=special_items[1]) & ...)
> (The latter should be OK because in typical use cases special_items is a 
> small set.)
>
> Note: I did try to implement this as a custom lookup, but I guess because 
> it's spanning a relationship you can't use the regular custom lookup API. 
> If there is an easy way to add this as a custom lookup I'd appreciate some 
> guidance.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/abca5d8c-1ec9-4267-9c09-d20e51bbc750%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Template Tag `url` returns empty string, does not raise NoReverseMatch

[Florian Apolloner]

On Thursday, May 19, 2016 at 2:41:43 PM UTC+2, guettli wrote:
>
> Do you use the following use case provided in the docs?
>

Yes, the admin makes (or at least made) also use of this.
 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3e3748f8-575c-44e7-a0c9-0ac36e18bf70%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

need not_equal and not_in query lookup types for a natural use case that can't easily be implemented with existing queries

[David Xiao]

Hi folks,

Django is missing a not_equal lookup type for database queries.  I know 
this has been requested before and that the usual response is to use exclude 
or ~Q.  While that works for a lot of use cases, I have a natural use case 
that I haven't seen discussed and that seems to escape easy implementation 
with the existing queryset implementation.  (I already posted to the 
django-users group and nobody was able to give a satisfactory answer.  If 
there is an easy answer, one using a single query, please let me know.)

Example: suppose we have the following models:

class Bundle(Model)
  items = ManyToManyField("Item")

class Item(Model)
  pass

(So one item can belong to many Bundles and one Bundle can have many Items.)

The query: I want to select all Bundles whose items are contained in an 
array special_items.  (Think of special_items as being relatively small and 
the set of all possible items as large.)  

Example: Suppose the database contains Items item1, item2, item3 (possibly 
among many other items) and that special_items = [item1, item2].  Suppose 
also the following Bundles exist in the database:
1. Bundle containing item1
2. Bundle containing item2
3. Bundle containing item1, item2
4. Bundle containing item2, item3
5. Bundle containing item1, item2, item3

So the query should return bundles 1, 2, 3 but exclude bundles 4, 5 because 
they contain item3, which is not a special item.

Queries that don't work:
- Bundle.exclude(items__in=special_items) which returns no bundles
- Bundle.filter(items__in=special_items) which returns bundles 1,2,3,4,5
- Bundle.exclude(~Q(items__in=special_items)) which returns 1,2,3,4,5

One work-around that does work is 
Bundle.exclude(items__in=not_special_items) where not_special_items contains 
all possible items that are not in special_items.  But clearly this is 
impractical/unusable if the set of possible items is infinite or very large.

On the other hand, if we had a not_equal (or even better, a not_in) 
operator, we could do the query as follows:
Bundle.exclude(items__not_in=special_items)
Bundle.filter(~Q(items__not_equal=special_items[0]) & 
~Q(items__not_equal=special_items[1]) & ...)
(The latter should be OK because in typical use cases special_items is a 
small set.)

Note: I did try to implement this as a custom lookup, but I guess because 
it's spanning a relationship you can't use the regular custom lookup API. 
If there is an easy way to add this as a custom lookup I'd appreciate some 
guidance.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/24bfac80-c102-404a-9b91-c35059545742%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Template Tag `url` returns empty string, does not raise NoReverseMatch

[guettli]

If you use this syntax, then no NoReverseMatch exception gets raised:

{% url 'some-url-name' arg arg2 as the_url %}

This is documented here: 
https://docs.djangoproject.com/en/1.9/ref/templates/builtins/#url

> This {% url ... as var %} syntax will not cause an error if the view is 
> missing.
> In practice you’ll use this to link to views that are optional ...

This caused an error on a production system which was not detected in CI.

Is the current implementation really the way you like it?

Do you use the following use case provided in the docs?

> % url 'some-url-name' as the_url %}> {% if the_url %}
>  Link to optional stuff> {% endif %}

Regards,
  Thomas Güttler

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/74b6df31-916e-4924-acc4-56db553da4db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Tracking/logging bruteforcing, especially on admin accounts?

[Florian Apolloner]

On Thursday, May 19, 2016 at 1:57:50 PM UTC+2, Aymeric Augustin wrote:
>
> Django’s “last_login” field in the user model is a common cause of 
> performance issues in large sites. (I believe it’s required to secure 
> password resets so we can’t remove it.) Let’s avoid adding more fields with 
> similar behavior. (The PostgreSQL experts subscribed to this mailing list 
> should be able to confirm that part.)
>

Not a postgres expert -- but an "often changing" field in a row certainly 
bloats the table. It would be nice if it were in an extra table, so 
rewriting the row does not include all the data like username and email 
(though I do realize this its probably not an option).

As for any changes to the admin itself, I am somewhat -0 to -1 on them, 
unless it can be clearly shown that we cannot do this via decorators etc in 
a generic way (there are probably plenty of views out there which could 
benefit from ratelimiting).

As for the rest of the proposed changes (see the gist):
 * Let's encrypt -- this should not be done by Django
 * 2FA: Yes, I'd like to see at least U2F and TOTP support ootb.
 
When it comes to tracking/rate limiting: No new fields for the models 
please, this should all be in cache (this is absolutely information you do 
not want to persist and also changes rapidly during attacks…)

Cheers,
Florian

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b0de12b2-71a8-4beb-b6a3-a8670eeef939%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Tracking/logging bruteforcing, especially on admin accounts?

[Aymeric Augustin]

Hello Vaibhav,

> On 19 May 2016, at 08:59, Vaibhav Mallya  wrote:
> 
> In my view, #2 would be the best starting point - there are going to be a 
> relatively small number of admin accounts on the average Django site. Ergo, 
> focused brute-forcing or spearfishing seems to be a greater threat than 
> getting into an admin account via a lot of scanning.

This makes sense for django.contrib.admin. However I suggest to consider this 
feature as an enhancement of django.contrib.auth and consider a more general 
use case. That seems significantly more useful — assuming the generalization 
doesn’t make the problem intractable.

You should look at prior art such as django-ratelimit-backend.

> I am proposing a solution broken into two parts - tracking and enforcing.
> 
> Tracking - End goal would be logging SuspiciousOperation if appropriate 
> thresholds were crossed. We’d need to store server-side state. I don’t 
> believe we don’t have a clean heap data structure across all DBs that the 
> Django ORM supports, but we could, say, keep two additional columns on each 
> user object: last_login_attempt_window_start, and 
> num_login_attepts_on_window_start, and checking / updating both on any / all 
> login attempts. Alternatively, simply serializing a Python heap-list for each 
> user may work.
> 
> Or we can simply leverage the cache backend to store state.

Please use the cache.

Django’s “last_login” field in the user model is a common cause of performance 
issues in large sites. (I believe it’s required to secure password resets so we 
can’t remove it.) Let’s avoid adding more fields with similar behavior. (The 
PostgreSQL experts subscribed to this mailing list should be able to confirm 
that part.)

> Enforcing - Rejecting login attempts [on any basis] is probably not a good 
> idea for a default - we can’t guarantee we don’t introduce some other 
> DoS-style attack vector. But there are some NIST/etc guidelines around, say, 
> forcing pauses between login attempts, exponential backoff, forcing 
> email-distributed tokens to be used, etc.
> 
> We’re already storing custom auth/session information for the Django user 
> model, so storing state/migrations/etc somewhere wouldn’t be too much of a 
> departure.

This is an interesting part of the problem. If it has been explored by current 
third-party solutions, then I’m not aware of it. Django should provide a 
reasonable default policy. Besides it would be nice to support selecting 
another policy. Such configurability would be consistent with how Django 
handles situations where various behaviors are acceptable.

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/BCA82CDC-FACF-41C7-985F-53E4D878B9CF%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: Tracking/logging bruteforcing, especially on admin accounts?

[Cristiano Coelho]

IP based throttling like django-rest-framework would be ideal! I know there 
are some 3rd party libraries that tries to add ip based throttling to 
django although not as cool as drf ones.

El jueves, 19 de mayo de 2016, 8:11:27 (UTC-3), Vaibhav Mallya escribió:
>
> Hi everyone,
>
> I've been having a great chat with @jacobian here about potential security 
> improvements to the Django admin UI: 
> https://gist.github.com/mallyvai/bcb0bb827d6d53212879dff23cf15d03
>
> The admin UI is core to Django's value-prop, and it seems undersecured by 
> modern standards. I wanted to raise the possibility on this list of 
> guarding against brute-force login attempts on admin user accounts.
>
> You could imagine tracking / throttle in two main ways.
>
>
> 1. By originating IP
> 2. By number of login attempts on a user account
>
>
> In my view, #2 would be the best starting point - there are going to be a 
> relatively small number of admin accounts on the average Django site. Ergo, 
> focused brute-forcing or spearfishing seems to be a greater threat than 
> getting into an admin account via a lot of scanning.
>
> I am proposing a solution broken into two parts - tracking and enforcing.
>
> Tracking - End goal would be logging SuspiciousOperation if appropriate 
> thresholds were crossed. We’d need to store server-side state. I don’t 
> believe we don’t have a clean heap data structure across all DBs that the 
> Django ORM supports, but we could, say, keep two additional columns on each 
> user object: last_login_attempt_window_start, and 
> num_login_attepts_on_window_start, and checking / updating both on any / 
> all login attempts. Alternatively, simply serializing a Python heap-list 
> for each user may work.
>
> Or we can simply leverage the cache backend to store state.
>
> Enforcing - Rejecting login attempts [on any basis] is probably not a good 
> idea for a default - we can’t guarantee we don’t introduce some other 
> DoS-style attack vector. But there are some NIST/etc guidelines around, 
> say, forcing pauses between login attempts, exponential backoff, forcing 
> email-distributed tokens to be used, etc.
>
> We’re already storing custom auth/session information for the Django user 
> model, so storing state/migrations/etc somewhere wouldn’t be too much of a 
> departure.
>
> Thanks!
> -Vaibhav
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f3619d7d-99e6-4061-9e5d-dff9e8f423cd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: New CharField attribute to handle CharField(null=True, blank=True, unique=True) in model forms

[Claude Paroz]

The design and the patch look good to me. Thanks!

Claude

Le jeudi 19 mai 2016 05:01:37 UTC+2, Jon Dufresne a écrit :
>
> Hi,
>
> Occasionally I'll need to define a CharField on a model that is unique but 
> also allow blank values. At the database level, this is easily handled by 
> storing NULL for the blank values. (Storing the empty string multiple times 
> will result in a DB unique constraint violation.) This use case has 
> surfaced several times as evident by the 9 year old ticket #4136 [1].
>
> I have created a POC solution to the ticket at PR 6624 [2]. The change 
> adds the attribute "empty_value" to forms.CharField. The value specified by 
> empty_value is used as the Python empty value for the field. This value 
> defaults to the empty string (current behavior) but could also be changed 
> to None. The change also modifies model forms to set empty_value to None 
> for CharField when null=True is set. The model forms change allows the use 
> of the admin to save blank, unique char values without unique constraint 
> violations.
>
> This choice to create the empty_value attribute API was based on the prior 
> art of the empty_value attribute of TypedChoiceField [3].
>
> In the PR Tim Graham requested I raise the new API on the developers list. 
> I'm looking for concerns or feedback regarding the new attribute or any 
> other issue with the PR.
>
> Thanks!
>
> Cheers,
> Jon
>
>
> [1] https://code.djangoproject.com/ticket/4136
> [2] https://github.com/django/django/pull/6624
> [3] 
> https://github.com/django/django/blob/218175b/django/forms/fields.py#L832-L856
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ddebcca8-31f3-464e-a993-e4bdd288860d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.