Re: contrib.sessions test oddities

[Justin Holmes]

We figured out the phenomenon that triggered this problem:

The djangobb middleware was saving items to the cache using integers
(user ids) as the key and True as the value.  Obviously this is an
abhorrent practice in its own right both for disambiguation and
security.

We have submitted a patch (which has since been committed) which
generates a (hopefully) helpful fail message instead of raising
AttributeError.

Additionally, this practice has explained the strange timing issue:
The key in the cache expired after 15 seconds.

However, at least two questions remain:

1) In response to Stephen's point: I understand that the cached_db
backend reads from localmem by default, by why isn't .set() being run
on the db in the test case?  .set() does in fact get run in the
regular use of this app, yet it is never tested by our test suite.

2) What is the canonical style practice for cache keys?  Where does it
need to be documented?  Is the use of a naked integer as a key in fact
contrary to our guidelines?  It certainly needs to be, right?  If so,
shall we raise a warning if in fact this happens?  I'd be concerned
that testing the incoming cache key might be unduly expensive for very
cache-heavy projects and that the test itself will punish those who do
follow the guidelines.


On Tue, Oct 4, 2011 at 9:11 PM, Stephen Burrows
 wrote:
> Hi,
>
> It sounds like you're looking at tests for
> django.contrib.sessions.backends.cached_db. If that is the case, then
> it's not unusual that you would end up calling the LocMem cache
> backend. The cached_db session backend does all of its reads from the
> cache, only falling back to the database sessions if the key is not
> found in the cache. So, unless you define a different cache backend in
> your settings.py file, the cached_db session backend will use the
> LocMem cache backend.
>
> That doesn't necessarily mean you haven't run into a problem, of
> course. But it may explain some of what you're experiencing.
>
> Best,
> Stephen Burrows
>
> On Oct 4, 9:22 pm, Justin Holmes  wrote:
>> I posted this message on -users today, but at PaulM's urging, I'm
>> bringing it over here out of concern that it may reflect a bug in the
>> contrib.sessions test suite.
>>
>> Here's the original message from -users (some -dev only notes follow) :
>>
>> I am having a problem with contrib.sessions.tests.test_valid_key (line
>> 159).  It's a mixin which in the failure case is mixed with
>> CacheDBSessionTests.
>>
>> The test raises:
>>
>> "AttributeError: 'bool' object has no attribute 'get'"
>>
>> The error is actually raised by session.save() on 164.
>>
>> Stepping inward reveals that sessions.backends.cached_db (line 18)
>> sets data thusly:
>>
>> data = cache.get(self.session_key, None)
>>
>> QUESTION #1 (maybe more suited for django-dev?): If I step in, I find
>> that debugging core.cache.backends.locmem.LocMemCache.get(), which is
>> odd since the TestCase is CacheDBSessionTests.  Why is it testing
>> LocMemCache and not core.cache.backends.db.DatabaseCache?
>>
>> In this case, self.session_key is 1, and indeed the cache has a
>> pickled "True" for the key 1.  In fact, during the course of debugging
>> core.cache.backends.locmem.LocMemCache.get(), we have found that in
>> every case during which the testrunner hits this method, 1 is a key
>> for pickled True.
>>
>> QUESTION #2: Why is the key 1 set to a pickled True?
>>
>> In this case, that True ends up being returned in
>> contrib.sessions.backends.base.SessionBase.get() (line 64).
>>
>> Thus, of course an error is raised.  We are unable to understand why
>> the key 1 is set to pickled True in the locmem backend.
>>
>> NOW THE STRANGE PART:
>>
>> Iff we step in with a debugger and wait for > 3 seconds or so, the
>> method no longer returns True and instead returns an empty dict.  In
>> this case, the test passes!
>>
>> We are baffled.  Is this possibly an underlying race condition or
>> hotel room scenario (http://stackoverflow.com/questions/6441218)?
>>
>> [END DJANGO-USERS MESSAGE]
>>
>> I also want to point a few additional things here on the dev list:
>>
>> *We have tried this on two different computers with OSX and Linux and
>> we encounter the timing issue either way, every time.
>> *We put a breakpoint on django.core.cache.backends.db.get() (line 38)
>> which was never hit during the course of the sessions tests.
>> *We were hoping to find that the test runner hit one backend if we
>> didn't wait in state and another if we did.  However, as noted in the
>> point above, that doesn't appear to be the case (unless we're missing
>> something about the way the backends are implemented, which is
>> possible)
>> *Thinking that using cached_db for the backend might result in .get()
>> being used only in LocMem, we also put a breakpoint in
>> backends.db.set(), thinking that both were going to be set.  This
>> breakpoint was also never hit during the course of the sessions tests.
>> 

Re: deprecation vs removal

[ptone]

On Oct 1, 9:10 am, Luke Plant  wrote:
> Hi all,
>
> The Django deprecation timeline [1] is very inconsistent in its usage of
> the terminology 'deprecated'. For example, the 1.5 section often says
> "is  deprecated" or "has been deprecated", when what they mean is "will
> be removed", which is what the other sections generally tend to say.
> Some in section 1.6 say a feature "will be deprecated".
>
> Can we have a consistent policy on this terminology?


I've opened a ticket on this and submitted a patch with an attempt to
be more consistent in language - review would be welcome:

https://code.djangoproject.com/ticket/16988
https://github.com/django/django/pull/58

-Preston

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: contrib.sessions test oddities

[Stephen Burrows]

Hi,

It sounds like you're looking at tests for
django.contrib.sessions.backends.cached_db. If that is the case, then
it's not unusual that you would end up calling the LocMem cache
backend. The cached_db session backend does all of its reads from the
cache, only falling back to the database sessions if the key is not
found in the cache. So, unless you define a different cache backend in
your settings.py file, the cached_db session backend will use the
LocMem cache backend.

That doesn't necessarily mean you haven't run into a problem, of
course. But it may explain some of what you're experiencing.

Best,
Stephen Burrows

On Oct 4, 9:22 pm, Justin Holmes  wrote:
> I posted this message on -users today, but at PaulM's urging, I'm
> bringing it over here out of concern that it may reflect a bug in the
> contrib.sessions test suite.
>
> Here's the original message from -users (some -dev only notes follow) :
>
> I am having a problem with contrib.sessions.tests.test_valid_key (line
> 159).  It's a mixin which in the failure case is mixed with
> CacheDBSessionTests.
>
> The test raises:
>
> "AttributeError: 'bool' object has no attribute 'get'"
>
> The error is actually raised by session.save() on 164.
>
> Stepping inward reveals that sessions.backends.cached_db (line 18)
> sets data thusly:
>
> data = cache.get(self.session_key, None)
>
> QUESTION #1 (maybe more suited for django-dev?): If I step in, I find
> that debugging core.cache.backends.locmem.LocMemCache.get(), which is
> odd since the TestCase is CacheDBSessionTests.  Why is it testing
> LocMemCache and not core.cache.backends.db.DatabaseCache?
>
> In this case, self.session_key is 1, and indeed the cache has a
> pickled "True" for the key 1.  In fact, during the course of debugging
> core.cache.backends.locmem.LocMemCache.get(), we have found that in
> every case during which the testrunner hits this method, 1 is a key
> for pickled True.
>
> QUESTION #2: Why is the key 1 set to a pickled True?
>
> In this case, that True ends up being returned in
> contrib.sessions.backends.base.SessionBase.get() (line 64).
>
> Thus, of course an error is raised.  We are unable to understand why
> the key 1 is set to pickled True in the locmem backend.
>
> NOW THE STRANGE PART:
>
> Iff we step in with a debugger and wait for > 3 seconds or so, the
> method no longer returns True and instead returns an empty dict.  In
> this case, the test passes!
>
> We are baffled.  Is this possibly an underlying race condition or
> hotel room scenario (http://stackoverflow.com/questions/6441218)?
>
> [END DJANGO-USERS MESSAGE]
>
> I also want to point a few additional things here on the dev list:
>
> *We have tried this on two different computers with OSX and Linux and
> we encounter the timing issue either way, every time.
> *We put a breakpoint on django.core.cache.backends.db.get() (line 38)
> which was never hit during the course of the sessions tests.
> *We were hoping to find that the test runner hit one backend if we
> didn't wait in state and another if we did.  However, as noted in the
> point above, that doesn't appear to be the case (unless we're missing
> something about the way the backends are implemented, which is
> possible)
> *Thinking that using cached_db for the backend might result in .get()
> being used only in LocMem, we also put a breakpoint in
> backends.db.set(), thinking that both were going to be set.  This
> breakpoint was also never hit during the course of the sessions tests.
> *The commit comment, by Russell Keith-Magee, seems like it is likely
> related: "Fixed #15026 -- Added cleanup to the invalid key session
> tests; when using Memcached as a cache backend, the cache-backed
> session backends would fail on the second run due to leftover cache
> artefacts. Thanks to jsdalton for the report and patch."
> *#15026 seems to describe a substantially different problem, but it is
> possible that the root cause is similar or the same: that unflushed
> material in the cache persists between tests.
>
> We encountered this issue at DiscSpace, which is cool enough to let
> freelancers work on issues in the django internals on company time if
> we encounter them.
>
> --
> Justin Holmes
>
> Head Instructor, SlashRoot Collective
> SlashRoot: Coffee House and Tech Dojo
> 60 Main Street
> New Paltz, NY 12561
> 845.633.8330

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

contrib.sessions test oddities

[Justin Holmes]

I posted this message on -users today, but at PaulM's urging, I'm
bringing it over here out of concern that it may reflect a bug in the
contrib.sessions test suite.

Here's the original message from -users (some -dev only notes follow) :

I am having a problem with contrib.sessions.tests.test_valid_key (line
159).  It's a mixin which in the failure case is mixed with
CacheDBSessionTests.

The test raises:

"AttributeError: 'bool' object has no attribute 'get'"

The error is actually raised by session.save() on 164.

Stepping inward reveals that sessions.backends.cached_db (line 18)
sets data thusly:

data = cache.get(self.session_key, None)

QUESTION #1 (maybe more suited for django-dev?): If I step in, I find
that debugging core.cache.backends.locmem.LocMemCache.get(), which is
odd since the TestCase is CacheDBSessionTests.  Why is it testing
LocMemCache and not core.cache.backends.db.DatabaseCache?

In this case, self.session_key is 1, and indeed the cache has a
pickled "True" for the key 1.  In fact, during the course of debugging
core.cache.backends.locmem.LocMemCache.get(), we have found that in
every case during which the testrunner hits this method, 1 is a key
for pickled True.

QUESTION #2: Why is the key 1 set to a pickled True?

In this case, that True ends up being returned in
contrib.sessions.backends.base.SessionBase.get() (line 64).

Thus, of course an error is raised.  We are unable to understand why
the key 1 is set to pickled True in the locmem backend.

NOW THE STRANGE PART:

Iff we step in with a debugger and wait for > 3 seconds or so, the
method no longer returns True and instead returns an empty dict.  In
this case, the test passes!

We are baffled.  Is this possibly an underlying race condition or
hotel room scenario (http://stackoverflow.com/questions/6441218)?

[END DJANGO-USERS MESSAGE]

I also want to point a few additional things here on the dev list:

*We have tried this on two different computers with OSX and Linux and
we encounter the timing issue either way, every time.
*We put a breakpoint on django.core.cache.backends.db.get() (line 38)
which was never hit during the course of the sessions tests.
*We were hoping to find that the test runner hit one backend if we
didn't wait in state and another if we did.  However, as noted in the
point above, that doesn't appear to be the case (unless we're missing
something about the way the backends are implemented, which is
possible)
*Thinking that using cached_db for the backend might result in .get()
being used only in LocMem, we also put a breakpoint in
backends.db.set(), thinking that both were going to be set.  This
breakpoint was also never hit during the course of the sessions tests.
*The commit comment, by Russell Keith-Magee, seems like it is likely
related: "Fixed #15026 -- Added cleanup to the invalid key session
tests; when using Memcached as a cache backend, the cache-backed
session backends would fail on the second run due to leftover cache
artefacts. Thanks to jsdalton for the report and patch."
*#15026 seems to describe a substantially different problem, but it is
possible that the root cause is similar or the same: that unflushed
material in the cache persists between tests.

We encountered this issue at DiscSpace, which is cool enough to let
freelancers work on issues in the django internals on company time if
we encounter them.


-- 
Justin Holmes

Head Instructor, SlashRoot Collective
SlashRoot: Coffee House and Tech Dojo
60 Main Street
New Paltz, NY 12561
845.633.8330

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: prevent brute force attacks by acting on several failed logins, for example by using a captcha

[Paul McMillan]

> I was thinking of adding a recaptcha implementation, based on the work
> of others, if that would work with licenses. But probably that choice
> would indeed introduce an external (unwanted) dependency.

Yeah... it's really a tricky problem. That solution is probably best
as an external app. Some users deploy Django apps with no net access
at all.

> About ticket 16860, I believe that to be over my head for now. In your
> opinion, is that ticket tied to the User discussion, about a more
> flexible/abstract User model (ticket 3011)?

It's tied to it, but 3011 is really hairy, and I believe we can do
good work in 16860 with or without the changes under discussion in
3011. I don't know when we will sort out 3011, but the stuff in 16860
can happen on a different timeframe.

> And/Or do you believe we have to deal with database migration first?

That would be nice, but I think we can work with what we currently have.

-Paul

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Handling admin media paths in the transition from 1.3 to 1.4

[Aramgutang]

> I think that's not a dilemma at all, Django 1.3 requires setting
> ADMIN_MEDIA_PREFIX to an appropriate value as before.

No, that's exactly why this is a problem. Django 1.3 doesn't require
setting ADMIN_MEDIA_PREFIX in the local settings, because it has a
default value that works for most users (i.e. anyone whose STATIC_URL
is /static/). Hence, most Django 1.3 projects I've worked on don't
define an ADMIN_MEDIA_PREFIX in their local settings. It seems
unreasonable for third-party apps to now require it to be set, since
it is now deprecated.

> A certain level of migration is expected (hence the deprecation warnings),
> so it'd be best to include setup instructions in your app's documentation.

That's another part of my point. The deprecation warnings in this case
are not helpful, since the {% admin_media_prefix %} tag simply stops
working as expected without any hints given as to why. For example,
say I have a 1.3 project, without an ADMIN_MEDIA_PREFIX set in the
local settings (I'm happy with the default value). I am also
overriding one of the admin templates, which contains a {%
admin_media_prefix %} tag. I now decide to upgrade to Django 1.4.
Suddenly, my admin CSS and JS stop working, and there's no warnings
that may help explain why this is happening, nor anything in the
release notes.

One of the following two options would solve the problem:

* Make the warning level on the use of {% admin_media_prefix %} match
the warning level of using ADMIN_MEDIA_PREFIX. Right now the former is
a PendingDeprecationWarning, and the latter is a DeprecationWarning.
They should probably both trigger DeprecationWarning.

* Make it clear in the release notes that {% admin_media_prefix %} no
longer works as expected in Django 1.4, and note that this affects
anyone overriding the base admin templates.

Keep in mind that the above only applies to the case of a local Django
project developer. This still doesn't fully address my initial reason
for posting, which is to provide a way for a third-party app that
overrides admin templates to provide compatibility with both 1.3 and
1.4.

> Can you elaborate what you had in mind?

Yes, I think the {% admin_media_prefix %} tag should return something
along the following lines:

getattr(settings, 'ADMIN_MEDIA_PREFIX',
urjoin(getattr(settings, 'STATIC_URL', ''), 'admin/')

This would allow third party apps to continue using {%
admin_media_prefix %} to provide support for both 1.3 and 1.4. When
1.5 hits, and the PendingDeprecationWarning on {% admin_media_prefix
%} becomes a DeprecationWarning, they will then be able to switch to
using the {% static %} tag to avoid the visible warnings while
providing support for both 1.4 and 1.5.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Django sprint in North Carolina, Nov. 12-13, 2011

[Jacob Kaplan-Moss]

On Tue, Oct 4, 2011 at 2:57 PM, Tobias McNulty  wrote:
> We'd like to host another Django sprint here at Caktus the weekend of
> November 12 and 13, 2011.

Hooray! Thanks for taking the initiative and getting this rolling.

If you need any help (financial or otherwise) from the DSF let me know
and I'll make it happen.

Jacob

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Django sprint in North Carolina, Nov. 12-13, 2011

[Tobias McNulty]

Hi All,

We'd like to host another Django sprint here at Caktus the weekend of
November 12 and 13, 2011.  A development sprint is an excuse to get
together, write some code, and have a good time doing it.  The purpose of
this sprint will be to help finish features and push out bug fixes in
preparation for the Django 1.4 release.  If you're interested in coming to
work on other open source Django-based projects, that's welcome too.

We'll be meeting at the Caktus Group office and we'll be here to open the
doors at 9am both days.

For more information, please check out the corresponding wiki page and RSVP
via Eventbrite if you're interested:

https://code.djangoproject.com/wiki/Sprint20TriangleNC

http://nc-django-sprint-2011-11.eventbrite.com/

If you you can't make it to NC but would like to participate online, you can
RSVP by adding your name to the following page:

https://code.djangoproject.com/wiki/Sprint20

We're still looking for a few sponsors to help out with lunch and snacks, so
check out the sponsors section of the wiki and add yourself (or your
company) if you'd like to bring something.  Feel free to drop me an email
off-list if you have any questions about sponsoring.

Hope to see you there!

Cheers,
Tobias
-- 
Tobias McNulty, Managing Member
Caktus Consulting Group, LLC
http://www.caktusgroup.com

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Patch for using custom managers in a reverse relation. (#3871)

[Vivek Narayanan]

Well, this is useful in the case where you have multiple custom
managers
defined on a model and you need to a select a specific manager. Doing
so,
 common filter operations that are long expressions or are quite
complicated,
 need not be repeated every time.

On Oct 4, 9:40 am, Tai Lee  wrote:
> Your custom manager could do nothing to filter results by default and
> so behave the same as the default automatic manager, but provide
> additional methods that filter the results as you need.

If the get_query_set() method of a manager is overridden, the
filtering
behavior can actually be changed. In the tests there is an example
does this.

>
> Cheers.
> Tai.
>
> On Oct 3, 3:25 pm, Vivek Narayanan  wrote:
>
>
>
>
>
>
>
> > Hi,
>
> > I've added a patch that provides functionality for selecting a custom
> > manager in a reverse relation , rather than the default manager. For
> > example:
>
> >          author = Author.objects.get(id=1)
> >          # Selects the manager 'foobar' of Post.
> >          mgr = author.post_set.managers("foobar")
>
> >https://code.djangoproject.com/ticket/3871
>
> > Would be great if someone could review it.
>
> > Vivek

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: ILIKE vs. LIKE LOWER() for PostgreSQL

[Wim Lewis]

On 4 Oct 2011, at 9:19 AM, Jonas H. wrote:
> On 10/04/2011 05:51 PM, Ted Gruenloh wrote:
>> The django online documentation mentions that the SQL equivalent for  
>> __icontains is something like:
>> 
>> SELECT ... WHERE headline ILIKE '%Lennon%';
>> However, for postgresql - one of the dbs that actually supports ILIKE - I 
>> noticed __icontains was actually performing something similar to:
>> SELECT ... WHERE LOWER(headline) LIKE LOWER('%Lennon%');
>> 
>> The ILIKE is obviously much faster [...]
> 
> https://code.djangoproject.com/ticket/3575

Though apparently pg 9.1's trigram indexes can be used for ILIKE searches (but 
aren't a win for all query patterns). It seems to me it'd be worthwhile to have 
a setting to tell the backend whether to use ILIKE vs. LIKE LOWER, though 
personally I'm not using __icontains so I don't have a dog in this fight :)


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

UPPER vs. ILIKE for __icontains

[Ted Gruenloh]

Jonas:

Thanks for the response.  The ticket you referenced refers to __iexact.   In 
that case, I agree with the use of UPPER.  My question/suggestion is about 
__icontains, which must either use '...LIKE UPPER()' or ILIKE.  In my quick 
tests, ILIKE wins by about 10% on a table with 1 million records.  ILIKE gets 
better the more columns you're comparing, too (30% better, in my example).  See 
my examples below.


Anyway, I changed it in my implementation because it helped significantly in my 
specific case.  Hope this helps.

thanks
ted


sdb=# explain analyze select * from activity where msg ilike '%exploit abc 
akndkf%';
  QUERY 
PLAN  
--
 Seq Scan on activity  (cost=0.00..42147.00 rows=1 width=192) (actual 
time=1802.493..1802.493 rows=0 loops=1)
   Filter: ((msg)::text ~~* '%exploit abc akndkf%'::text)
 Total runtime: 1802.553 ms
(3 rows)

sdb=# explain analyze select * from activity where upper(msg) like 
upper('%exploit abc akndkf%');
   QUERY 
PLAN   

 Seq Scan on activity  (cost=0.00..44647.00 rows=100 width=192) (actual 
time=2074.101..2074.101 rows=0 loops=1)
   Filter: (upper((msg)::text) ~~ '%EXPLOIT ABC AKNDKF%'::text)
 Total runtime: 2074.158 ms
(3 rows)



sdb=# explain analyze select * from activity where msg ilike '%123%' or 
text_src ilike '%123%' or text_dst ilike '%123%' order by msg;
    QUERY 
PLAN 
---
 Sort  (cost=48622.98..48646.30 rows=9327 width=192) (actual 
time=2241.154..2270.481 rows=3774 loops=1)
   Sort Key: msg
   Sort Method: external merge  Disk: 808kB
   ->  Seq Scan on activity  (cost=0.00..47147.00 rows=9327 width=192) (actual 
time=11.701..2178.329 rows=3774 loops=1)
 Filter: (((msg)::text ~~* '%123%'::text) OR ((text_src)::text ~~* 
'%123%'::text) OR ((text_dst)::text ~~* '%123%'::text))
 Total runtime: 2298.757 ms
(6 rows)

sdb=# explain analyze select * from activity where upper(msg) like 
upper('%123%') or upper(text_src) like upper('%123%') or upper(text_dst) like 
upper('%123%') order by msg;
 QUERY 
PLAN  
-
 Sort  (cost=85617.58..85905.74 rows=115264 width=192) (actual 
time=3117.558..3147.699 rows=3774 loops=1)
   Sort Key: msg
   Sort Method: external merge  Disk: 808kB
   ->  Seq Scan on activity  (cost=0.00..54647.00 rows=115264 width=192) 
(actual time=8.239..3064.163 rows=3774 loops=1)
 Filter: ((upper((msg)::text) ~~ '%123%'::text) OR 
(upper((text_src)::text) ~~ '%123%'::text) OR (upper((text_dst)::text) ~~ 
'%123%'::text))
 Total runtime: 3177.240 ms
(6 rows)






- Original Message -
From: Jonas H. 
To: django-developers@googlegroups.com
Cc: 
Sent: Tuesday, October 4, 2011 11:19 AM
Subject: Re:

On 10/04/2011 05:51 PM, Ted Gruenloh wrote:
> The django online documentation mentions that the SQL equivalent for  
> __icontains is something like:
>
>      SELECT ... WHERE headline ILIKE '%Lennon%';
> However, for postgresql - one of the dbs that actually supports ILIKE - I 
> noticed __icontains was actually performing something similar to:
>      SELECT ... WHERE LOWER(headline) LIKE LOWER('%Lennon%');
>
> The ILIKE is obviously much faster [...]

https://code.djangoproject.com/ticket/3575

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re:

[Jonas H.]

On 10/04/2011 05:51 PM, Ted Gruenloh wrote:

The django online documentation mentions that the SQL equivalent for  
__icontains is something like:

 SELECT ... WHERE headline ILIKE '%Lennon%';
However, for postgresql - one of the dbs that actually supports ILIKE - I 
noticed __icontains was actually performing something similar to:
 SELECT ... WHERE LOWER(headline) LIKE LOWER('%Lennon%');

The ILIKE is obviously much faster [...]


https://code.djangoproject.com/ticket/3575

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

[no subject]

[Ted Gruenloh]

All:

I'm not a Django developer, nor will I pretend to be.  But I did come across 
something I thought you should be aware of.

The django online documentation mentions that the SQL equivalent for  
__icontains is something like:

    SELECT ... WHERE headline ILIKE '%Lennon%';
However, for postgresql - one of the dbs that actually supports ILIKE - I 
noticed __icontains was actually performing something similar to:
    SELECT ... WHERE LOWER(headline) LIKE LOWER('%Lennon%');

The ILIKE is obviously much faster, so, I dug around a little and changed a 
line or two in these files:

/usr/local/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py
/usr/local/lib/python2.7/site-packages/django/db/backends/postgresql/operations.py

...so __icontains actually uses ILIKE instead, and it's running great.

OK:

1) Any reason I shouldn't have done that, performance wise?
2) Should this be changed in the codebase going forward?


thanks, and I hope this helps,
ted

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Feature proposal: Q.push(...) and QuerySet.filter(relfield=Q(...))

[Javier Guerra Giraldez]

On Tue, Oct 4, 2011 at 4:35 AM, Johannes Dollinger
 wrote:
> as you could write Q(foo=Q(...)) instead of Q(..).push('foo')

I thought that would work magically, since the Q object would just
pass that to the filter(); but a quick test proved me wrong.

I've just refactored as a Node._prefix() method that calls a
freestanding _prefix(t,pfx), and moved the relfield=Q smarts to the Q
constructor.

-- 
Javier

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Feature proposal: Q.push(...) and QuerySet.filter(relfield=Q(...))

[Javier Guerra Giraldez]

On Tue, Oct 4, 2011 at 4:35 AM, Johannes Dollinger
 wrote:
> Thanks for starting this!
> Why did you chose a staticmethod instead of an instancemethod?

because it's applied recursively not just to tree.Node, but to the
children and all members.  it would be cleaner as a free function and
an instance method on tree.Node that just calls it.

> While I'm not sold on the method name (perhaps prefix() or shift() or even 
> __rshift__() would be clearer), I believe the ability to modify Q objects 
> outside of filter()-Expressions is important.

__rshift__() is too C++ for my taste :-)  but I see the motivation.

> Just like &, |, and ~, adding a prefix is an operation on Q objects. And the 
> API should provide a way to manipulate them fully outside of QuerySets.
>
> As an afterthought: The cleanest solution would probably be an implementation 
> in Q.__init__.
> The .push() method would then be unnecessary, as you could write 
> Q(foo=Q(...)) instead of Q(..).push('foo').
> And .filter()/.exclude() would just work, as they simply pass their arguments 
> through to Q.__init__.

yes, that sounds cleaner.  i'm still not sure if that should be on Q
or on tree.Node.

-- 
Javier

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Handling admin media paths in the transition from 1.3 to 1.4

[Jannis Leidel]

Aramgutang,

> Currently, it is unclear as to how apps that override admin templates
> should maintain compatibility with both 1.3 and 1.4 when referring to
> the admin media file URLs in the overridden templates. This applies
> primarily to third-party apps like django-admin-tools, that want to
> support at least the latest two major Django versions.
> 
> The changes I'm referring to are ones in the giant r16487 commit
> ( 
> https://github.com/django/django/commit/09395597145d35f3793d57f70b7795214f64
> ).
> 
> Since that change, Django now constructs URLs using `{% static "admin/
> css/base.css" %}` internally, instead of the old `{%
> admin_media_prefix %}css/base.css` method. That change also saw the
> removal of `ADMIN_MEDIA_PREFIX = '/static/admin/'` from
> djago.conf.global_settings. This results in the following dilemma for
> any app that wishes to maintain compatibility with both 1.3 and 1.4:
> 
> To maintain support for 1.3, the app cannot use {% static %}, as it is
> not available yet, so it uses {% admin_media_prefix %}, which causes a
> PendingDeprecationWarning that we're OK to ignore. However, due to the
> removal of the default ADMIN_MEDIA_PREFIX value, the {%
> admin_media_prefix %} tag no longer works as expected, returning "css/
> base.css" instead of "/static/css/base.css". To overcome this, the app
> needs to ask that the project define an ADMIN_MEDIA_PREFIX, or define
> one itself, nether of which is desirable. Furthermore, having an
> ADMIN_MEDIA_PREFIX defined now triggers a DeprecationWarning, which is
> visible, and we're not OK to ignore it anymore.

I think that's not a dilemma at all, Django 1.3 requires setting
ADMIN_MEDIA_PREFIX to an appropriate value as before. But you probably
want to prepare your user to remove it at some point (once they upgrade
to 1.4.X) to prevent any deprecation warnings. Feel free to point to the
1.4 release notes.

Also, as a hint to your users, I recommend to raise an ImproperlyConfigured
exception in your own app (e.g. in the management or models module) if
django.VERSION is lower than 1.4 and no `ADMIN_MEDIA_PREFIX` is specified. 

> Of course, the app can provide its own {% admin_media_prefix %} tag,
> where it checks for ADMIN_MEDIA_PREFIX first, and returns STATIC_URL/
> admin if it's not found, but this seems like the kind of functionality
> that should be provided by Django itself, to reduce the number of apps
> that will break with the upgrade to 1.4.

A certain level of migration is expected (hence the deprecation warnings),
so it'd be best to include setup instructions in your app's documentation.

> I'm happy to write a ticket and a patch, but just wanted to hear from
> the core devs' opinion on the issue first.

IMO, there is no need for a ticket or patch as this isn't a bug, Django
needs to move away from special casing the admin, now that we have a
general solution with `staticfiles`. Can you elaborate what you had in
mind?

Jannis

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Feature proposal: Q.push(...) and QuerySet.filter(relfield=Q(...))

[Johannes Dollinger]

Am 03.10.2011 um 20:01 schrieb Javier Guerra Giraldez:

> On Fri, Sep 30, 2011 at 4:37 PM, Johannes Dollinger
>  wrote:
>> The aim of this proposal is to reuse Q objects for models that are related 
>> through FK or M2M fields.
> 
> i really want to have this feature!  so i went and did a quick
> implementation and created ticket #16979[1]
> 
> [1]:https://code.djangoproject.com/ticket/16979
> 
> really untested, but a variation that doesn't modify core seems to
> work well (but the API is far too ugly to share)

Thanks for starting this!
Why did you chose a staticmethod instead of an instancemethod?
While I'm not sold on the method name (perhaps prefix() or shift() or even 
__rshift__() would be clearer), I believe the ability to modify Q objects 
outside of filter()-Expressions is important.
Just like &, |, and ~, adding a prefix is an operation on Q objects. And the 
API should provide a way to manipulate them fully outside of QuerySets.

As an afterthought: The cleanest solution would probably be an implementation 
in Q.__init__.
The .push() method would then be unnecessary, as you could write Q(foo=Q(...)) 
instead of Q(..).push('foo').
And .filter()/.exclude() would just work, as they simply pass their arguments 
through to Q.__init__.

__
Johannes

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: deprecation vs removal

[Luke Plant]

On 04/10/11 05:56, Tai Lee wrote:
> 
> 
> On Oct 4, 11:17 am, Russell Keith-Magee 
> wrote:
>> I'm completely agreed that the 'soft' deprecation is useful. I'm just
>> complaining about the ambiguity in the language: "We're deprecating
>> this feature by marking it PendingDeprecation...".
> 
> What about just changing "PendingDeprecation..." to
> "SoftDeprecation..." or "QuietDeprecation..."? It's not really pending
> deprecation. It is deprecated already, but we just don't loudly
> complain (yet) if people haven't updated their code immediately.

PendingDeprecationWarning and DeprecationWarning are Python builtins:

http://docs.python.org/library/exceptions.html

so we can't change the names unless we want to introduce our own
alternative, which would definitely be a bad idea IMO.

PendingDeprecation is already quiet by default.

Regards,

Luke

-- 
"I was sad because I had no shoes, until I met a man who had no
feet. So I said, "Got any shoes you're not using?"  (Steven Wright)

Luke Plant || http://lukeplant.me.uk/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: prevent brute force attacks by acting on several failed logins, for example by using a captcha

[Wim Feijen]

Hi Paul,

Thanks for your review and remarks. I will think on it more, and will
write a longer response later.

For the moment just this quick reponse:

I was thinking of adding a recaptcha implementation, based on the work
of others, if that would work with licenses. But probably that choice
would indeed introduce an external (unwanted) dependency.

About ticket 16860, I believe that to be over my head for now. In your
opinion, is that ticket tied to the User discussion, about a more
flexible/abstract User model (ticket 3011)?

And/Or do you believe we have to deal with database migration first?

Wim

On Oct 2, 9:21 am, Paul McMillan  wrote:
> My sense is that you're conflating 2 kinds of protection here because
> you haven't made a decision. Do you want to propose rate limiting, or
> a captcha? Answers to your points depend on that.
>
> Prior to more specific work on this matter (and before anything can be
> included in core), we need to address the issues in 16860. The generic
> groundwork must be done first. It would be helpful to examine this
> proposal in light of those concerns. What hooks does Django need to
> provide to allow this to be implemented cleanly as a third-party
> installable?
>
> This needs to be pluggable because (no matter what we include), it
> won't meet the requirements for a sizable subset of Django's users
> (many of whom have very explicit requirements).
>
> > Points 1-3
>
> Our protection should be on by default if we include it in Django at
> all. This means that the default configuration will be universal
> enough that it doesn't get in the way for most installations.
>
> > 4. After x failed login attempts, protection kicks in. x is a
> > configurable amount of times, which by default is 3?
>
> 3 is too few for a default on a general user-facing site. Brute force
> attempts try hundreds of passwords. If normal users see this
> protection with any regularity, people will turn it off.
>
> > 5. Failed logins are either stored in a database (which works well for
> > small systems and protects against slow distributed attacks), or in
> > memory (for large systems). Default: use the database? Because most
> > users operate on small systems?
>
> Probably the database. An extra column on users would be the most
> obvious place, but it's a no-go because we don't have migrations and
> this functionality should be separately pluggable anyway. We would
> need to ship with a default set to off in the base settings file, and
> explicitly set it on for new projects. If it adds database columns,
> that might be an argument for shipping with it disabled.
>
> > 6. We protect against the following scenarios:
> >    a. Login attempts coming from many IP-addresses and targeting a
> > single user account;
> >    b. From a single IP-address, targeting a single user;
> >    c. Single IP-address, more than one user.
> >    Case 6a and - in a lesser extent - 6b are strong indicators for a
> > brute force attack. Case 6c might be brute force, but might also be
> > triggered by many users behind a proxy.
>
> These are all attack vectors. There's also multiple IP multiple
> account slow brute force, and many other variations. Any of these
> options is going to need to be quite configurable to work for most
> Django users.
>
> > 7. Protection may consist of:
> >    a. denying access for x minutes for a given user or IP-address. x
> > is configurable, and by default: 5 minutes?
>
> Rate limiting is more user friendly than a hard cutoff like this. The
> hard cutoff is easier to explain to a user, though. This allows a DoS
> attack against specific users.
>
> >    b. adding a sleeptime to login requests (mentioned several times,
> > but sounds very weak to me, because it can be easily passed by opening
> > a new connection?).
>
> Absolutely no. Adding sleep() anywhere in Django opens nasty DoS
> avenues. Sleep ties up workers for no reason.
>
> >    c. logging it, and/or notifying the admin
> >    d. adding a captcha to login form
>
> Which captcha do you propose? Is there a good one which does not add
> external dependencies? We can't require compiled dependencies like PIL
> out of the box.
>
> > 8. Protection should be configurable as well. By default: use a
> > captcha? Using a captcha prevents an attacker from using the denial
> > trigger for a DoS-attack.
>
> Captchas do that. They also introduce usability issues. Do you have a
> pure-python captcha which is also ADA compliant? How do you recommend
> we balance the difficulty (for both humans and robots) of the captcha?
>
> > 9. Rate limitors should be generably applicable, to all views.
>
> Yes. This is why they are probably best viewed as a separate feature.
>
> > 10. Final question:
> > Should this be in Django contrib? I argue in favor, in order to
> > protect the innocent and keep everyone safe.
>
> I agree that Django should ship some form of login protection.
>
> >  django.contrib.security
> > seems a proper place to me.
>
>