Re: ModelForm.Meta.overrides

2012-08-20 Thread Jannis Leidel
Simon,

> A couple of months ago Jannis closed #/17924 [1] as wontfix, stating "I'm 
> violently -1 on the whole topic "meta programming form fields after they've 
> been instantiated", it's a mess. Yes it would be more DRY, but also much 
> harder to know how the hell the form fields are composed as. Just override 
> the form field, it's not a huge deal code wise and much easier to grok." This 
> has since been contested [2] (though for invalid reasons in my opinion).
> 
> I'm not sure that Jannis and I were talking about the same thing; the 
> solution I had in mind did not involve changing form-fields after their 
> instantiation, but rather providing a more elegant and flexible mechanism 
> that works in a similar way to ModelForm.Meta.widgets and formfield_callback.

Yup, we're talking about the same thing. No matter how you turn it, whether 
changing attributes of form fields after they've been instantiated *or* 
providing an "override" for arguments passed to the __init__ of those forms 
fields it's still unintuitive and not Pythonic.

We added the widgets option to the model form Meta class to allow overriding 
the widgets without having to specify the whole field again. It helped us solve 
a annoying bit of reusability of form fields without increasing the number of 
places users need to be aware of when they write model forms. That works well 
with widgets since they are clearly defined pieces of code with just one 
purpose.

Opening up the Meta class to be a place where you can define the arguments 
passed to the form fields' __init__ method would be a step in the wrong 
direction because it increases the number of places users need to look for 
initial arguments of form fields. It's a bad idea to allow users to shoot 
themselves in the foot like that. We want to promote standard Python behavior, 
not add more magic to save a couple lines of code.

Jannis

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Re: Docs mistake

2012-08-20 Thread Russell Keith-Magee
Hi,

Thanks for the report -- if you want to make sure this isn't
forgotten, you should open a ticket and describe the fix you think
needs to be made. Reporting to the mailing list is helpful, but is
just as likely to get lost in someone's inbox.

Yours,
Russ Magee %-)

On Tue, Aug 21, 2012 at 9:45 AM, Yo-Yo Ma  wrote:
> At:
>
> https://docs.djangoproject.com/en/dev/ref/models/fields/#django.db.models.FileField.upload_to
>
> The docs make mention of the "url" attribute being MEDIA_ROOT + upload_to. I 
> think whoever wrote it meant that the aforementioned is how the file 
> name/path is determined, and that also MEDIA_URL + upload_to is used to make 
> the "url" attribute.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Django developers" group.
> To view this discussion on the web visit 
> https://groups.google.com/d/msg/django-developers/-/yWuBLG3QTY8J.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/django-developers?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Re: Streaming HttpResponse revisted. Any core devs, please take a look if you can :)

2012-08-20 Thread Tai Lee
It sounds like the preferred solution is still "don't do that", which I 
think corresponds most closely with option 1. Simply consuming generator 
content in `HttpResponse.__init__()` in all cases would prevent some of the 
surprising behaviour we are seeing, but it would completely break any 
support for streaming responses.

I thought that Jacob's "It's clear to me that we have to do *something* 
about this" comment when he marked #7581 as "accepted" would take "don't do 
that" off the table.

>
It might not count as a "backwards incompatible" change to revoke all 
support for streaming responses, as I couldn't find explicit support for 
them mentioned in the docs, but streaming responses have been around for a 
long time and people are relying on on this behaviour, as evidenced by 
several tickets and reported use cases (not limited to "large" responses, 
but also including "long running" responses).

Those people just have to avoid some core middleware that currently break 
streaming responses by prematurely consuming content, and Django doesn't 
provide a generic or documented way for people to hook into this part of 
the framework on a per-response basis. They just have to choose "all or 
nothing" when it comes to those middleware, unless they also have control 
over the middleware (which they don't for core, contrib and third party 
middleware).

I don't see a huge difference between wrapping or replacing the content of 
response, compared to wrapping or replacing the entire response. The key 
issue (for me) still remains, being that middleware would still have no way 
to know when it is appropriate for them to wrap or replace the entire 
response. It may also cause new complications with headers or properties of 
the wrapped response not being preserved.

Removing `HttpResponse.content` entirely is probably a non-starter, as it 
would be a backwards incompatible change. It has been part of the public 
API since at least 1.0.

Could somebody please explain their primary objections to option 2? This 
has a working up-to-date implementation already (just missing tests and 
docs, which I would be happy to contribute), is backwards compatible, fixes 
buggy behaviour in the general case where iterators are passed to 
`HttpResponse` as content but the developer does not explicitly want a 
streaming response, and it provides a hook for developers to explicitly 
tell middleware that the response should be streamed with 
`HttpResponse(stream_content=True)`.

I've seen it asked, but not answered yet (unless I missed it), if there is 
really a need for more fine grained control than "is it streaming, or not?" 
(e.g. a capabilities based API on responses). If not, I think this is 
currently the most likely candidate for inclusion.

Thanks.
Tai.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/F5cUHy1oOnsJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Docs mistake

2012-08-20 Thread Yo-Yo Ma
At:

https://docs.djangoproject.com/en/dev/ref/models/fields/#django.db.models.FileField.upload_to

The docs make mention of the "url" attribute being MEDIA_ROOT + upload_to. I 
think whoever wrote it meant that the aforementioned is how the file name/path 
is determined, and that also MEDIA_URL + upload_to is used to make the "url" 
attribute.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/yWuBLG3QTY8J.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Re: Streaming HttpResponse revisted. Any core devs, please take a look if you can :)

2012-08-20 Thread Forest Bond
Hi Tai,

On Mon, Aug 20, 2012 at 08:39:07AM -0700, Tai Lee wrote:
> I'd like to re-visit the discussion surrounding #7581 [1], a ticket about
> streaming responses that is getting quite long in the tooth now, which Jacob
> finally "accepted" 11 months ago (after a long time as DDN) and said that it 
> is
> clear we have to do *something*, but *what* remains to be seen.

Since the last time this came up I've wondered if the simplest solution might be
to make it easier to integrate raw WSGI apps into a site.  I still am not 100%
sure it's the right approach but I thought it was worth mentioning.

The argument for this is that the use cases for streaming responses are usually
corner cases that are well outside of Django's sweet spot.  They go against the
grain of the Django's flexible request/response handling (middleware in
particular).  But they would be trivial to handle in a raw WSGI app.

If Django made it easy to hand off requests to a WSGI app (i.e. mount the app
somewhere in the URLconf and perhaps expose authentication and/or other session
data) the framework could get out of the way and let the WSGI app have more
direct control over response handling.  I think there are probably other
scenarios where easy WSGI integration would be beneficial, too.

For what it's worth...

Thanks,
Forest
-- 
Forest Bond
http://www.alittletooquiet.net
http://www.rapidrollout.com


signature.asc
Description: Digital signature


Re: Draft branch: Swappable User models in contrib.auth

2012-08-20 Thread Russell Keith-Magee
On Mon, Aug 20, 2012 at 7:11 AM, Russell Keith-Magee
 wrote:
> On Sun, Aug 19, 2012 at 10:02 AM, Victor Hooi  wrote:
>> Hi,
>>
>> I'm just wondering, has there been any updates on the User model refactor?
>>
>> My understanding is that this is the official way of handling Users going
>> forward.
>>
>> Is there any roadmap on when it might hit trunk? I didn't see any reference
>> to User models in the 1.5 release notes.
>
> The precise answer? Soon. Ish. :-)
>
> There's a draft branch which works [1], but requires some more testing
> and documentation. It also hasn't been updated to reflect the recent
> py3k changes.
>
> I'm sprinting at PyCon AU this year, so I'm hoping to at least get the
> branch up to date. I might also be able to convince someone to work on
> the documentation issue.
>
> I still want to get this in for 1.5; it's just a matter of find the spare 
> time.
>
> [1] https://github.com/freakboy3742/django/tree/t3011

A follow up on post-PyCon AU sprint activity -- I've gotten my branch
up to date with Django trunk; there has been some work done on getting
some documentation ready; and a few more pairs of eyes have been over
the code looking for admin integration problems and other assumptions
that I've made that might not shake out. Thanks to everyone at PyCon
AU that pitched in (Thomas Ashelford, Greg Turner, Brenda Moon, Simon
Meers, and a few others).

There's one issue lurking with testing that has some potential to turn
into a yak shaving exercise, but if we shave that yak, we'll have
another big feature to put on the list for Django 1.5. I'll try to
post to django-developers about this in the near future (hopefully
before DjangoCon US).

Yours,
Russ Magee %-)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Re: Streaming HttpResponse revisted. Any core devs, please take a look if you can :)

2012-08-20 Thread Alex Ogier
On Mon, Aug 20, 2012 at 3:21 PM, Alex Gaynor  wrote:

> My inclination is that we should kill .content entirely, middleware that
> wants to rewrite the response will have two choices:
>
> 1) explicitly evaluate the entire response, and return a new HttpResponse
> object
> 2) return s anew HttpResponse object that has a generator that iteratively
> evaluates the previous one and rewrite it incrementally.
>
> Alex
>

Doesn't that break anyone who hangs arbitrary attributes on their
HttpResponse objects for the sake of their middleware? I've never done this
on the way out, only on the way in on HttpRequest objects, but I can
imagine it being useful if you wanted to conditionally disable middleware
or something. You could argue that depending on the same instance making it
through several layers of middleware is already iffy, but recommending as a
best practice to create new HttpResponses would *really* break this.

Best,
Alex Ogier

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Re: Streaming HttpResponse revisted. Any core devs, please take a look if you can :)

2012-08-20 Thread Anssi Kääriäinen
On 20 elo, 22:21, Alex Gaynor  wrote:
> My inclination is that we should kill .content entirely, middleware that
> wants to rewrite the response will have two choices:
>
> 1) explicitly evaluate the entire response, and return a new HttpResponse
> object
> 2) return s anew HttpResponse object that has a generator that iteratively
> evaluates the previous one and rewrite it incrementally.

+1

The above still leaves one part of the problem unsolved. If you have a
middleware that wants to access the content of the response for some
reason, it has no way to know if the content is too large to be
accessed safely. I do think we need in addition some way to tell the
middleware that the content is too large to be handled at all in
Python. As an addition to the already suggested ideas maybe a view
decorator could work?

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Re: ManyToManyField

2012-08-20 Thread MN TS
Ok
Thaks for you reply.


On Tue, Aug 21, 2012 at 3:37 AM, Karen Tracey  wrote:

> Please ask questions about using Django on django-users. The topic of this
> list is the development of Django itself.
>
> Thanks,
> Karen
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Re: ManyToManyField

2012-08-20 Thread Karen Tracey
Please ask questions about using Django on django-users. The topic of this
list is the development of Django itself.

Thanks,
Karen

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



ManyToManyField

2012-08-20 Thread MN TS
Hello
I'm new in Django using 1.4version.
So got some error.

'myapp.models.AdversiteImages'> has no ForeignKey to 

*
MODEL*

class AdversiteImages(models.Model):
image = models.FileField(u'Photo' , upload_to='adversiteimage/%Y/%
m/%d', null=True, blank=True)

class Adversite(models.Model):
category = models.ForeignKey(AdversiteCategory, verbose_name=u'Зарын
ангилал', related_name='adv_cat', blank=True, null=True)
image_many = models.ManyToManyField(AdversiteImages,
verbose_name=u'Photos',related_name="photos")
title = models.CharField(u'Title', max_length=128)
body = models.TextField(u'Description')


*ADMIN*

class
AdversiteImagesInline(admin.TabularInline):

model = Adversite.image_many.through
extra = 1

class AdversiteAdmin(admin.StackedInline):
fieldsets = ((u'Ерөнхий',
{'fields':('category','title','body','code')}),
 (u'Тайлбар',
{'fields':('price','phone','email','image1','image2','image3','image4','image5'

,'is_active','is_special','is_premium','start_at','finish_at')}))
list_display = ('category','title','code','is_active','is_special')
search_fields = ('category','title','code','is_active')
model = Adversite.image_many.through

Please help me. What happened to me?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Re: Streaming HttpResponse revisted. Any core devs, please take a look if you can :)

2012-08-20 Thread Alex Gaynor
On Mon, Aug 20, 2012 at 12:19 PM, Anssi Kääriäinen
wrote:

>
>
> On 20 elo, 18:39, Tai Lee  wrote:
> > I'd like to re-visit the discussion surrounding #7581 [1], a ticket about
> > streaming responses that is getting quite long in the tooth now, which
> > Jacob finally "accepted" 11 months ago (after a long time as DDN) and
> said
> > that it is clear we have to do *something*, but *what* remains to be
> seen.
> >
> > I'd like to try provide a little refresher and summarise the options that
> > have been suggested, and ask any core devs to please weigh in with their
> > preference so that I can work up a new patch that will be more likely to
> > gain acceptance.
> >
> > THE PROBLEM:
> >
> > 1. There are bugs and surprising behaviour that arise when creating an
> > HttpResponse with a generator as its content, as a result of "quantum
> > state" of `HttpResponse.content` (measuring it changes it).
> >
> >
> >
> > >>> from django.http import HttpResponse
> > >>> r = HttpResponse(iter(['a', 'b']))
> > >>> r.content
> > 'ab'
> > >>> r.content
> > ''
> > >>> r2 = HttpResponse(iter(['a', 'b']))
> > >>> print r2.content
> > ab
> > >>> print r2.content
> > >>> r3 = HttpResponse(iter(['a', 'b']))
> > >>> r3.content == r3.content
> >
> > False
> >
> > 2. Some middleware prematurely consume generator content by accessing
> > `HttpResponse.content`, which can use a lot of memory and cause browser
> > timeouts when attempting to stream large amounts of data or
> > slow-to-generate data.
> >
> > There have been several tickets [2] [3] [4] and django-developers
> > discussions [5] [6] [7] [8] about these issues.
> >
> > SOME USE CASES FOR STREAMING RESPONSES:
> >
> > A. Generating and exporting CSV data directly from the database.
> >
> > B. Restricting file access to authenticated users for files that may be
> > hosted on external servers.
> >
> > C. Drip-feeding chunks of content to prevent timeout when requesting a
> page
> > that takes a long time to generate.
> >
> > OPTION 1:
> >
> > Remove support for "streaming" responses. If an iterator is passed in as
> > content to `HttpResponse`, consume it in `HttpResponse.__init__()` to
> > eliminate buggy behaviour. Middleware won't have to worry about what type
> > of content a response has.
> >
> > Now that Jacob has accepted #7581 and said that it is clear we need to do
> > *something*, I hope we can rule out this option.
> >
> > OPTION 2:
> >
> > Make `HttpResponse.__init__()` consume any iterator content, and add an
> > `HttpResponseStreaming` class or an `HttpResponse(streaming=False)`
> > argument. Allow middleware to check via `hasattr()` or `isinstance()`
> > whether or not the response has generator content, and conditionally skip
> > code that is incompatible with streaming responses.
> >
> > Some middleware will have to be updated for compatibility with streaming
> > responses, and any 3rd party middleware that prematurely consumes
> generator
> > content will continue to work, only without the bugs (and potentially
> with
> > increased memory usage and browser timeouts).
> >
> > OPTION 3:
> >
> > Build a capabilities API for `HttpResponse` objects, and have middleware
> > inspect responses to determine "can I read content?", "can I replace
> > content?", "can I change etag?", etc. This will likely become a bigger
> and
> > more complicated design decision as we work out what capabilities we want
> > to support. Some have argued that it should be sufficient to know if we
> > have generator content or not, for all the cases that people have
> reported
> > so far.
> >
> > OPTION 4:
> >
> > Provide a way for developers to specify on an `HttpResponse` object or
> > subclass that specific middleware should be skipped for that response.
> This
> > would be problematic because 3rd party views won't know what other
> > middleware is installed in a project in order to name them for exclusion.
> >
> > OPTION 5:
> >
> > Add Yet Another Setting that would allow developers to define
> > `CONDITIONAL_MIDDLEWARE_CLASSES` at a project level. At the project
> level,
> > developers would know which middleware classes they are using, and when
> > they should be executed or skipped. This would give very fine grained
> > control at a project level to match middleware conditionally with
> > `HttpResponse` subclasses, without requiring any changes to existing or
> 3rd
> > party middleware. This could look something like this:
> >
> > MIDDLEWARE_CLASSES = (
> > 'django.middleware.common',
> > )
> >
> > CONDITIONAL_MIDDLEWARE_CLASSES = {
> > 'exclude': {
> > 'django.http.HttpResponseStreaming': ['django.middleware.common',
> > 'otherapp.othermiddleware', ...],
> > },
> > 'include': {
> > 'myapp.MyHttpResponse': ['myapp.mymiddleware', ...],
> > },
> >
> > }
>
> My initial feeling is that anything accessing "large content" in
> Python is broken and we should not try to make sure these things
> continue to work. The problem is how we define if the conten

Re: Streaming HttpResponse revisted. Any core devs, please take a look if you can :)

2012-08-20 Thread Anssi Kääriäinen


On 20 elo, 18:39, Tai Lee  wrote:
> I'd like to re-visit the discussion surrounding #7581 [1], a ticket about
> streaming responses that is getting quite long in the tooth now, which
> Jacob finally "accepted" 11 months ago (after a long time as DDN) and said
> that it is clear we have to do *something*, but *what* remains to be seen.
>
> I'd like to try provide a little refresher and summarise the options that
> have been suggested, and ask any core devs to please weigh in with their
> preference so that I can work up a new patch that will be more likely to
> gain acceptance.
>
> THE PROBLEM:
>
> 1. There are bugs and surprising behaviour that arise when creating an
> HttpResponse with a generator as its content, as a result of "quantum
> state" of `HttpResponse.content` (measuring it changes it).
>
>
>
> >>> from django.http import HttpResponse
> >>> r = HttpResponse(iter(['a', 'b']))
> >>> r.content
> 'ab'
> >>> r.content
> ''
> >>> r2 = HttpResponse(iter(['a', 'b']))
> >>> print r2.content
> ab
> >>> print r2.content
> >>> r3 = HttpResponse(iter(['a', 'b']))
> >>> r3.content == r3.content
>
> False
>
> 2. Some middleware prematurely consume generator content by accessing
> `HttpResponse.content`, which can use a lot of memory and cause browser
> timeouts when attempting to stream large amounts of data or
> slow-to-generate data.
>
> There have been several tickets [2] [3] [4] and django-developers
> discussions [5] [6] [7] [8] about these issues.
>
> SOME USE CASES FOR STREAMING RESPONSES:
>
> A. Generating and exporting CSV data directly from the database.
>
> B. Restricting file access to authenticated users for files that may be
> hosted on external servers.
>
> C. Drip-feeding chunks of content to prevent timeout when requesting a page
> that takes a long time to generate.
>
> OPTION 1:
>
> Remove support for "streaming" responses. If an iterator is passed in as
> content to `HttpResponse`, consume it in `HttpResponse.__init__()` to
> eliminate buggy behaviour. Middleware won't have to worry about what type
> of content a response has.
>
> Now that Jacob has accepted #7581 and said that it is clear we need to do
> *something*, I hope we can rule out this option.
>
> OPTION 2:
>
> Make `HttpResponse.__init__()` consume any iterator content, and add an
> `HttpResponseStreaming` class or an `HttpResponse(streaming=False)`
> argument. Allow middleware to check via `hasattr()` or `isinstance()`
> whether or not the response has generator content, and conditionally skip
> code that is incompatible with streaming responses.
>
> Some middleware will have to be updated for compatibility with streaming
> responses, and any 3rd party middleware that prematurely consumes generator
> content will continue to work, only without the bugs (and potentially with
> increased memory usage and browser timeouts).
>
> OPTION 3:
>
> Build a capabilities API for `HttpResponse` objects, and have middleware
> inspect responses to determine "can I read content?", "can I replace
> content?", "can I change etag?", etc. This will likely become a bigger and
> more complicated design decision as we work out what capabilities we want
> to support. Some have argued that it should be sufficient to know if we
> have generator content or not, for all the cases that people have reported
> so far.
>
> OPTION 4:
>
> Provide a way for developers to specify on an `HttpResponse` object or
> subclass that specific middleware should be skipped for that response. This
> would be problematic because 3rd party views won't know what other
> middleware is installed in a project in order to name them for exclusion.
>
> OPTION 5:
>
> Add Yet Another Setting that would allow developers to define
> `CONDITIONAL_MIDDLEWARE_CLASSES` at a project level. At the project level,
> developers would know which middleware classes they are using, and when
> they should be executed or skipped. This would give very fine grained
> control at a project level to match middleware conditionally with
> `HttpResponse` subclasses, without requiring any changes to existing or 3rd
> party middleware. This could look something like this:
>
> MIDDLEWARE_CLASSES = (
>     'django.middleware.common',
> )
>
> CONDITIONAL_MIDDLEWARE_CLASSES = {
>     'exclude': {
>         'django.http.HttpResponseStreaming': ['django.middleware.common',
> 'otherapp.othermiddleware', ...],
>     },
>     'include': {
>         'myapp.MyHttpResponse': ['myapp.mymiddleware', ...],
>     },
>
> }

My initial feeling is that anything accessing "large content" in
Python is broken and we should not try to make sure these things
continue to work. The problem is how we define if the content is too
large to access. Assuming any generator passed to the __init__ is
large maybe too drastic.

We should either remove the content attribute altogether from
generator based responses, or at least raise an error if the content
is accessed a second time. The current behaviour is definitely broken,
and we should not

Streaming HttpResponse revisted. Any core devs, please take a look if you can :)

2012-08-20 Thread Tai Lee
I'd like to re-visit the discussion surrounding #7581 [1], a ticket about 
streaming responses that is getting quite long in the tooth now, which 
Jacob finally "accepted" 11 months ago (after a long time as DDN) and said 
that it is clear we have to do *something*, but *what* remains to be seen.

I'd like to try provide a little refresher and summarise the options that 
have been suggested, and ask any core devs to please weigh in with their 
preference so that I can work up a new patch that will be more likely to 
gain acceptance.


THE PROBLEM:

1. There are bugs and surprising behaviour that arise when creating an 
HttpResponse with a generator as its content, as a result of "quantum 
state" of `HttpResponse.content` (measuring it changes it).

>>> from django.http import HttpResponse
>>> r = HttpResponse(iter(['a', 'b'])) 
>>> r.content 
'ab'
>>> r.content 
''
>>> r2 = HttpResponse(iter(['a', 'b'])) 
>>> print r2.content
ab
>>> print r2.content

>>> r3 = HttpResponse(iter(['a', 'b']))
>>> r3.content == r3.content
False

2. Some middleware prematurely consume generator content by accessing 
`HttpResponse.content`, which can use a lot of memory and cause browser 
timeouts when attempting to stream large amounts of data or 
slow-to-generate data.

There have been several tickets [2] [3] [4] and django-developers 
discussions [5] [6] [7] [8] about these issues.


SOME USE CASES FOR STREAMING RESPONSES:

A. Generating and exporting CSV data directly from the database.

B. Restricting file access to authenticated users for files that may be 
hosted on external servers.

C. Drip-feeding chunks of content to prevent timeout when requesting a page 
that takes a long time to generate.


OPTION 1:

Remove support for "streaming" responses. If an iterator is passed in as 
content to `HttpResponse`, consume it in `HttpResponse.__init__()` to 
eliminate buggy behaviour. Middleware won't have to worry about what type 
of content a response has.

Now that Jacob has accepted #7581 and said that it is clear we need to do 
*something*, I hope we can rule out this option.


OPTION 2:

Make `HttpResponse.__init__()` consume any iterator content, and add an 
`HttpResponseStreaming` class or an `HttpResponse(streaming=False)` 
argument. Allow middleware to check via `hasattr()` or `isinstance()` 
whether or not the response has generator content, and conditionally skip 
code that is incompatible with streaming responses.

Some middleware will have to be updated for compatibility with streaming 
responses, and any 3rd party middleware that prematurely consumes generator 
content will continue to work, only without the bugs (and potentially with 
increased memory usage and browser timeouts).


OPTION 3:

Build a capabilities API for `HttpResponse` objects, and have middleware 
inspect responses to determine "can I read content?", "can I replace 
content?", "can I change etag?", etc. This will likely become a bigger and 
more complicated design decision as we work out what capabilities we want 
to support. Some have argued that it should be sufficient to know if we 
have generator content or not, for all the cases that people have reported 
so far.


OPTION 4:

Provide a way for developers to specify on an `HttpResponse` object or 
subclass that specific middleware should be skipped for that response. This 
would be problematic because 3rd party views won't know what other 
middleware is installed in a project in order to name them for exclusion.


OPTION 5:

Add Yet Another Setting that would allow developers to define 
`CONDITIONAL_MIDDLEWARE_CLASSES` at a project level. At the project level, 
developers would know which middleware classes they are using, and when 
they should be executed or skipped. This would give very fine grained 
control at a project level to match middleware conditionally with 
`HttpResponse` subclasses, without requiring any changes to existing or 3rd 
party middleware. This could look something like this:

MIDDLEWARE_CLASSES = (
'django.middleware.common',
)

CONDITIONAL_MIDDLEWARE_CLASSES = {
'exclude': {
'django.http.HttpResponseStreaming': ['django.middleware.common', 
'otherapp.othermiddleware', ...],
},
'include': {
'myapp.MyHttpResponse': ['myapp.mymiddleware', ...],
},
}


MY TAKE:

I think that option 1 and option 4 are non-starters.

I think option 3 is perhaps a little overkill and will be more difficult to 
get committed once we start thinking about what capabilities we want to 
support.

I think option 2 is probably going to be the easiest solution. It's 
practically implemented and up-to-date already (missing docs and tests).

Although it does involve Yet Another Setting, I think option 5 provides the 
most flexibility, where it is most needed. It gives developers working at 
the project level a way to override and conditionally skip or execute 3rd 
party middleware without having to make any changes to 3rd party middleware.

I would be happy with eith

Re: GSoC Check-in: Security Enhancements

2012-08-20 Thread Rohan Jain
Hi,

Today is the 'pencils down' date for this GSoC project. Past 4 months
have been a great learning experience from me. Just being in the
context of security side of the web has been really beneficial. Moving
around in a very well written code base is also delightful.
Meanwhile, I did get to work on multiple sections of the code base.
But compared to the expectations at the initial phase of the program,
I myself am a little disappointed about the output I was able to
generate out of the time - specially regarding CSRF.
This can be attributed to my being unable to give consistent attention
to the project through the duration and to solicit feedback. Also, the
project comprised of multiple discreet tasks, so not enough
consideration was there for each task.

This is a overview of various complete/incomplete topics I worked on.

Centralized Tokenization:
This was about integrating a central tokenization api throughout the
django project. I tracked this at [centralized-tokenization][0] branch
of my fork on github. This is near completion, and also has a
documentation[1] of the API with basic usage examples.

Improvements to sessions:
My first task, related to some improvements to the sessions framework.
Includes signing and session cleanup based on the engine. Tracked at
[sessions-improvements][2] branch.

CSRF Enhancements:
It occupied most of my time through the project. I tried multiple ways
to solve the issues under this. Tracked at [csrf-enhancements][3].
First up, I started with using the origin header for doing a CSRF
check. One implementation with some tests is at
[csrf-origin-checking][4]. Recently I started on the idea of
implementing moduler checkers for each kind of CSRF check, but
haven't got anything useful out of it yet. While progressing, it
seemed like I was virtually writing a middleware per checker, so now I
have moved on to attempt on CSRF cookie store. Basically something
which should seamlessly switch Session/Cookie for storage based on the
availability of session.

I'll try to hang around the django bug tracker and submit some
patches.

--
Thanks
Rohan Jain

[0]: https://github.com/crodjer/django/tree/centralized-tokenization
[1]: 
https://github.com/crodjer/django/blob/centralized-tokenization/docs/topics/tokenization.txt
[2]: https://github.com/crodjer/django/tree/sessions-improvements
[3]: https://github.com/crodjer/django/tree/csrf-enhancements
[4]: https://github.com/crodjer/django/tree/csrf-origin-checking
[5]: https://github.com/crodjer/django/tree/csrf-moduler-checkers

On 01:57 +0530 /  7 Aug, Rohan Jain wrote:
> Hi,
> 
> Sorry for the delay in getting back. I was meanwhile working on
> centralized tokenization for few days, while still trying to figure
> something better for CSRF.
> 
> On 03:52 -0400 / 25 Jul, Alex Ogier wrote:
> > On Tue, Jul 24, 2012 at 11:37 PM, Rohan Jain  wrote:
> > >
> > > I had one more idea, "Pluggable CSRF checkers".
> > >
> > > Currently, the CSRF middleware has two kinds of checks, referer (for
> > > https) and secret validation token (common). These with origin header
> > > based checker (if we add it) come in conditional blocks, making
> > > switching them difficult. So what I propose to do is decouple their
> > > logic from CSRF middleware and each of them provide a checker. It goes
> > > like this:
> > >
> > > A setting for configuring global CSRF checkers:
> > >
> > > CSRF_CHECKERS = {
> > > 'django.middleware.csrf.checkers.OriginChecker',
> > > # This one can be strict for https and lax for http
> > > 'django.middleware.csrf.checkers.RefererChecker',
> > > # contrib.sessions could provide a csrf checker maintained
> > > # with sessions. This stores the token in session data.
> > > 'django.contrib.sessions.csrf_checkers.SessionChecker'
> > > }
> > >
> > 
> > I don't think this is a good idea. If you enumerate security features
> > in settings.py, then later additions won't be picked up by default. If
> > Django add a new CSRF checking mechanism, we want everybody to take
> > advantage of it with no modifications.
> > 
> > Ordinarily I agree with you, explicit is better than implicit.
> > However, in the case of security features, I think this is inverted:
> > Django sites should be implicitly enrolled in all security mechanisms
> > if possible, and should be able to explicitly opt out if necessary.
> > Almost everyone should be using every single protection Django offers
> > on all their requests, and therefore it should be verbose and
> > discouraged to turn off these protections.
> >
> 
> Hmm, that is a valid point. I can drop the configurable CSRF settings.
> But still a modular CSRF checkers might be useful, in which the
> checkers are selected dynamically. When sessions app is present, use
> sessions checker instead of cookies based CSRF token store. Also we
> can have switches which incorporate existing behaviour based on
> https/http connection and also origin header checking based

Re: #16455 PostGIS 2.0 support

2012-08-20 Thread Anssi Kääriäinen
On 20 elo, 13:27, Anders Steinlein  wrote:
> Any chance of having this patch backported to 1.4? Or is this a 1.5-only
> thing?

New features do not get backpatched.

However, the changes to actual code are around 5 lines. The rest is
documentation changes. It should be fairly trivial to backpatch this
yourself if you absolutely need PostGIS 2.0 support before Django 1.5.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Re: #16455 PostGIS 2.0 support

2012-08-20 Thread Anders Steinlein
On Mon, Aug 20, 2012 at 11:58 AM, Anssi Kääriäinen
wrote:

> On 13 elo, 21:25, Anssi Kääriäinen  wrote:
> > On 10 elo, 20:12, Jeremy Dunck  wrote:
> >
> > I can take care of the patch. As said earlier I don't understand GIS
> > well, but based on the discussions I have no doubt about the patch.
> > So, I will try to get this committed soon.
>
> I have updated the ticket with some comments. The docs of the previous
> patch weren't ready for commit so I had to do an update to that part
> of the patch. In addition I do get some errors from the gis tests (see
> the ticket for details), but I am not sure if the errors are caused by
> updated PostGIS & dependencies, or if the errors are unrelated to the
> PostGIS 2.0 work.
>
> In any case the patch needs another review before commit.
>

Any chance of having this patch backported to 1.4? Or is this a 1.5-only
thing?

/anders

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



Re: #16455 PostGIS 2.0 support

2012-08-20 Thread Anssi Kääriäinen
On 13 elo, 21:25, Anssi Kääriäinen  wrote:
> On 10 elo, 20:12, Jeremy Dunck  wrote:
>
> > That sounds good to me, though I'm not a committer on the tree as far
> > as I know.   Anssi?
>
> I can take care of the patch. As said earlier I don't understand GIS
> well, but based on the discussions I have no doubt about the patch.
> So, I will try to get this committed soon.

I have updated the ticket with some comments. The docs of the previous
patch weren't ready for commit so I had to do an update to that part
of the patch. In addition I do get some errors from the gis tests (see
the ticket for details), but I am not sure if the errors are caused by
updated PostGIS & dependencies, or if the errors are unrelated to the
PostGIS 2.0 work.

In any case the patch needs another review before commit.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.