[Repoze-dev] [issue179] respoze.profile doesn't handle WSGI iterables properly.

[Chris McDonough]

Chris McDonough  added the comment:

Grr.  Let's try that URL again.



__
Repoze Bugs 

__
___
Repoze-dev mailing list
Repoze-dev@lists.repoze.org
http://lists.repoze.org/listinfo/repoze-dev

[Repoze-dev] [issue179] respoze.profile doesn't handle WSGI iterables properly.

[Chris McDonough]

Chris McDonough  added the comment:

https://github.com/repoze/repoze.profile/commit/8fa63f072de5b14a6b8e4ccf5f78e98
a62d6cfdc has the change.  Does that work for you?

__
Repoze Bugs 

__
___
Repoze-dev mailing list
Repoze-dev@lists.repoze.org
http://lists.repoze.org/listinfo/repoze-dev

[Repoze-dev] [issue179] respoze.profile doesn't handle WSGI iterables properly.

[Chris McDonough]

Chris McDonough  added the comment:

That's a good idea.  I'll just take out the generator check entirely, and put 
in a knob.

__
Repoze Bugs 

__
___
Repoze-dev mailing list
Repoze-dev@lists.repoze.org
http://lists.repoze.org/listinfo/repoze-dev

[Repoze-dev] [issue179] respoze.profile doesn't handle WSGI iterables properly.

[Graham Dumpleton]

Graham Dumpleton  added the comment:

Or have it check a WSGI environ flag to modify behaviour as to whether consume 
flatten iterable/generator. I would rather not have to modify code every time 
if you 
are going to make it so doesn't track iterable/generator and close() call. I 
can leave 
with a configuration option.

__
Repoze Bugs 

__
___
Repoze-dev mailing list
Repoze-dev@lists.repoze.org
http://lists.repoze.org/listinfo/repoze-dev

[Repoze-dev] [issue179] respoze.profile doesn't handle WSGI iterables properly.

[Chris McDonough]

Chris McDonough  added the comment:

The case this that eagerly unwinding the app_iter is trying to defend against 
is (as 
you probably know) this one:

def someapp(environ, start_response):
 start_response('200 OK', [... headers ...])
 for item in listofbytes:
 yield item

The pattern of calling start_response during the first iteration is, IMO, 
insane and 
supporting is is only a courtesty/kindness.  Unwinding every iterator from 
every 
arbitrary downstream applications just in case someone does this, in my 
estimation at least, will cause more problems (resource exhaustion) for people 
who never do this more than not doing it will cause problems for people that 
use 
this pattern.   You're right that someone could return a generator that streams 
and 
exhaust RAM that way but if it's consistency we're after, I'll just rip it out 
instead.

__
Repoze Bugs 

__
___
Repoze-dev mailing list
Repoze-dev@lists.repoze.org
http://lists.repoze.org/listinfo/repoze-dev

[Repoze-dev] [issue179] respoze.profile doesn't handle WSGI iterables properly.

[Graham Dumpleton]

Graham Dumpleton  added the comment:

I would say that using:

def __iter__(self):
for item in self.generator:
yield item

would be a fairly common recipe for implementing WSGI middleware wrappers 
wouldn't it. The generator which is wrapped by this could just as well be 
calling 
start_response() on first call.

Couldn't a generator just as easily be streaming data as well and exhaust all 
memory? Not sure I see the difference really.

__
Repoze Bugs 

__
___
Repoze-dev mailing list
Repoze-dev@lists.repoze.org
http://lists.repoze.org/listinfo/repoze-dev

[Repoze-dev] [issue179] respoze.profile doesn't handle WSGI iterables properly.

[Chris McDonough]

Chris McDonough  added the comment:

Hi Graham,

The intent here is to only unwind generators because they may be calling 
start_response() as a side effect of the first iteration.  While other 
iterables might 
do that, I think they can lose, because this case is fairly uncommon, and the 
only 
side effect of not doing it is that profiling data is skewed.

Unwinding all iterables would mean running out of RAM when you unwound an 
app_iter that happened to serve up e.g. a big file, which I think would be 
worse 
than providing skewed profile data.

Thoughts?

--
status: unread -> chatting

__
Repoze Bugs 

__
___
Repoze-dev mailing list
Repoze-dev@lists.repoze.org
http://lists.repoze.org/listinfo/repoze-dev