[issue7464] circular reference in HTTPResponse by urllib2

[Irit Katriel]

Irit Katriel  added the comment:

> It doesn't appear to be an issue in py3k.

I agree, I can't find this code anywhere. Closing.

--
nosy: +iritkatriel
resolution:  -> out of date
stage: test needed -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7464] circular reference in HTTPResponse by urllib2

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

No, the socket is actually closed when response's close() method is called.  
The problem is that the HTTPResponse object, buried deep within the nested 
classes returned from do_open(), has a circular reference, and _it_ will not go 
away.

No one is _relying_ on garbage collection in the sense that this is not, I 
think, designed behaviour, merely an unintentional effect of storing a bound 
method in the object inance.

As always, circular reference should be avoided when possible since relying on 
gc is not something to be done lightly.

Now, I think that changing the complicated wrapping at this stage is not 
possible, but merely replacing the bound method with a weak method might just 
do the trick.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7464] circular reference in HTTPResponse by urllib2

[Martin Panter]

Martin Panter added the comment:

Sounds like urlopen() is relying on garbage collection to close the socket and 
connection. Maybe it would be better to explicitly close the socket, even if 
you do eliminate all the garbage reference cycles.

My test code for Issue 19524 might be useful here. It verifies close() has been 
called on the HTTP socket.

--
nosy: +vadmium

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7464] circular reference in HTTPResponse by urllib2

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7464] circular reference in HTTPResponse by urllib2

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

Here it is.
Notice the incredible nesting depth in python 2.7.
The socket itself is found at
response.fp._sock.fp._sock
There are two socket._fileobjects in use!

--
Added file: http://bugs.python.org/file33205/httpleak.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7464] circular reference in HTTPResponse by urllib2

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Could you please provide some tests?

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7464] circular reference in HTTPResponse by urllib2

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

This is still a horrible, horrible, cludge.
I've recently done some work in this area and will suggest a different approach.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7464] circular reference in HTTPResponse by urllib2

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
stage: needs patch -> test needed
versions:  -Python 3.1, Python 3.2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7464] circular reference in HTTPResponse by urllib2

[Kristján Valur Jónsson]

Kristján Valur Jónsson  added the comment:

in python/trunk/Lib/urllib2.py, line 1161

It doesn't appear to be an issue in py3k.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7464] circular reference in HTTPResponse by urllib2

[David Stanek]

David Stanek  added the comment:

Does this issue still exist? I did a little poking around at could not find the 
quoted code.

--
nosy: +dstanek

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7464] circular reference in HTTPResponse by urllib2

[Senthil Kumaran]

Senthil Kumaran  added the comment:

Lets just investigate the circular reference part here for this ticket.

--
assignee:  -> orsenthil
versions: +Python 2.7, Python 3.1, Python 3.2 -Python 2.5, Python 2.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7464] circular reference in HTTPResponse by urllib2

[Senthil Kumaran]

Senthil Kumaran  added the comment:

On Fri, Dec 11, 2009 at 09:40:40AM +, Kristján Valur Jónsson wrote:
> Is there any chance of getting a weakmethod class into the weakref 
> module?

This is a separate feature request on weakref module. It may opened
and discussion carried out there?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7464] circular reference in HTTPResponse by urllib2

[Kristján Valur Jónsson]

Kristján Valur Jónsson  added the comment:

cool.  The mindtrove one in particular seems nice.  I didn't realize 
that one could build boundobjects oneself.
Is there any chance of getting a weakmethod class into the weakref 
module?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7464] circular reference in HTTPResponse by urllib2

[Senthil Kumaran]

Senthil Kumaran  added the comment:

weak method idea seems interesting. I have not used it anytime yet, and
in this case,it seems okay.

--
nosy: +orsenthil

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7464] circular reference in HTTPResponse by urllib2

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc  added the comment:

The WeakMethod idea is not new:

http://code.activestate.com/recipes/81253/

http://mindtrove.info/articles/python-weak-references/#toc-extending-weak-
references

--
nosy: +amaury.forgeotdarc

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7464] circular reference in HTTPResponse by urllib2

[Kristján Valur Jónsson]

Kristján Valur Jónsson  added the comment:

I have two solutions for this problem.  The first is a mundane one, and 
what I employed in our production environment:
class RecvAdapter(object):
def __init__(self, wrapped):
self.wrapped = wrapped
def recv(self, amt):
return self.wrapped.read(amt)
def close(self):
self.wrapped.close()

...
fp = socket._fileobject(RecvAdapter(r), close=True)


The second solution is a bit more interesting.  It involves applying 
what I call a weakmethod: A bound method that holds a weak ref to the 
object instance:

import weakref
class WeakMethod(object):
def __init__(self, bound):
self.weakself = weakref.proxy(bound.im_self)
self.methodname = bound.im_func.func_name
def __call__(self, *args, **kw):
return getattr(self.weakself, self.methodname)(*args, **kw)

We then do:
r.recv = WeakMethod(r.read)
fp = socket._fileobject(r, close=True)

I've had many uses for a WeakMethod through the years.  I wonder if such 
a class might be considered useful enough to be put into the weakref 
module.

--
keywords: +patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7464] circular reference in HTTPResponse by urllib2

[Kristján Valur Jónsson]

New submission from Kristján Valur Jónsson :

in urllib2, you will find these lines:
# Wrap the HTTPResponse object in socket's file object adapter
# for Windows.  That adapter calls recv(), so delegate recv()
# to read().  This weird wrapping allows the returned object to
# have readline() and readlines() methods.

# XXX It might be better to extract the read buffering code
# out of socket._fileobject() and into a base class.

r.recv = r.read
fp = socket._fileobject(r, close=True)

This, storing a bound method in the instance, will cause a reference 
cycle that the user knows nothing about.

I propose creating a wrapper instance with a recv() method instead.  Or, 
is there a standard way of storing bound methods on instances?  A 
'weakmethod', perhaps?

--
components: Library (Lib)
messages: 96175
nosy: krisvale
severity: normal
stage: needs patch
status: open
title: circular reference in HTTPResponse by urllib2
type: resource usage
versions: Python 2.5, Python 2.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com