[issue2632] performance problem in socket._fileobject.read

[Gregory P. Smith]

Gregory P. Smith <[EMAIL PROTECTED]> added the comment:

committed to release25-maint in r64754.

--
resolution:  -> fixed
status: open -> closed

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Benjamin Peterson]

Benjamin Peterson <[EMAIL PROTECTED]> added the comment:

Well, it's a release blocker for 2.5.3.

--
nosy: +benjamin.peterson

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

If it's fixed in 2.6 and 3.0, it shouldn't be release blocker anymore,
now should it?

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Ralf Schmitt]

Ralf Schmitt <[EMAIL PROTECTED]> added the comment:

Can we get the fix for release25-maint? It will not get worse than the
current state is.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Gregory P. Smith]

Changes by Gregory P. Smith <[EMAIL PROTECTED]>:


--
versions:  -Python 2.6

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Gregory P. Smith]

Gregory P. Smith <[EMAIL PROTECTED]> added the comment:

I'm going to let the committed change bake in Python trunk for a beta
release or two before backporting it to release25-maint.

__
Tracker <[EMAIL PROTECTED]>

__
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Georg Brandl]

Changes by Georg Brandl <[EMAIL PROTECTED]>:


--
priority: critical -> release blocker

__
Tracker <[EMAIL PROTECTED]>

__
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Barry A. Warsaw]

Barry A. Warsaw <[EMAIL PROTECTED]> added the comment:

Bumping this down to critical for the alpha release.  It should be
release blocker for the first beta.

--
nosy: +barry
priority: release blocker -> critical

__
Tracker <[EMAIL PROTECTED]>

__
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Ralf Schmitt]

Ralf Schmitt <[EMAIL PROTECTED]> added the comment:

the above easy_install now works for me. thanks.

__
Tracker <[EMAIL PROTECTED]>

__
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Gregory P. Smith]

Gregory P. Smith <[EMAIL PROTECTED]> added the comment:

The bug introduced in r62627 has been fixed in r62744.

__
Tracker <[EMAIL PROTECTED]>

__
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Gregory P. Smith]

Gregory P. Smith <[EMAIL PROTECTED]> added the comment:

I'm glad i put that assert in...  The problem occurs due to a mixture of
fixed size reads followed by unbounded readlines on an unbuffered
_fileobject.  A case that the code currently doesn't handle.  I'm fixing it.

__
Tracker <[EMAIL PROTECTED]>

__
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Raghuram Devarakonda]

Raghuram Devarakonda <[EMAIL PROTECTED]> added the comment:

Similar problem is reported in #2760.

__
Tracker <[EMAIL PROTECTED]>

__
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Ralf Schmitt]

Ralf Schmitt <[EMAIL PROTECTED]> added the comment:

I have a case which triggers the assert in readline:

  File "build/bdist.linux-x86_64/egg/setuptools/command/develop.py",
line 27, in run
  File "build/bdist.linux-x86_64/egg/setuptools/command/develop.py",
line 102, in install_for_development
  File
"build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line
519, in process_distribution
  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 522, in resolve
  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 758, in
best_match
  File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 770, in obtain
  File
"build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py", line
433, in easy_install
  File "build/bdist.linux-x86_64/egg/setuptools/package_index.py", line
462, in fetch_distribution
  File "build/bdist.linux-x86_64/egg/setuptools/package_index.py", line
303, in find_packages
  File "build/bdist.linux-x86_64/egg/setuptools/package_index.py", line
617, in scan_url
  File "build/bdist.linux-x86_64/egg/setuptools/package_index.py", line
198, in process_url
  File "/home/ralf/py26/lib/python2.6/socket.py", line 326, in read
data = self._sock.recv(rbufsize)
  File "/home/ralf/py26/lib/python2.6/httplib.py", line 512, in read
return self._read_chunked(amt)
  File "/home/ralf/py26/lib/python2.6/httplib.py", line 548, in
_read_chunked
line = self.fp.readline()
  File "/home/ralf/py26/lib/python2.6/socket.py", line 391, in readline
assert buf.tell() == 0
AssertionError

this happens when I run :

easy_install 'mwlib<0.6'

__
Tracker <[EMAIL PROTECTED]>

__
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Gregory P. Smith]

Changes by Gregory P. Smith <[EMAIL PROTECTED]>:


--
priority: critical -> release blocker

__
Tracker <[EMAIL PROTECTED]>

__
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Gregory P. Smith]

Gregory P. Smith <[EMAIL PROTECTED]> added the comment:

I fixed the min vs max use that Ralf noted and have submitted this as
r62627.  all tests pass for me but I believe it deserves a wider
audience and testing outside of just the test suite.

could those who reported the original problems (both the memory use
issue and the speed issue) run their tests with this applied?

... watching the buildbots ...

 http://codereview.appspot.com/621/diff/61/81

__
Tracker <[EMAIL PROTECTED]>

__
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Matthias Klose]

Changes by Matthias Klose <[EMAIL PROTECTED]>:


--
nosy: +doko

__
Tracker <[EMAIL PROTECTED]>

__
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Gregory P. Smith]

Gregory P. Smith <[EMAIL PROTECTED]> added the comment:

available for an easy side by side review here:

 http://codereview.appspot.com/212

Also, yes I think you're right Ralf.  With these changes I should be
able to return that to a max() within the while True: for sized reads
and things will work as desired.

__
Tracker <[EMAIL PROTECTED]>

__
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Ralf Schmitt]

Ralf Schmitt <[EMAIL PROTECTED]> added the comment:

(I think the twisted issue your talking about is:
http://twistedmatrix.com/trac/ticket/1079)

Your patch still contains this code:

 recv_size = min(rbufsize, left)
 data = self._sock.recv(recv_size)

This is IMHO wrong.  See my comments above.

__
Tracker <[EMAIL PROTECTED]>

__
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Forest Wilkinson]

Changes by Forest Wilkinson <[EMAIL PROTECTED]>:


--
nosy: +forest

__
Tracker <[EMAIL PROTECTED]>

__
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Gregory P. Smith]

Gregory P. Smith <[EMAIL PROTECTED]> added the comment:

Twisted fixed their problem for issue 1092502 by making recv()ed data
short lived by putting it into a StringIO for buffering.

I've attached a patch that does that for the socket module -and- gets
rid of any possibility of doing tiny size recvs in read() which is the
cause of the poor performance that this issue was filed about.

My patch also documents what _rbufsize is used for (readline) and makes
sane use of it with read.

When viewing the change, it is best to view it side by side next to the
old version rather than trying to comprehend it as a diff.

I believe this really fixes both issue 1092502 while at the same time
prevents introduction of a speed problem (this issue).

--
assignee:  -> gregory.p.smith
nosy: +gregory.p.smith
Added file: http://bugs.python.org/file10071/socket-strio-gps01.patch.txt

__
Tracker <[EMAIL PROTECTED]>

__
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Ralf Schmitt]

Ralf Schmitt <[EMAIL PROTECTED]> added the comment:

However it should be fixed in release25-maint. Can anyone install curt's
patch? It does what the original fix intended to do.

__
Tracker <[EMAIL PROTECTED]>

__
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

As A.M. Kuchling said in the other bug :), there is no need to fix 2.5.2
as the offending change is posterior to the release.
By the way, somewhat really ought to close either this bug or #2601,
discussing things in parallel in two different bug entries is too confusing.

__
Tracker <[EMAIL PROTECTED]>

__
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2632] performance problem in socket._fileobject.read

[Curt Hagenlocher]

Curt Hagenlocher <[EMAIL PROTECTED]> added the comment:

At least in 2.5.2, there appear to be only two places in the standard 
library where an arbitrary and unchecked size is passed directly to 
_socketobject.recv.  _fileobject.read is one such place and the other 
is in logging/config.py.  In both cases, the pattern is something like

while len(data) < size:
data = data + sock.recv(size - len(data))

Of course, the same pattern may exist in any number of instances of 
user code that performs a socket.recv, and all of these would be 
subject to the same memory thrashing problem that originally beset 
imaplib.

As such, I now agree with Ralf that the fix ultimately belongs in 
_socketobject.  However, deploying a fix for 2.5.2 is probably easier 
if it's in socket.py than if it requires a recompile.

--
title: socket._fileobject.read(n) should ignore _rbufsize when 1 -> performance 
problem in socket._fileobject.read

__
Tracker <[EMAIL PROTECTED]>

__
___
Python-bugs-list mailing list 
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com