[issue3766] socket.socket.recv broken (unbearably slow)
Konstantin Osipov kostja.osi...@gmail.com added the comment: I was able to observe the same issue: I'm using Python 2.6.5 on Ubuntu 10.0.4 LTS. My system is 64 bit Intel Core I7, Quad core, Linux kernel 2.6.32-generic x86_64, Ubuntu EGLIBC 2.11.1-0ubuntu7.5. A simple client TCP socket, which sends 35 bytes over to a server on localhost and receives 20 bytes in response, produces only 22 RPS. An identical application written in C gives me 7000 RPS. TCP_NODELAY is on on the server side. Turning on TCP_NODELAY on the client gives me ~500 RPS in Python (which I'm satisfied with, 'cause I think I then hit other bottlenecks). Still, such low performance on by default can be surprising and hard to debug. -- nosy: +kostja ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3766 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3766] socket.socket.recv broken (unbearably slow)
Sam samsli...@gmail.com added the comment:
I know this bug is closed, but I too am experiencing it under Linux
2.6.24-22 and Python 2.5.2.
I'm using urllib2, and it's just spending an obscene amount of cpu time
in {method 'recv' of '_socket.socket' objects}
Anyone have any ideas?
Would switching to httplib2 help?
--
nosy: +samslists
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3766
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3766] socket.socket.recv broken (unbearably slow)
Antoine Pitrou [EMAIL PROTECTED] added the comment: Yes, obviously. Although adding it to the client socket did make no difference after I had already done so for the server. Still communication is too slow by orders of magnitude. (Sorry for pointing this out again) Well, if this precise use case is really important for you, I suggest using Wireshark (or any other packet analyzer) to see what happens in terms of latency between packets over the wire. Something else: try replacing localhost with 127.0.0.1, perhaps your DNS resolution eats a lot of time. I would greatly appreciate any help on the subject. How do *BSD sockets differ from Linux sockets and what do I do to make things faster. I don't know, but I suspect the difference is more in the TCP stack implementation than in the sockets layer. In any case, I'm gonna close this bug, as this is very likely not a Python problem. Please ask further questions on comp.lang.python, lots of people there may help you :) ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3766 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3766] socket.socket.recv broken (unbearably slow)
Changes by Antoine Pitrou [EMAIL PROTECTED]: -- resolution: - invalid status: open - closed ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3766 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3766] socket.socket.recv broken (unbearably slow)
Thorben [EMAIL PROTECTED] added the comment:
That's more like it. Now I'd like to see the same behavior on Linux...
2008/9/12 Gabriel Genellina [EMAIL PROTECTED]:
Gabriel Genellina [EMAIL PROTECTED] added the comment:
I've tested it on Windows XP. MSG_WAITALL is not supported, but I
replaced it using a while loop. I didn't notice any extraneous delay.
500 packets @ 2 tokens each (500 very short lists)
0.140999794006
16008 function calls in 0.146 CPU seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
15000.0360.0000.0360.000 {method 'recv' of
'_socket.socket'
objects}
15000.0330.0000.0330.000 string:1(sendall)
15000.0160.0000.0650.000 Client.py:15(read_int)
15000.0150.0000.0530.000 Client.py:7(send_int)
5000.0090.0000.0770.000
Client.py:22(read_int_list)
5000.0070.0000.0600.000
Client.py:10(send_int_list)
15000.0070.0000.0100.000 struct.py:77(unpack)
15000.0050.0000.0050.000 struct.py:54(pack)
5000.0040.0000.1410.000 Client.py:31(spam)
20010.0040.0000.0040.000 {len}
10.0030.0030.1460.146 runme.py:11(bench)
15000.0030.0000.0030.000 {method 'unpack' of
'Struct' objec
ts}
10010.0030.0000.0030.000 {range}
10000.0020.0000.0020.000 {method 'append' of
'list' objects
}
10.0000.0000.0000.000 struct.py:35(_compile)
20.0000.0000.0000.000 {time.time}
10.0000.0000.1460.146 string:1(module)
10.0000.0000.0000.000 {method 'disable' of
'_lsprof.Prof
iler' objects}
None
1 packet @ 5 tokens (1 very long list)
4.8913242
450019 function calls in 4.893 CPU seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
500011.2120.0001.2120.000 string:1(sendall)
500011.0620.0001.0620.000 {method 'recv' of
'_socket.socket'
objects}
500010.5940.0002.2820.000 Client.py:15(read_int)
500010.5170.0001.9820.000 Client.py:7(send_int)
10.3540.3542.7322.732
Client.py:22(read_int_list)
500010.3350.0000.5240.000 struct.py:77(unpack)
500010.2530.0000.2530.000 struct.py:54(pack)
500010.1890.0000.1890.000 {method 'unpack' of
'Struct' objec
ts}
10.1760.1762.1582.158
Client.py:10(send_int_list)
500020.1020.0000.1020.000 {len}
50.0970.0000.0970.000 {method 'append' of
'list' objects
}
20.0020.0010.0020.001 {range}
10.0020.0024.8934.893 runme.py:19(bench2)
10.0000.0004.8904.890 Client.py:31(spam)
20.0000.0000.0000.000 {time.time}
10.0000.0004.8934.893 string:1(module)
10.0000.0000.0000.000 {method 'disable' of
'_lsprof.Prof
iler' objects}
--
nosy: +gagenellina
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3766
___
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3766
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3766] socket.socket.recv broken (unbearably slow)
Changes by Giampaolo Rodola' [EMAIL PROTECTED]: -- nosy: +giampaolo.rodola ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3766 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3766] socket.socket.recv broken (unbearably slow)
Antoine Pitrou [EMAIL PROTECTED] added the comment: Thorben, is the problem still there if you use fork() rather than launching a separate thread for the server? The implementation of the recv() method is straightforward and I don't see anything that could cause a huge latency there, it's just a simple wrapper over the C library's recv() function. The waiting certainly occurs inside the OS rather than inside the Python interpreter. -- nosy: +pitrou ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3766 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3766] socket.socket.recv broken (unbearably slow)
Thorben [EMAIL PROTECTED] added the comment: The problem exists even if the server part is replaced by a server written in C. I only wrote up the dummy server in python to be able to provide a testcase. The C server works with reasonable speed when connected to a client written in perl by the way. My employer is quite disappointed with Python's performance... (He provided the profiler data for the Mac by the way) I almost wish that I did something wrong, but this does not seem to be the case. Nevertheless, I will try out your suggestion. Thanks for replying, Thorben 2008/9/11 Antoine Pitrou [EMAIL PROTECTED]: Antoine Pitrou [EMAIL PROTECTED] added the comment: Thorben, is the problem still there if you use fork() rather than launching a separate thread for the server? The implementation of the recv() method is straightforward and I don't see anything that could cause a huge latency there, it's just a simple wrapper over the C library's recv() function. The waiting certainly occurs inside the OS rather than inside the Python interpreter. -- nosy: +pitrou ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3766 ___ -- nosy: +Thorben ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3766 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3766] socket.socket.recv broken (unbearably slow)
Antoine Pitrou [EMAIL PROTECTED] added the comment: You can use setsockopt() to set the TCP_NODELAY flag and see if that improves things; sending thousands of 4-bytes packets isn't exactly the expected utilization of TCP. As I said, socket.recv() is just a thin wrapper above the same-named C library function. Your code expects the kernel's TCP stack to send things as soon as you ask it to, but TCP is a high-level stream protocol and implementations usually have sophisticated algorithms in order to minimize the number of individual packets sent over the wire. By the way, if you want to build network applications (clients and servers), I suggest you take a look at the Twisted framework. It will free you from many low-level issues, although it won't prevent you from shooting yourself in the foot :-) ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3766 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3766] socket.socket.recv broken (unbearably slow)
Thorben [EMAIL PROTECTED] added the comment: 2008/9/11 Antoine Pitrou [EMAIL PROTECTED]: Antoine Pitrou [EMAIL PROTECTED] added the comment: You can use setsockopt() to set the TCP_NODELAY flag and see if that improves things; sending thousands of 4-bytes packets isn't exactly the expected utilization of TCP. You are obviously right, but IIRC, I mentioned that this is simply a dummy testcase... now thats interesting: adding the line sock.setsockopt(SOL_TCP, TCP_NODELAY, 1) decreased the delay by half. It still is extremely high but it's a start. Do you think I could get that value down much further? I don't know much about TCP... As I said, socket.recv() is just a thin wrapper above the same-named C library function. Your code expects the kernel's TCP stack to send things as soon as you ask it to, but TCP is a high-level stream protocol and implementations usually have sophisticated algorithms in order to minimize the number of individual packets sent over the wire. Would be interesting to examine the differences between the Perl wrapper and the Python wrapper to figure out why Perl does the right thing in this case and Python doesn't. By the way, if you want to build network applications (clients and servers), I suggest you take a look at the Twisted framework. It will free you from many low-level issues, although it won't prevent you from shooting yourself in the foot :-) Thanks, I'll make sure to try it out. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3766 ___ ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3766 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3766] socket.socket.recv broken (unbearably slow)
Antoine Pitrou [EMAIL PROTECTED] added the comment: now thats interesting: adding the line sock.setsockopt(SOL_TCP, TCP_NODELAY, 1) decreased the delay by half. It still is extremely high but it's a start. Did you do it on both the client and server sockets? Would be interesting to examine the differences between the Perl wrapper and the Python wrapper to figure out why Perl does the right thing in this case and Python doesn't. Perhaps the Perl wrapper is less thin as the Python one. In any case, it's by design if the Python socket wrapper doesn't try to be smart: the intent is to provide an access to the C API and let people do what they want with it. Smart things are relegated to higher-level modules or libraries. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3766 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3766] socket.socket.recv broken (unbearably slow)
Thorben [EMAIL PROTECTED] added the comment: 2008/9/11 Antoine Pitrou [EMAIL PROTECTED]: Antoine Pitrou [EMAIL PROTECTED] added the comment: now thats interesting: adding the line sock.setsockopt(SOL_TCP, TCP_NODELAY, 1) decreased the delay by half. It still is extremely high but it's a start. Did you do it on both the client and server sockets? Yes, obviously. Although adding it to the client socket did make no difference after I had already done so for the server. Still communication is too slow by orders of magnitude. (Sorry for pointing this out again) Would be interesting to examine the differences between the Perl wrapper and the Python wrapper to figure out why Perl does the right thing in this case and Python doesn't. Perhaps the Perl wrapper is less thin as the Python one. In any case, it's by design if the Python socket wrapper doesn't try to be smart: the intent is to provide an access to the C API and let people do what they want with it. Smart things are relegated to higher-level modules or libraries. I would greatly appreciate any help on the subject. How do *BSD sockets differ from Linux sockets and what do I do to make things faster. I think this might be where the real issue is. Low level networking voodoo. Do you think twisted might help me there? ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3766 ___ ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3766 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3766] socket.socket.recv broken (unbearably slow)
Gabriel Genellina [EMAIL PROTECTED] added the comment:
I've tested it on Windows XP. MSG_WAITALL is not supported, but I
replaced it using a while loop. I didn't notice any extraneous delay.
500 packets @ 2 tokens each (500 very short lists)
0.140999794006
16008 function calls in 0.146 CPU seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
15000.0360.0000.0360.000 {method 'recv' of
'_socket.socket'
objects}
15000.0330.0000.0330.000 string:1(sendall)
15000.0160.0000.0650.000 Client.py:15(read_int)
15000.0150.0000.0530.000 Client.py:7(send_int)
5000.0090.0000.0770.000
Client.py:22(read_int_list)
5000.0070.0000.0600.000
Client.py:10(send_int_list)
15000.0070.0000.0100.000 struct.py:77(unpack)
15000.0050.0000.0050.000 struct.py:54(pack)
5000.0040.0000.1410.000 Client.py:31(spam)
20010.0040.0000.0040.000 {len}
10.0030.0030.1460.146 runme.py:11(bench)
15000.0030.0000.0030.000 {method 'unpack' of
'Struct' objec
ts}
10010.0030.0000.0030.000 {range}
10000.0020.0000.0020.000 {method 'append' of
'list' objects
}
10.0000.0000.0000.000 struct.py:35(_compile)
20.0000.0000.0000.000 {time.time}
10.0000.0000.1460.146 string:1(module)
10.0000.0000.0000.000 {method 'disable' of
'_lsprof.Prof
iler' objects}
None
1 packet @ 5 tokens (1 very long list)
4.8913242
450019 function calls in 4.893 CPU seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
500011.2120.0001.2120.000 string:1(sendall)
500011.0620.0001.0620.000 {method 'recv' of
'_socket.socket'
objects}
500010.5940.0002.2820.000 Client.py:15(read_int)
500010.5170.0001.9820.000 Client.py:7(send_int)
10.3540.3542.7322.732
Client.py:22(read_int_list)
500010.3350.0000.5240.000 struct.py:77(unpack)
500010.2530.0000.2530.000 struct.py:54(pack)
500010.1890.0000.1890.000 {method 'unpack' of
'Struct' objec
ts}
10.1760.1762.1582.158
Client.py:10(send_int_list)
500020.1020.0000.1020.000 {len}
50.0970.0000.0970.000 {method 'append' of
'list' objects
}
20.0020.0010.0020.001 {range}
10.0020.0024.8934.893 runme.py:19(bench2)
10.0000.0004.8904.890 Client.py:31(spam)
20.0000.0000.0000.000 {time.time}
10.0000.0004.8934.893 string:1(module)
10.0000.0000.0000.000 {method 'disable' of
'_lsprof.Prof
iler' objects}
--
nosy: +gagenellina
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3766
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3766] socket.socket.recv broken (unbearably slow)
Thorben Krueger [EMAIL PROTECTED] added the comment:
As promised, here is the profiler output for a Mac (thanks Stefan). The
problem does NOT occur here, which should help greatly in tracking down
the cause in the socket library. Anyone into this?
$ python runme.py
500 packets @ 2 tokens each (500 very short lists)
0.0814669132233
14508 function calls in 0.082 CPU seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
15000.0480.0000.0480.000 {method 'recv' of
'_socket.socket' objects}
15000.0150.0000.0150.000 string:1(sendall)
15000.0030.0000.0210.000 Client.py:8(send_int)
15000.0030.0000.0530.000 Client.py:16(read_int)
5000.0030.0000.0570.000 Client.py:19(read_int_list)
15000.0020.0000.0020.000 struct.py:77(unpack)
15000.0020.0000.0020.000 struct.py:54(pack)
5000.0010.0000.0220.000 Client.py:11(send_int_list)
10.0010.0010.0820.082 runme.py:12(bench)
10000.0010.0000.0010.000 {method 'insert' of 'list'
objects}
10010.0010.0000.0010.000 {range}
5000.0010.0000.0800.000 Client.py:28(spam)
15000.0000.0000.0000.000 {method 'unpack' of
'Struct' objects}
5010.0000.0000.0000.000 {len}
1 packet @ 5 tokens (1 very long list)
5.20953297615
400018 function calls in 5.210 CPU seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
500012.2670.0002.2670.000 string:1(sendall)
500011.2630.0001.2630.000 {method 'recv' of
'_socket.socket' objects}
51.1120.0001.1120.000 {method 'insert' of 'list'
objects}
500010.1370.0001.4900.000 Client.py:16(read_int)
500010.1340.0002.4780.000 Client.py:8(send_int)
10.0820.0822.6852.685 Client.py:19(read_int_list)
500010.0770.0000.0770.000 struct.py:54(pack)
500010.0730.0000.0890.000 struct.py:77(unpack)
10.0440.0442.5222.522 Client.py:11(send_int_list)
500010.0160.0000.0160.000 {method 'unpack' of
'Struct' objects}
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3766
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3766] socket.socket.recv broken (unbearably slow)
New submission from Thorben Krueger [EMAIL PROTECTED]:
Under Linux 2.6.24-1-amd64 as well as 2.6.26 (x86-32), Python versions
2.5.2 and 2.4.4 (on both machines), there is a huge discrepancy between
socket read latencies, depending on code context.
Some detail:
For a university project, I wrote a python client for a query-server. A
reference implementation existed in Perl, so the job was pretty straight
forward. However, for reasons unknown to me, the Python implementation
was by several orders of magnitude slower than the reference
implementation. To track down the issue, I wrote a dummy version of the
query-server in Python, where the problem persisted. I then stripped
down both client and server to the minimal functionality and still the
problem persisted.
I wrote a demo inside a single file using socketpair() to post here, but
the issue was not reproducible.
Finally, I stripped down the original client/server to a postable level
and ran a profiler on a testcase.
Here is the gist of it:
Sending 500 packets @ 2 tokens each (500 very short lists) takes 40
seconds on average.
In detail:
14508 function calls in 39.980 CPU seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
1500 39.8340.027 39.8340.027 {method 'recv' of
'_socket.socket' objects}
15000.0240.000 39.8770.027 Client.py:16(read_int)
15000.0200.0000.0200.000 string:1(sendall)
15000.0180.0000.0480.000 Client.py:8(send_int)
5000.0160.000 39.9030.080 Client.py:19(read_int_list)
15000.0150.0000.0190.000 struct.py:77(unpack)
5000.0100.0000.0600.000 Client.py:11(send_int_list)
15000.0100.0000.0100.000 struct.py:54(pack)
10.0090.009 39.980 39.980 dummytest.py:12(bench)
10000.0070.0000.0070.000 {method 'insert' of 'list'
objects}
10010.0060.0000.0060.000 {range}
5000.0050.000 39.9680.080 Client.py:28(spam)
15000.0050.0000.0050.000 {method 'unpack' of
'Struct' objects}
5010.0020.0000.0020.000 {len}
Here is the same for 1 packet @ 5 tokens (1 very long list), taking
below 10 seconds on average.
8.51872587204
400018 function calls in 8.519 CPU seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
500012.9800.0002.9800.000 {method 'recv' of
'_socket.socket' objects}
52.9800.0002.9800.000 {method 'insert' of 'list'
objects}
500010.9930.0000.9930.000 string:1(sendall)
500010.4100.0001.5580.000 Client.py:8(send_int)
500010.3340.0003.5810.000 Client.py:16(read_int)
10.2470.2476.8096.809 Client.py:19(read_int_list)
500010.1910.0000.2660.000 struct.py:77(unpack)
500010.1540.0000.1540.000 struct.py:54(pack)
10.1460.1461.7031.703 Client.py:11(send_int_list)
500010.0750.0000.0750.000 {method 'unpack' of
'Struct' objects}
I don't get the reason for the huge speed discrepancy. I include all
source code files for reproducing the issue.
Notably, the original software (non stripped-down version) runs without
these issues using a OS X Python version. Details may follow, I don't
own a Mac but know people who do.
Also note that I can't get the server to shut down properly (the thread
does not terminate). You have to kill the process manually and wait for
the port to be freed by the kernel. Maybe this is easily fixable but at
least I don't know how.
The attached archive contains all source code plus README and the
socketpair() version.
--
components: Library (Lib)
files: socket_bug.tar.bz2
messages: 72424
nosy: thorben
severity: normal
status: open
title: socket.socket.recv broken (unbearably slow)
versions: Python 2.4, Python 2.5
Added file: http://bugs.python.org/file11363/socket_bug.tar.bz2
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3766
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
