[issue18329] for line in socket.makefile() speed degradation

Марк Коренберг added the comment: mmarkk@mmwork:~$ python2.7 --version Python 2.7.12 mmarkk@mmwork:~$ python2.7 ./qwe.py TCP mode, makefile method. 292731.3 lines per second (279.2 MB/s). Delay is 3.42 seconds TCP mode, fdopen method. 2410875.9 lines per second (2299.2 MB/s). Delay is

[issue18329] for line in socket.makefile() speed degradation

Christian Heimes added the comment: Is this still a problem in 3.5 and newer? -- status: open -> pending versions: +Python 3.6, Python 3.7 -Python 3.2, Python 3.3, Python 3.4, Python 3.5 ___ Python tracker

[issue18329] for line in socket.makefile() speed degradation

Changes by Giampaolo Rodola' g.rod...@gmail.com: -- nosy: +giampaolo.rodola ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18329 ___ ___

[issue18329] for line in socket.makefile() speed degradation

Changes by Christian Heimes li...@cheimes.de: -- nosy: +christian.heimes ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18329 ___ ___

[issue18329] for line in socket.makefile() speed degradation

Christian Heimes added the comment: I first suspected that file based IO involves too many syscalls and context switches. However ``strace -c`` showed only minor differences in the amount of syscalls. The difference between read() and recvfrom() should not make any difference. perf revealed

[issue18329] for line in socket.makefile() speed degradation

Christian Heimes added the comment: Summary: makefile: 611990 primitive calls fdopen: 1629 primitive calls -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18329 ___

[issue18329] for line in socket.makefile() speed degradation

Richard Oudkerk added the comment: I think I know what's going on here. For socket IO readline() uses a readahead buffer size of 1. Why is that? I think that makefile(mode='rb') and fdopen() both create BufferedReader objects with the same buffer size. It looks to me like there are the

[issue18329] for line in socket.makefile() speed degradation

Richard Oudkerk added the comment: Using while True: if not fileobj.read(8192): break instead of for line in fileobj: pass results in higher throughput, but a similar slowdown with makefile(). So this is not a problem specific to

[issue18329] for line in socket.makefile() speed degradation

Christian Heimes added the comment: Err, you are right. It's buffered. I misread one function call and thought it's not buffered at all. COUNT = 100 LINELEN = 1000 int(LINELEN * COUNT / 8192) 122070 The number looks familiar, doesn't it? :) I still stand by my analysis that the slow

[issue18329] for line in socket.makefile() speed degradation

Richard Oudkerk added the comment: The only real reason for implementing SocketIO in pure Python is because read() and write() do not work on Windows with sockets. (I think there are also a few complications involving SSL sockets and the close() method.) On Windows I have implemented a file

[issue18329] for line in socket.makefile() speed degradation

Christian Heimes added the comment: There is little bit more to it. A comment in the class' body describes the reason for SockteIO: # One might wonder why not let FileIO do the job instead. There are two # main reasons why FileIO is not adapted: # - it wouldn't work under Windows

[issue18329] for line in socket.makefile() speed degradation

Richard Oudkerk added the comment: Ah. I had not thought of socket timeouts. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18329 ___ ___

[issue18329] for line in socket.makefile() speed degradation

STINNER Victor added the comment: On Python 2, socket.makefile(rb) duplicates the file descriptor and creates a new file object: fileobj.read() calls the C function read(). On Python 3, socket.makefile(rb) creates a SocketIO wrapped in BufferedReader: fileobj.read() calls the C function

[issue18329] for line in socket.makefile() speed degradation

STINNER Victor added the comment: My results of the benchmark on Linux 3.9, first with a loop calling fileobj.read(8192) (read), then with the for line in fileobj: pass (readline). $ python2.7 bench_socket_read.py TCP mode, makefile method. 1775085.8 lines per second (1692.9 MB/s). Delay is

[issue18329] for line in socket.makefile() speed degradation

Richard Oudkerk added the comment: I find that by adding the lines fileobj.raw.readinto = ss.recv_into fileobj.raw.read = ss.recv the speed with makefile() is about 30% slower than with fdopen(). -- ___ Python tracker

[issue18329] for line in socket.makefile() speed degradation

Serhiy Storchaka added the comment: 3.4 have same performance as 3.3. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18329 ___ ___

[issue18329] for line in socket.makefile() speed degradation

New submission from Марк Коренберг: Results or running attached program: $ python2.7 qwe.py TCP mode, makefile method. 198807.2 lines per second (189.6 MB/s). Delay is 5.03 seconds TCP mode, fdopen method. 1041666.7 lines per second (993.4 MB/s). Delay is 0.96 seconds UNIX mode, makefile

[issue18329] for line in socket.makefile() speed degradation

Марк Коренберг added the comment: Yes, results are repeatable, and for python 2.7 I have roughly same timings for UNIX socket. Also, I have straced all variants and see that in all 4 cases (and for both python versions) IO is done using 8192 blocks in size, so buffering is not cause of

[issue18329] for line in socket.makefile() speed degradation

Марк Коренберг added the comment: Well, python 3.3 is slightly faster: $ python3.3 qwe.py TCP mode, makefile method. 380228.1 lines per second (362.6 MB/s). Delay is 2.63 seconds TCP mode, fdopen method. 877193.0 lines per second (836.6 MB/s). Delay is 1.14 seconds UNIX mode, makefile

[issue18329] for line in socket.makefile() speed degradation

Serhiy Storchaka added the comment: Try to wrap socket.makefile() with io.BufferedReader(). -- nosy: +serhiy.storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18329 ___

[issue18329] for line in socket.makefile() speed degradation

Richard Oudkerk added the comment: I think in Python 3 makefile() returns a TextIOWrapper object by default. To force the use of binary you need to specfiy the mode: fileobj = ss.makefile(mode='rb') -- nosy: +sbt ___ Python tracker

[issue18329] for line in socket.makefile() speed degradation

Changes by Марк Коренберг socketp...@gmail.com: Removed file: http://bugs.python.org/file30731/qwe.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18329 ___

[issue18329] for line in socket.makefile() speed degradation

Марк Коренберг added the comment: Eliminate unicode conversion for python3, but results still the same $ python2.7 qwe.py TCP mode, makefile method. 211416.5 lines per second (201.6 MB/s). Delay is 4.73 seconds TCP mode, fdopen method. 1041666.7 lines per second (993.4 MB/s). Delay is

[issue18329] for line in socket.makefile() speed degradation

Марк Коренберг added the comment: Can anyone test in python 3.4 ? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18329 ___ ___ Python-bugs-list