I have been testing network I/O performance of Python and PyPy. I have used the small script in the attachment to test TCP bandwidth between two powerful servers, each has two 10Gb nics with multi-queue hardware support to form a bond, so the physical bandwidth between the servers are 20Gbps.
The script is a simple socket client/server with multiple connections on
multiple threads. The server tries its best to receive all data, and the client
tries its best to send data, after 30 sec, the client stops and calculates the
data sent.
It seems that there is a huge performance difference for the server to use pypy
or CPython:
(python = CPython, pypy = PyPy, 100.102.4.7 is my test server IP address, and
32 is the number of concurrent connections)
With python testio.py -s on another server:
pypy testio.py -c 100.102.4.7 32
total time = 30.05s, total send = 66601 MB, speed = 17.31 Gbps
python testio.py -c 100.102.4.7 32
total time = 30.08s, total send = 67008 MB, speed = 17.40 Gbps
With pypy testio.py -s on another server:
pypy testio.py -c 100.102.4.7 32
total time = 30.36s, total send = 5838 MB, speed = 1.50 Gbps
python testio.py -c 100.102.4.7 32
total time = 30.36s, total send = 5742 MB, speed = 1.48 Gbps
But when I change
while s.recv_into(recv_buf, 0x200000):
to
while s.recv(0x200000):
The performance difference disappears, and both PyPy server and CPython server
have good performance (17+Gbps)
It is worth to point out that when using socket.recv_into(), PyPy server only
uses 100% CPU (one core), while CPython uses much more. Maybe there are some
unexpected issues about socket.recv_into, e.g. GIL not released?
P.S. it is interesting that though I thought recv_into() should be more
efficient thant recv() since it reduces extra object creation / destruction,
the test result shows that recv() outperforms recv_into(), even with CPython.
With CPython, it seems server with recv() costs less CPU time than recv_into(),
but having the same I/O performance.
2016-08-08
hubo
testio.py
Description: Binary data
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
