New submission from Kumar Aditya <[email protected]>:
Currently, _SelectorSocketTransport transport creates a copy of the data before
sending which in case of large amount of data, can create multiple giga bytes
copies of data before sending.
Script demonstrating current behavior:
-------------------------------------------------------------------------
import asyncio
import memory_profiler
@memory_profiler.profile
async def handle_echo(reader: asyncio.StreamReader, writer:
asyncio.StreamWriter):
data = b'x' * 1024 * 1024 * 1000 # 1000 MiB payload
writer.write(data)
await writer.drain()
writer.close()
async def main():
server = await asyncio.start_server(
handle_echo, '127.0.0.1', 8888)
addrs = ', '.join(str(sock.getsockname()) for sock in server.sockets)
print(f'Serving on {addrs}')
async with server:
asyncio.create_task(server.start_serving())
reader, writer = await asyncio.open_connection('127.0.0.1', 8888)
while True:
data = await reader.read(1024 * 1024 * 100)
if not data:
break
asyncio.run(main())
-------------------------------------------------------------------------
Memory profile result:
------------------------------------------------------------------------
Filename: test.py
Line # Mem usage Increment Occurrences Line Contents
=============================================================
4 17.7 MiB 17.7 MiB 1 @memory_profiler.profile
5 async def handle_echo(reader:
asyncio.StreamReader, writer: asyncio.StreamWriter):
6 1017.8 MiB 1000.1 MiB 1 data = b'x' * 1024 * 1024 *
1000 # 1000 MiB payload
7 2015.3 MiB 997.5 MiB 1 writer.write(data)
8 2015.3 MiB -988.1 MiB 2 await writer.drain()
9 1027.1 MiB -988.1 MiB 1 writer.close()
------------------------------------------------------------------------
To make it zero copy, python's buffer protocol can be used and use memory views
of data to save RAM. The writelines method currently joins all the data before
sending whereas it can use `socket.sendmsg` to make it more memory efficient.
Links:
- writelines -
https://github.com/python/cpython/blob/2153daf0a02a598ed5df93f2f224c1ab2a2cca0d/Lib/asyncio/transports.py#L116
- socket.sendmsg -
https://docs.python.org/3/library/socket.html#socket.socket.sendmsg
- memory_profiler -
https://pypi.org/project/memory-profiler/
----------
components: asyncio
messages: 415124
nosy: asvetlov, gvanrossum, kumaraditya303, yselivanov
priority: normal
severity: normal
status: open
title: Implement zero copy writes in SelectorSocketTransport in asyncio
type: resource usage
versions: Python 3.11
_______________________________________
Python tracker <[email protected]>
<https://bugs.python.org/issue47010>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
