[issue32251] Add asyncio.BufferedProtocol

Yury Selivanov added the comment: New changeset 07627e9a6a5f418354ff3dc99a0f36bc5b79dcd8 by Yury Selivanov in branch 'master': bpo-32251: Fix docs (#5408) https://github.com/python/cpython/commit/07627e9a6a5f418354ff3dc99a0f36bc5b79dcd8 --

[issue32251] Add asyncio.BufferedProtocol

Change by Yury Selivanov : -- pull_requests: +5241 ___ Python tracker ___ ___

[issue32251] Add asyncio.BufferedProtocol

Change by Yury Selivanov : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed type: -> enhancement ___ Python tracker

[issue32251] Add asyncio.BufferedProtocol

Yury Selivanov added the comment: New changeset 631fd38dbf04dbf0127881f3977982e401a849e4 by Yury Selivanov in branch 'master': bpo-32251: Implement asyncio.BufferedProtocol. (#4755) https://github.com/python/cpython/commit/631fd38dbf04dbf0127881f3977982e401a849e4

[issue32251] Add asyncio.BufferedProtocol

Yury Selivanov added the comment: >> Looks nice. Can it speed up aiohttp too? > Yes. > aiohttp uses own streams but public API and internal implementation are > pretty close to asyncio streams. > Moreover C accelerated HTTP parser should work with proposed

[issue32251] Add asyncio.BufferedProtocol

Yury Selivanov added the comment: > See https://eklitzke.org/goroutines-nonblocking-io-and-memory-usage for an > interesting discussion of the drawbacks of some buffer handling idioms. Thanks for the link! It does make sense to use a pool of buffers for the proposed

[issue32251] Add asyncio.BufferedProtocol

Antoine Pitrou added the comment: See https://eklitzke.org/goroutines-nonblocking-io-and-memory-usage for an interesting discussion of the drawbacks of some buffer handling idioms. -- ___ Python tracker

[issue32251] Add asyncio.BufferedProtocol

Andrew Svetlov added the comment: Yes. aiohttp uses own streams but public API and internal implementation are pretty close to asyncio streams. Moreover C accelerated HTTP parser should work with proposed BufferedProtocol seamlessly. --

[issue32251] Add asyncio.BufferedProtocol

INADA Naoki added the comment: Looks nice. Can it speed up aiohttp too? -- nosy: +inada.naoki ___ Python tracker ___

[issue32251] Add asyncio.BufferedProtocol

Łukasz Langa added the comment: +1 on the idea, I would use this. -- nosy: +lukasz.langa ___ Python tracker ___

[issue32251] Add asyncio.BufferedProtocol

Change by Guido van Rossum : -- nosy: -gvanrossum ___ Python tracker ___ ___

[issue32251] Add asyncio.BufferedProtocol

Andrew Svetlov added the comment: New protocol may speed up not only `reader.readexactly()`. Every `reader.feed_data()` appends a new chunk to existing buffer. We can try to read into unused tail of the buffer instead. --

[issue32251] Add asyncio.BufferedProtocol

Yury Selivanov added the comment: > Do you think it's possible to fold BufferedProtocol into Protocol? It would be a backwards incompatible change :( Coincidentally, there might be protocols that already implement 'get_buffer()' and 'buffer_updated()' methods that do

[issue32251] Add asyncio.BufferedProtocol

Antoine Pitrou added the comment: Do you think it's possible to fold BufferedProtocol into Protocol? i.e., either `get_buffer()` returns None (the default) and `data_received()` is called with a bytes object, or `get_buffer()` returns a writable buffer and `buffer_updated()`

[issue32251] Add asyncio.BufferedProtocol

Yury Selivanov added the comment: > 1. What happens if size of read data is greater than pre-allocated buffer? Let's say we have 2Kb of data in the socket's network buffer, and we only preallocated 0.5Kb in our buffer. We will the receive 0.5Kb in our buffer,

[issue32251] Add asyncio.BufferedProtocol

Antoine Pitrou added the comment: I have another question: what happens if there is a partial read? For example, let's says I return a 1024-bytes buffer in get_buffer(), but recv_into() receives data in 512 chunks. Is it: 1. getbuffer() is called, returns 1024 bytes buffer

[issue32251] Add asyncio.BufferedProtocol

Andrew Svetlov added the comment: Numbers are great! Couple questions. 1. What happens if size of read data is greater than pre-allocated buffer? 2. Is flow control logic changed or not? If I understand correctly pause_reading() / resume_reading() continue to work

[issue32251] Add asyncio.BufferedProtocol

Yury Selivanov added the comment: I've made a PR that implements the change for selector_events.py. With the change: vanilla asyncio:120-135 Mb/s vanilla asyncio/get_buffer: 220-230 Mb/s uvloop: 320-330 Mb/s uvloop/get_buffer:

[issue32251] Add asyncio.BufferedProtocol

Change by Yury Selivanov : -- keywords: +patch pull_requests: +4658 stage: -> patch review ___ Python tracker ___

[issue32251] Add asyncio.BufferedProtocol

New submission from Yury Selivanov : A couple emails from async-sig for the context: 1. https://mail.python.org/pipermail/async-sig/2017-October/000392.html 2. https://mail.python.org/pipermail/async-sig/2017-December/000423.html I propose to add another Protocol base

[issue32251] Add asyncio.BufferedProtocol

Change by Yury Selivanov : -- assignee: yselivanov components: asyncio nosy: asvetlov, gvanrossum, pitrou, yselivanov priority: normal severity: normal status: open title: Add asyncio.BufferedProtocol versions: Python 3.7 ___