On 11 June 2015 at 10:55, Paul Sokolovsky <[email protected]> wrote: > Hello, > > On Thu, 11 Jun 2015 02:05:23 -0700 (PDT) > Martin Teichmann <[email protected]> wrote: > > [] > > > What I am doing is the following: several tasks in my program are > > generating big amounts of data to be shipped out on a StreamWriter. > > This can easily overload the receiver of all that data. This is why > > every task, after calling > > writer.write also calls "yield from writer.drain()". Unfortunately, > > while draining > > another task may write to the same stream writer, also wants to call > > drain. This raises an AssertionError. > > This is a big problem, about which I wanted to write for a long time. > The root of the problem is however not drain(), but a synchronous > write() method, whose semantics seems to be drawn as to easily allow DoS > attacks on the platform where the code runs - it's required to buffer > unlimited amounts of data, which is not possible on any physical > platform, and will only lead to excessive virtual memory swapping and > out-of-memory killings on real systems (why the reference to DoS). > > Can we please-please have async_write() method? Two boundary > implementations of it would be: > > # Same behavior as currently - unlimited buffering > def async_write(...): > return self.write() > yield > > > # Memory-conscious implementation > def async_write(...): > self.write() > yield from self.drain() >
I have some concerns about encouraging such an API. Many applications will want to do small writes, of a few bytes at a time. Making every write() call a coroutine causes an enormous amount of overhead, as each time you write some small piece of data you have to suspend the current coroutine and go back to the main loop. So, I'm happy with the current API, plus documentation explaining that you need "yield from self.drain()" at appropriate places. -- Gustavo J. A. M. Carneiro Gambit Research "The universe is always one step beyond logic." -- Frank Herbert
