On Sep 5, 2019, at 11:13, Barry Scott <ba...@barrys-emacs.org> wrote:
>
>
>
>> On 5 Sep 2019, at 12:06, Dan Sommers <2qdxy4rzwzuui...@potatochowder.com>
>> wrote:
>>
>> Yes, you need some way to produce "back pressure" from downstream to
>> upstream, and to stop making new work (with new memory consumption)
>> until there's a place to put it.
>
> It seems that this is the important idea.
>
> How does the back pressure work in different use cases.
>
> 1) I don't care.
>
> Just add items to the queue. (The current API?)
>
> 2) I can be blocked when the queue is at a limit.
>
> Need an API to allows the block to happen
> and resumes the code when there is space in the queue.
>
> 3) I cannot be blocked as this is async code, but I need a bound on the queue.
>
> Maybe call "queue has space" callback when the queue has space?
> Maybe submit returns an "queue full" status?
>
> I have needed all 3 uses cases.
Have you actually needed case 3 with Executor, or only with other kinds of
async models?
Anyway, failing makes sense, is usable for backpressure, and is trivial to
implement. Although I think it should raise an exception rather than return a
special value in place of a tuple. Here’s a design (borrowed from Queue of
course):
* submit always succeeds, blocking if necessary
* submit_nowait always completes immediately, raising executor.Full if
necessary
And of course if you don’t pass a max_queue_len, they both always succeed
immediately.
I’m not sure submit_nowait is needed. And if we add max_queue_len in 3.x and
people start asking for submit_nowait, it would be trivial to add it in 3.x+1.
But if someone has an actual use case for it now, I don’t see any problem at
all adding it now. Do you have real code or realistic sample code that needs it?
I don’t think the queue-has-room callback, on the other hand, is very useful.
That’s just clunkily simulating an unbounded queue on top of a bounded queue
plus an implicit unbounded queue of callbacks.
Anyway, in theory, there are a zillion different things you could conceivably
want to happen on failure, which is why Java and Ruby have a zillion different
fail protocols, and Ruby lets you set a fail method if even that isn’t good
enough. But in practice, I’m pretty sure either that people only need “block”,
or that they only need “block” and “raise”. I’m just not sure which of those
two.
_______________________________________________
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/NJT3AWQTDIZWFUKDF6MURWRL5RB75X75/
Code of Conduct: http://python.org/psf/codeofconduct/
