[Async-sig] avoiding accidentally calling blocking code
Hi, This is a general Python async question I've had that I haven't seen a discussion of. Do people know of any techniques other than manually inspecting code line by line to find out if you're accidentally making a raw call to a blocking function in a coroutine? Related to this, again, outside of inspecting the external source code, is there any way to know if a function you're calling (e.g. in someone else's library or in the standard lib) has the potential of blocking? What would be the way of detecting something like this if it made its way into "production"? What would be some of the symptoms? --Chris ___ Async-sig mailing list Async-sig@python.org https://mail.python.org/mailman/listinfo/async-sig Code of Conduct: https://www.python.org/psf/codeofconduct/
Re: [Async-sig] avoiding accidentally calling blocking code
Let me try to answer the question behind the question.
Like any code validation, it's a healthy mix of:
* unit tests [perhaps with
mock.patch_all_known_blocking_calls(side_effect=Exception)]
* good judgement [open("foo").read() technically blocks, but only a
problem on network filesystems]
* monitoring prod [at least to ensure that tests correspond to prod]
* peer review
Specific issues you've raised:
re: detecting blocking calls ahead of time.
That's a hard problem (insert quip about Turing completeness)
Perhaps it can be alleviated by function annotations, but that's
dangerously close to Java way...
re: detecting blocking calls at run time.
Let's say there's a per-thread global "blocking lock" that's managed
by event loop executor and it's in "allowed" state when executor has
no runnables and wishes to select/poll/epoll and it's in "denied"
state when user code is running.
Some calls can be caught (e.g. check fd fcntl bits before any recv/send/...)
Some calls can never be caught (anything via ctypes or extensions in general)
Case in point: what do you propose to do about socket.gethostbyname()?
re: after the fact.
hack1. run a separate thread/process that inspects event loop thread's
`wchan` value (Linux)
hack2. same via taskstats
hack3. detect event loop lag (false positives are cpu-bound tasks and
overloaded system, but I think you'd like to detect those too)
On 13 February 2018 at 12:10, Chris Jerdonek wrote:
> Hi,
>
> This is a general Python async question I've had that I haven't seen a
> discussion of. Do people know of any techniques other than manually
> inspecting code line by line to find out if you're accidentally making
> a raw call to a blocking function in a coroutine?
>
> Related to this, again, outside of inspecting the external source
> code, is there any way to know if a function you're calling (e.g. in
> someone else's library or in the standard lib) has the potential of
> blocking?
>
> What would be the way of detecting something like this if it made its
> way into "production"? What would be some of the symptoms?
>
> --Chris
> ___
> Async-sig mailing list
> Async-sig@python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/
Re: [Async-sig] avoiding accidentally calling blocking code
loop.slow_callback_duration + PYTHONASYNCIODEBUG=1
Does it fork for you?
Do you need extra info?
On Tue, Feb 13, 2018 at 7:07 AM Dima Tisnek wrote:
> Let me try to answer the question behind the question.
>
> Like any code validation, it's a healthy mix of:
> * unit tests [perhaps with
> mock.patch_all_known_blocking_calls(side_effect=Exception)]
> * good judgement [open("foo").read() technically blocks, but only a
> problem on network filesystems]
> * monitoring prod [at least to ensure that tests correspond to prod]
> * peer review
>
>
> Specific issues you've raised:
>
> re: detecting blocking calls ahead of time.
> That's a hard problem (insert quip about Turing completeness)
> Perhaps it can be alleviated by function annotations, but that's
> dangerously close to Java way...
>
> re: detecting blocking calls at run time.
> Let's say there's a per-thread global "blocking lock" that's managed
> by event loop executor and it's in "allowed" state when executor has
> no runnables and wishes to select/poll/epoll and it's in "denied"
> state when user code is running.
> Some calls can be caught (e.g. check fd fcntl bits before any
> recv/send/...)
> Some calls can never be caught (anything via ctypes or extensions in
> general)
> Case in point: what do you propose to do about socket.gethostbyname()?
>
> re: after the fact.
> hack1. run a separate thread/process that inspects event loop thread's
> `wchan` value (Linux)
> hack2. same via taskstats
> hack3. detect event loop lag (false positives are cpu-bound tasks and
> overloaded system, but I think you'd like to detect those too)
>
> On 13 February 2018 at 12:10, Chris Jerdonek
> wrote:
> > Hi,
> >
> > This is a general Python async question I've had that I haven't seen a
> > discussion of. Do people know of any techniques other than manually
> > inspecting code line by line to find out if you're accidentally making
> > a raw call to a blocking function in a coroutine?
> >
> > Related to this, again, outside of inspecting the external source
> > code, is there any way to know if a function you're calling (e.g. in
> > someone else's library or in the standard lib) has the potential of
> > blocking?
> >
> > What would be the way of detecting something like this if it made its
> > way into "production"? What would be some of the symptoms?
> >
> > --Chris
> > ___
> > Async-sig mailing list
> > Async-sig@python.org
> > https://mail.python.org/mailman/listinfo/async-sig
> > Code of Conduct: https://www.python.org/psf/codeofconduct/
> ___
> Async-sig mailing list
> Async-sig@python.org
> https://mail.python.org/mailman/listinfo/async-sig
> Code of Conduct: https://www.python.org/psf/codeofconduct/
>
--
Thanks,
Andrew Svetlov
___
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/
