Mocking everything sounds promising but the problem is in *everything* word.
slow_callback_duration is a viable compromise.
Personally I don't care about fast blocking IO until it is really very fast.
On Wed, Feb 14, 2018 at 10:42 AM Chris Jerdonek <chris.jerdo...@gmail.com>
wrote:
> Thanks, Dima and Andrew, for your suggestions.
>
> Re: loop.slow_callback_duration, that's interesting. I didn't know
> about that. However, it doesn't seem like that alone would work to
> catch many operations that are normally fast, but strictly speaking
> blocking. For example, it seems like simple disk I/O operations
> wouldn't be caught even with slow_callback_duration set to a small
> value. Dima suggests that such calls are okay. Is there a consensus on
> that?
>
> Dima's mock.patch_all_known_blocking_calls() is an interesting idea
> and seems like it would work for the case I mentioned. Has anyone
> started writing such a method (e.g. for certain standard lib modules)?
>
> --Chris
>
>
> On Mon, Feb 12, 2018 at 11:24 PM, Andrew Svetlov
> <andrew.svet...@gmail.com> wrote:
> > loop.slow_callback_duration + PYTHONASYNCIODEBUG=1
> > Does it fork for you?
> > Do you need extra info?
> >
> > On Tue, Feb 13, 2018 at 7:07 AM Dima Tisnek <dim...@gmail.com> wrote:
> >>
> >> Let me try to answer the question behind the question.
> >>
> >> Like any code validation, it's a healthy mix of:
> >> * unit tests [perhaps with
> >> mock.patch_all_known_blocking_calls(side_effect=Exception)]
> >> * good judgement [open("foo").read() technically blocks, but only a
> >> problem on network filesystems]
> >> * monitoring prod [at least to ensure that tests correspond to prod]
> >> * peer review
> >>
> >>
> >> Specific issues you've raised:
> >>
> >> re: detecting blocking calls ahead of time.
> >> That's a hard problem (insert quip about Turing completeness)
> >> Perhaps it can be alleviated by function annotations, but that's
> >> dangerously close to Java way...
> >>
> >> re: detecting blocking calls at run time.
> >> Let's say there's a per-thread global "blocking lock" that's managed
> >> by event loop executor and it's in "allowed" state when executor has
> >> no runnables and wishes to select/poll/epoll and it's in "denied"
> >> state when user code is running.
> >> Some calls can be caught (e.g. check fd fcntl bits before any
> >> recv/send/...)
> >> Some calls can never be caught (anything via ctypes or extensions in
> >> general)
> >> Case in point: what do you propose to do about socket.gethostbyname()?
> >>
> >> re: after the fact.
> >> hack1. run a separate thread/process that inspects event loop thread's
> >> `wchan` value (Linux)
> >> hack2. same via taskstats
> >> hack3. detect event loop lag (false positives are cpu-bound tasks and
> >> overloaded system, but I think you'd like to detect those too)
> >>
> >> On 13 February 2018 at 12:10, Chris Jerdonek <chris.jerdo...@gmail.com>
> >> wrote:
> >> > Hi,
> >> >
> >> > This is a general Python async question I've had that I haven't seen a
> >> > discussion of. Do people know of any techniques other than manually
> >> > inspecting code line by line to find out if you're accidentally making
> >> > a raw call to a blocking function in a coroutine?
> >> >
> >> > Related to this, again, outside of inspecting the external source
> >> > code, is there any way to know if a function you're calling (e.g. in
> >> > someone else's library or in the standard lib) has the potential of
> >> > blocking?
> >> >
> >> > What would be the way of detecting something like this if it made its
> >> > way into "production"? What would be some of the symptoms?
> >> >
> >> > --Chris
> >> > _______________________________________________
> >> > Async-sig mailing list
> >> > Async-sig@python.org
> >> > https://mail.python.org/mailman/listinfo/async-sig
> >> > Code of Conduct: https://www.python.org/psf/codeofconduct/
> >> _______________________________________________
> >> Async-sig mailing list
> >> Async-sig@python.org
> >> https://mail.python.org/mailman/listinfo/async-sig
> >> Code of Conduct: https://www.python.org/psf/codeofconduct/
> >
> > --
> > Thanks,
> > Andrew Svetlov
>
--
Thanks,
Andrew Svetlov
_______________________________________________
Async-sig mailing list
Async-sig@python.org
https://mail.python.org/mailman/listinfo/async-sig
Code of Conduct: https://www.python.org/psf/codeofconduct/
