Yeah, this is a fundamental problem, not only with Python but any other C library that uses blocking calls. At one point, when wrapping such a C library, I had the slightly insane notion of using LD_PRELOAD <http://jvns.ca/blog/2014/11/27/ld-preload-is-super-fun-and-easy/> to replace all the blocking functions in libc with our own versions that do the same thing but using Julia's yielding API.
On Tue, Oct 13, 2015 at 6:35 PM, Yakir Gagnon <[email protected]> wrote: > I'll try the remotecall option (next week), hopefully the fetch from that > won't get things stuck. > I imagine there are a lot of *other* Python libraries that don't > use libuv and get Julia's coroutines stuck... > > Thanks for all the attention! > > > Yakir Gagnon > The Queensland Brain Institute (Building #79) > The University of Queensland > Brisbane QLD 4072 > Australia > > cell +61 (0)424 393 332 > work +61 (0)733 654 089 > > On Tue, Oct 13, 2015 at 10:58 PM, Stefan Karpinski <[email protected]> > wrote: > >> I'm working on fixing up the API to Amit's PR here >> <https://github.com/JuliaLang/julia/pull/12503> that allows you to call >> a C function in another thread. That could also potentially be used for >> this. >> >> On Tue, Oct 13, 2015 at 2:43 PM, Mohammed El-Beltagy < >> [email protected]> wrote: >> >>> Short of doing a reimplementation, you could possibly run your python >>> code in another process via a remotecall (as described in the manual >>> http://julia.readthedocs.org/en/latest/manual/parallel-computing/). In >>> that case you REPL would be responsive as the I/O is done in another >>> process. >>> >>> On Monday, October 12, 2015 at 10:10:39 PM UTC+2, Yakir Gagnon wrote: >>>> >>>> I see, thanks for the great explanation! >>>> So there's nothing I can do. Would Escher get around it? I guess I'd >>>> need to implement that python code in Julia... >>>> On 12/10/2015 11:34 PM, "Steven G. Johnson" <[email protected]> wrote: >>>> >>>>> >>>>> >>>>> On Monday, October 12, 2015 at 1:59:59 AM UTC-4, Yakir Gagnon wrote: >>>>>> >>>>>> One important piece of information is the integration time (similar >>>>>> to the shutter speed in a camera): after I set the integration time the >>>>>> spectrometer start sampling the spectra at that frequency. When I try to >>>>>> retrieve the intensities it will spit them out only when one cycle ends. >>>>>> This means that when I try to run the function that retrieves the >>>>>> intensities it can take anything from 0 to integration-time seconds. >>>>>> >>>>>> Here's the weird thing: >>>>>> When I run my code the REPL becomes non-responsive for >>>>>> integration-time seconds, so if I try to type some text, the letters get >>>>>> typed in only one letter at an integration-time (note that CPU usage is >>>>>> less than 3%)... But, if I replace the function that retrieves the >>>>>> intensities with some mock function that `sleep`s for a random amount of >>>>>> time and returns a (equally long) vector of random floats, the REPL >>>>>> jitter >>>>>> is gone..! >>>>>> >>>>> >>>>> Julia I/O functions, and functions like sleep(t), use the libuv >>>>> library for asynchronous cooperative multitasking. That means that when >>>>> one task is waiting on I/O, another task (e.g. the REPL) can wake up if >>>>> there is something for it to do. >>>>> >>>>> However, Python I/O does not use libuv, so when the Python I/O task is >>>>> waiting to finish reading something then it just blocks, and nothing else >>>>> in Julia can run. >>>>> >>>> >> >
