I think the design of ZMQ is that each thread should have its own socket. But I don't know whether this would help with the issue you're seeing. And there may be an extra complication because the kernel binds the sockets (not connects), and I think only one socket can be bound to an endpoint at once.
Thomas On 19 December 2017 at 14:22, Christian Schafmeister < [email protected]> wrote: > Yes - this is how the pzmq (a Common Lisp wrapper library for the zmq > library) works as well. > Here is the code that I added the mutex to (it's Common Lisp but you > should be able to read it): > > https://github.com/clasp-developers/cl-jupyter/blob/ > master/src/message.lisp#L300 > > The calls to 'message-send' all pass the same sockets (iopub socket or > shell socket) - so if multiple threads call 'message-send' at the same time > there is nothing that distinguishes what thread is sending what parts. By > wrapping a mutex around the multiple pzmq:send calls it ensures that > different threads don't call pzmq:send at the same time. > > I don't know how threading works in Python - maybe this isn't an issue in > Python. > In Common Lisp I'm using the C pthread library for threading and threads > trample on each other if I don't take care like this. > > > > On Tuesday, December 19, 2017 at 7:37:12 AM UTC-5, takowl wrote: >> >> The Jupyter protocol relies on ZMQ 'multipart' messages. IIRC, a >> multipart message is a series of individual messages with the SNDMORE flag >> set on all but the last one. I don't know if those parts are meant to be >> separated out again if two threads are sending parts interleaved to the >> same endpoint. >> >> On 19 December 2017 at 12:28, Christian Schafmeister < >> [email protected]> wrote: >> >>> I'm certain it's a problem on my kernel side, it has to do with sending >>> messages to jupyter and not receiving them from jupyter. The messages are >>> send out using multiple calls to the pzmq:send function and when multiple >>> threads were doing this at the same time the messages got garbled. I >>> wrapped the sending code in a mutex and the problem went away. >>> I don't think it's a jupyter issue. >>> >>> On Tuesday, December 19, 2017 at 2:00:33 AM UTC-5, Roland Weber wrote: >>>> >>>> On Monday, December 18, 2017 at 3:00:23 PM UTC+1, Christian >>>> Schafmeister wrote: >>>>> >>>>> I fixed the problem last night - it turned out to be a threading issue. >>>>> When I rapidly evaluated notebook cells (hitting shift-enter really >>>>> fast) the Python in Jupyter Notebook would crash as shown. >>>>> Multiple processes were calling the pzmq:send function at the same >>>>> time and the messages were becoming interleaved and garbled. >>>>> I wrapped a mutex around the function that calls pzmq:send multiple >>>>> times and the problem went away. >>>>> >>>> >>>> That sounds like a serious bug. Jupyter is supposed to handle >>>> simultaneous requests from multiple clients to the same kernel, afaik. This >>>> should not garble messages. >>>> >>>> Are you sure that the problem is on the sending side? Or could it be >>>> that your kernel expects messages in sequence, although the messaging >>>> protocol makes no guarantees about that? It's not impossible that you found >>>> a problem in Jupyter. But it seems strange that something so fundamental >>>> would have gone unnoticed until now. >>>> >>>> If you think that problem is in Jupyter, please open an issue and point >>>> us to the code where you had to implement the fix. >>>> >>>> best regards, >>>> Roland >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Project Jupyter" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/jupyter/c190dae1-cf7d-43d8-99ec-943ca48ba71e%40googlegroups.com >>> <https://groups.google.com/d/msgid/jupyter/c190dae1-cf7d-43d8-99ec-943ca48ba71e%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Project Jupyter" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/jupyter/657c9a40-4a61-421b-961b-2f36fa9cca93%40googlegroups.com > <https://groups.google.com/d/msgid/jupyter/657c9a40-4a61-421b-961b-2f36fa9cca93%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAOvn4qi%2BZxr%2BO40cSMV5vbW0wNx9PpnYaCTPo%2BWWw3hMvpXEhA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
