I think the design of ZMQ is that each thread should have its own socket. But I don't know whether this would help with the issue you're seeing. And there may be an extra complication because the kernel binds the sockets (not connects), and I think only one socket can be bound to an endpoint at once.
Thomas On 19 December 2017 at 14:22, Christian Schafmeister < drschafmeis...@gmail.com> wrote: > Yes - this is how the pzmq (a Common Lisp wrapper library for the zmq > library) works as well. > Here is the code that I added the mutex to (it's Common Lisp but you > should be able to read it): > > https://github.com/clasp-developers/cl-jupyter/blob/ > master/src/message.lisp#L300 > > The calls to 'message-send' all pass the same sockets (iopub socket or > shell socket) - so if multiple threads call 'message-send' at the same time > there is nothing that distinguishes what thread is sending what parts. By > wrapping a mutex around the multiple pzmq:send calls it ensures that > different threads don't call pzmq:send at the same time. > > I don't know how threading works in Python - maybe this isn't an issue in > Python. > In Common Lisp I'm using the C pthread library for threading and threads > trample on each other if I don't take care like this. > > > > On Tuesday, December 19, 2017 at 7:37:12 AM UTC-5, takowl wrote: >> >> The Jupyter protocol relies on ZMQ 'multipart' messages. IIRC, a >> multipart message is a series of individual messages with the SNDMORE flag >> set on all but the last one. I don't know if those parts are meant to be >> separated out again if two threads are sending parts interleaved to the >> same endpoint. >> >> On 19 December 2017 at 12:28, Christian Schafmeister < >> drschaf...@gmail.com> wrote: >> >>> I'm certain it's a problem on my kernel side, it has to do with sending >>> messages to jupyter and not receiving them from jupyter. The messages are >>> send out using multiple calls to the pzmq:send function and when multiple >>> threads were doing this at the same time the messages got garbled. I >>> wrapped the sending code in a mutex and the problem went away. >>> I don't think it's a jupyter issue. >>> >>> On Tuesday, December 19, 2017 at 2:00:33 AM UTC-5, Roland Weber wrote: >>>> >>>> On Monday, December 18, 2017 at 3:00:23 PM UTC+1, Christian >>>> Schafmeister wrote: >>>>> >>>>> I fixed the problem last night - it turned out to be a threading issue. >>>>> When I rapidly evaluated notebook cells (hitting shift-enter really >>>>> fast) the Python in Jupyter Notebook would crash as shown. >>>>> Multiple processes were calling the pzmq:send function at the same >>>>> time and the messages were becoming interleaved and garbled. >>>>> I wrapped a mutex around the function that calls pzmq:send multiple >>>>> times and the problem went away. >>>>> >>>> >>>> That sounds like a serious bug. Jupyter is supposed to handle >>>> simultaneous requests from multiple clients to the same kernel, afaik. This >>>> should not garble messages. >>>> >>>> Are you sure that the problem is on the sending side? Or could it be >>>> that your kernel expects messages in sequence, although the messaging >>>> protocol makes no guarantees about that? It's not impossible that you found >>>> a problem in Jupyter. But it seems strange that something so fundamental >>>> would have gone unnoticed until now. >>>> >>>> If you think that problem is in Jupyter, please open an issue and point >>>> us to the code where you had to implement the fix. >>>> >>>> best regards, >>>> Roland >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Project Jupyter" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to jupyter+u...@googlegroups.com. >>> To post to this group, send email to jup...@googlegroups.com. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/jupyter/c190dae1-cf7d-43d8-99ec-943ca48ba71e%40googlegroups.com >>> <https://groups.google.com/d/msgid/jupyter/c190dae1-cf7d-43d8-99ec-943ca48ba71e%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Project Jupyter" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to jupyter+unsubscr...@googlegroups.com. > To post to this group, send email to jupyter@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/jupyter/657c9a40-4a61-421b-961b-2f36fa9cca93%40googlegroups.com > <https://groups.google.com/d/msgid/jupyter/657c9a40-4a61-421b-961b-2f36fa9cca93%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+unsubscr...@googlegroups.com. To post to this group, send email to jupyter@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAOvn4qi%2BZxr%2BO40cSMV5vbW0wNx9PpnYaCTPo%2BWWw3hMvpXEhA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
