Yes - this is how the pzmq (a Common Lisp wrapper library for the zmq library) works as well. Here is the code that I added the mutex to (it's Common Lisp but you should be able to read it):
https://github.com/clasp-developers/cl-jupyter/blob/master/src/message.lisp#L300 The calls to 'message-send' all pass the same sockets (iopub socket or shell socket) - so if multiple threads call 'message-send' at the same time there is nothing that distinguishes what thread is sending what parts. By wrapping a mutex around the multiple pzmq:send calls it ensures that different threads don't call pzmq:send at the same time. I don't know how threading works in Python - maybe this isn't an issue in Python. In Common Lisp I'm using the C pthread library for threading and threads trample on each other if I don't take care like this. On Tuesday, December 19, 2017 at 7:37:12 AM UTC-5, takowl wrote: > > The Jupyter protocol relies on ZMQ 'multipart' messages. IIRC, a multipart > message is a series of individual messages with the SNDMORE flag set on all > but the last one. I don't know if those parts are meant to be separated out > again if two threads are sending parts interleaved to the same endpoint. > > On 19 December 2017 at 12:28, Christian Schafmeister <[email protected] > <javascript:>> wrote: > >> I'm certain it's a problem on my kernel side, it has to do with sending >> messages to jupyter and not receiving them from jupyter. The messages are >> send out using multiple calls to the pzmq:send function and when multiple >> threads were doing this at the same time the messages got garbled. I >> wrapped the sending code in a mutex and the problem went away. >> I don't think it's a jupyter issue. >> >> On Tuesday, December 19, 2017 at 2:00:33 AM UTC-5, Roland Weber wrote: >>> >>> On Monday, December 18, 2017 at 3:00:23 PM UTC+1, Christian Schafmeister >>> wrote: >>>> >>>> I fixed the problem last night - it turned out to be a threading issue. >>>> When I rapidly evaluated notebook cells (hitting shift-enter really >>>> fast) the Python in Jupyter Notebook would crash as shown. >>>> Multiple processes were calling the pzmq:send function at the same time >>>> and the messages were becoming interleaved and garbled. >>>> I wrapped a mutex around the function that calls pzmq:send multiple >>>> times and the problem went away. >>>> >>> >>> That sounds like a serious bug. Jupyter is supposed to handle >>> simultaneous requests from multiple clients to the same kernel, afaik. This >>> should not garble messages. >>> >>> Are you sure that the problem is on the sending side? Or could it be >>> that your kernel expects messages in sequence, although the messaging >>> protocol makes no guarantees about that? It's not impossible that you found >>> a problem in Jupyter. But it seems strange that something so fundamental >>> would have gone unnoticed until now. >>> >>> If you think that problem is in Jupyter, please open an issue and point >>> us to the code where you had to implement the fix. >>> >>> best regards, >>> Roland >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Project Jupyter" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/jupyter/c190dae1-cf7d-43d8-99ec-943ca48ba71e%40googlegroups.com >> >> <https://groups.google.com/d/msgid/jupyter/c190dae1-cf7d-43d8-99ec-943ca48ba71e%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/657c9a40-4a61-421b-961b-2f36fa9cca93%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
