> > 3. Another situation is to have a single “main” thread setup all the > sockets at startup and then hand them off to other threads (and never touch > them again). Strictly speaking you may only need a memory barrier, but > using a mutex in this case ensures that you don’t try to use the socket > concurrently from more than one thread. >
The pattern of having a main thread set up sockets first, then handing them off to worker threads is, IMO, very sound, but, lets perform this small thought experiement - We use a mutex (I'm a C++ dev, so), std::mutex<socket>, concretely. Thread A acquires the mutex, initializes the socket, releases the mutex (ideally using one of the RAII scoped locking adapters). Thread B is then started, and acquires and, since we assume no further access from Thread A, simply holds the lock for the duration of thread_main(). If, thread A (or any other thread) attempted to acquire that mutex, it would deadlock, as Thread B now has ownership. This can be properly described as poor program behavior, from a contractual standpoint, you violated a fundamental assumption, so this is "undefined behavior" for the program. I don't think I agree with the "just use a mutex, it's easy and safe" logic in this design pattern. The requirement is for a "full memory fence". This is to ensure that Thread A completes all of it's socket setup (concretely in terms of ordered memory writes) before any other thread, in this case thread B, can use that socket (in terms of ordered memory read/writes). Issuing just the memory fence before ThreadB starts is sufficient to guarantee this ordering. If ThreadA or any other thread besides ThreadB accesses the socket after the memory fence, that is also undefined behavior. If instead, you always acquire and release the mutex for each use, then you wouldn't have the deadlocking issue, but you are then *ALWAYS* paying for a pair of atomic operations you didn't really need (atomic operations are most decidedly not free), because you weren't comfortable understanding your program's lifetime and data usage model. On Sat, May 12, 2018 at 4:08 PM, Bill Torpey <wallstp...@gmail.com> wrote: > 1. FWIW, my personal experience has been that it is safe to share a > ZMQ_PUB socket between threads if protected with a mutex. Basically the > sequence is lock/send/unlock. > > The alternative is to have an internal zmq_proxy with inproc on one side > and something like TCP on the other side, which is an extra send/receive > hop. I have not found that to be necessary, and I have done a *lot* of > testing (which has found several other edge cases). > > FWIW the docs specifically bless this approach (http://zeromq.org/area:faq > ): > > For those situations where a dedicated socket per thread is infeasible, a > socket may be shared *if and only if* each thread executes a full memory > barrier before accessing the socket. Most languages support a Mutex or > Spinlock which will execute the full memory barrier on your behalf. > > and here (http://zguide.zeromq.org/page:all#Multithreading-with-ZeroMQ): > > Technically it's possible to migrate a socket from one thread to another > but it demands skill. > > > I have not tried doing that with ZMQ_SUB sockets — there is really no > point, as a sigle thread sitting in a zmq_poll call on multiple sockets is > probably the cleanest design anyway. > > > 2. I can definitely testify that trying to share a ZMQ_SUB socket > *without* mutexes is guaranteed to crash, and that trying to share a > ZMQ_SUB socket *with* mutexes is almost impossible to do without > deadlocking. > > > 3. Another situation is to have a single “main” thread setup all the > sockets at startup and then hand them off to other threads (and never touch > them again). Strictly speaking you may only need a memory barrier, but > using a mutex in this case ensures that you don’t try to use the socket > concurrently from more than one thread. > > > 4. Last but not least, if you’re planning on using inproc sockets for > inter-thread communication, you may find this helpful: > https://github.com/WallStProg/zmqtests/tree/master/threads > > > On May 8, 2018, at 3:58 PM, Thomas Rodgers <rodg...@twrodgers.com> wrote: > > No, not really. I have designed my zmq based systems to either - > > * orchestrate connections over the inproc transport > * performed the delicate dance of explicitly passing a fresh socket to a > fresh thread the way that czmq actors > <http://czmq.zeromq.org/manual:zactor> work, > as implemented in azmq > <https://github.com/zeromq/azmq/blob/master/azmq/detail/actor_service.hpp> > > On Mon, May 7, 2018 at 11:58 AM, Attila Magyari <att...@gmail.com> wrote: > >> This sounds promising. My concern is if there are some background threads >> internally in the ZMQ library, which would use the socket out of sync with >> mine, or anything else that I'm not thinking about. I can ensure that the >> socket won't be used from multiple threads. >> >> Do you have any experience with this, or only theory? >> >> Thank you, >> Attila >> >> >> On Mon, May 7, 2018 at 5:49 PM Thomas Rodgers <rodg...@twrodgers.com> >> wrote: >> >>> The zmq docs say you need to ensure a ‘full fence’ a mutex is one way of >>> achieving that. >>> >>> If you are using C++ (-std=c++11 or later) and you can guarantee there >>> is no racy usage from another thread, you can issue a memory fence using - >>> >>> std::atomic_thread_fence >>> >>> On Mon, May 7, 2018 at 8:42 AM Attila Magyari <att...@gmail.com> wrote: >>> >>>> Hello, >>>> >>>> I know that zmq sockets are not thread safe. However I can't think of a >>>> good design (explained below) without using the socket in several threads. >>>> What I want to do is connect the socket in a thread, and then use it in a >>>> different one until the end, and never in parallel between several threads. >>>> I will guard the access with a mutex as well. Do you think this will be >>>> safe? >>>> >>>> Just in case someone might have a better idea for my design, here is >>>> what I'm trying to do: >>>> I have an application which starts another process, passes a port to >>>> the newly created process, so it can connect back to the main application >>>> through a ZMQ_PAIR socket. Both processes are local to the machine. The >>>> main application will request from the second one, but the second one can >>>> initiate messages without an explicit request as well. Do you have a nicer >>>> design to achieve something like this? >>>> >>>> Thank you in advance! >>>> >>> _______________________________________________ >>>> zeromq-dev mailing list >>>> zeromq-dev@lists.zeromq.org >>>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>> >>> _______________________________________________ >>> zeromq-dev mailing list >>> zeromq-dev@lists.zeromq.org >>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >> >> _______________________________________________ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > _______________________________________________ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > _______________________________________________ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > >
_______________________________________________ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
