> On May 12, 2018, at 7:50 PM, Thomas Rodgers <[email protected]> wrote: > > 3. Another situation is to have a single “main” thread setup all the sockets > at startup and then hand them off to other threads (and never touch them > again). Strictly speaking you may only need a memory barrier, but using a > mutex in this case ensures that you don’t try to use the socket concurrently > from more than one thread. > > The pattern of having a main thread set up sockets first, then handing them > off to worker threads is, IMO, very sound, but, lets perform this small > thought experiement - > > We use a mutex (I'm a C++ dev, so), std::mutex<socket>, concretely. Thread A > acquires the mutex, initializes the socket, releases the mutex (ideally using > one of the RAII scoped locking adapters). Thread B is then started, and > acquires and, since we assume no further access from Thread A, simply holds > the lock for the duration of thread_main(). If, thread A (or any other > thread) attempted to acquire that mutex, it would deadlock, as Thread B now > has ownership. > > This can be properly described as poor program behavior, from a contractual > standpoint, you violated a fundamental assumption, so this is "undefined > behavior" for the program. I don't think I agree with the "just use a mutex, > it's easy and safe" logic in this design pattern. > > The requirement is for a "full memory fence". This is to ensure that Thread A > completes all of it's socket setup (concretely in terms of ordered memory > writes) before any other thread, in this case thread B, can use that socket > (in terms of ordered memory read/writes). Issuing just the memory fence > before ThreadB starts is sufficient to guarantee this ordering. If ThreadA or > any other thread besides ThreadB accesses the socket after the memory fence, > that is also undefined behavior.
The thing the mutex buys you is that allows you to *detect* when you’ve broken the contract (by blocking) — the fence does not. > > If instead, you always acquire and release the mutex for each use, then you > wouldn't have the deadlocking issue, but you are then *ALWAYS* paying for a > pair of atomic operations you didn't really need (atomic operations are most > decidedly not free), because you weren't comfortable understanding your > program's lifetime and data usage model. Well, acquiring an uncontested mutex is actually pretty cheap. It’s when there’s a lot of contention on the mutex that it gets expensive. Still, I wouldn’t suggest using a mutex to share sockets between threads, except in the case of a ZMQ_PUB socket I mentioned earlier. > > > > > > On Sat, May 12, 2018 at 4:08 PM, Bill Torpey <[email protected] > <mailto:[email protected]>> wrote: > 1. FWIW, my personal experience has been that it is safe to share a ZMQ_PUB > socket between threads if protected with a mutex. Basically the sequence is > lock/send/unlock. > > The alternative is to have an internal zmq_proxy with inproc on one side and > something like TCP on the other side, which is an extra send/receive hop. I > have not found that to be necessary, and I have done a *lot* of testing > (which has found several other edge cases). > > FWIW the docs specifically bless this approach (http://zeromq.org/area:faq > <http://zeromq.org/area:faq>): >> For those situations where a dedicated socket per thread is infeasible, a >> socket may be shared if and only if each thread executes a full memory >> barrier before accessing the socket. Most languages support a Mutex or >> Spinlock which will execute the full memory barrier on your behalf. >> > > and here (http://zguide.zeromq.org/page:all#Multithreading-with-ZeroMQ > <http://zguide.zeromq.org/page:all#Multithreading-with-ZeroMQ>): > >> Technically it's possible to migrate a socket from one thread to another but >> it demands skill. > > > I have not tried doing that with ZMQ_SUB sockets — there is really no point, > as a sigle thread sitting in a zmq_poll call on multiple sockets is probably > the cleanest design anyway. > > > 2. I can definitely testify that trying to share a ZMQ_SUB socket *without* > mutexes is guaranteed to crash, and that trying to share a ZMQ_SUB socket > *with* mutexes is almost impossible to do without deadlocking. > > > 3. Another situation is to have a single “main” thread setup all the sockets > at startup and then hand them off to other threads (and never touch them > again). Strictly speaking you may only need a memory barrier, but using a > mutex in this case ensures that you don’t try to use the socket concurrently > from more than one thread. > > > 4. Last but not least, if you’re planning on using inproc sockets for > inter-thread communication, you may find this helpful: > https://github.com/WallStProg/zmqtests/tree/master/threads > <https://github.com/WallStProg/zmqtests/tree/master/threads> > > >> On May 8, 2018, at 3:58 PM, Thomas Rodgers <[email protected] >> <mailto:[email protected]>> wrote: >> >> No, not really. I have designed my zmq based systems to either - >> >> * orchestrate connections over the inproc transport >> * performed the delicate dance of explicitly passing a fresh socket to a >> fresh thread the way that czmq actors <http://czmq.zeromq.org/manual:zactor> >> work, >> as implemented in azmq >> <https://github.com/zeromq/azmq/blob/master/azmq/detail/actor_service.hpp> >> >> On Mon, May 7, 2018 at 11:58 AM, Attila Magyari <[email protected] >> <mailto:[email protected]>> wrote: >> This sounds promising. My concern is if there are some background threads >> internally in the ZMQ library, which would use the socket out of sync with >> mine, or anything else that I'm not thinking about. I can ensure that the >> socket won't be used from multiple threads. >> >> Do you have any experience with this, or only theory? >> >> Thank you, >> Attila >> >> >> On Mon, May 7, 2018 at 5:49 PM Thomas Rodgers <[email protected] >> <mailto:[email protected]>> wrote: >> The zmq docs say you need to ensure a ‘full fence’ a mutex is one way of >> achieving that. >> >> If you are using C++ (-std=c++11 or later) and you can guarantee there is no >> racy usage from another thread, you can issue a memory fence using - >> >> std::atomic_thread_fence >> >> On Mon, May 7, 2018 at 8:42 AM Attila Magyari <[email protected] >> <mailto:[email protected]>> wrote: >> Hello, >> >> I know that zmq sockets are not thread safe. However I can't think of a good >> design (explained below) without using the socket in several threads. What I >> want to do is connect the socket in a thread, and then use it in a different >> one until the end, and never in parallel between several threads. I will >> guard the access with a mutex as well. Do you think this will be safe? >> >> Just in case someone might have a better idea for my design, here is what >> I'm trying to do: >> I have an application which starts another process, passes a port to the >> newly created process, so it can connect back to the main application >> through a ZMQ_PAIR socket. Both processes are local to the machine. The main >> application will request from the second one, but the second one can >> initiate messages without an explicit request as well. Do you have a nicer >> design to achieve something like this? >> >> Thank you in advance! >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] <mailto:[email protected]> >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >> <https://lists.zeromq.org/mailman/listinfo/zeromq-dev> >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] <mailto:[email protected]> >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >> <https://lists.zeromq.org/mailman/listinfo/zeromq-dev> >> >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] <mailto:[email protected]> >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >> <https://lists.zeromq.org/mailman/listinfo/zeromq-dev> >> >> >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] <mailto:[email protected]> >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >> <https://lists.zeromq.org/mailman/listinfo/zeromq-dev> > > > _______________________________________________ > zeromq-dev mailing list > [email protected] <mailto:[email protected]> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > <https://lists.zeromq.org/mailman/listinfo/zeromq-dev> > > > _______________________________________________ > zeromq-dev mailing list > [email protected] > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________ zeromq-dev mailing list [email protected] https://lists.zeromq.org/mailman/listinfo/zeromq-dev
