Hi Jarred, > Thus I would just estimate how many threads are going to use 0MQ in > parallel (10, 100, 1000?) and set the context accordingly. > > > You start getting invalid value when you set app_threads above 63. Line > 226 in zmq.cpp. I imagine there is a good reason for this limitation?
It's because the notifications from I/O thread to application thread are sent via lock-free polling mechanism (ypollset_t). Lock-free algorithms are based on atomic operations and atomic operations work on a single word of data (32 bits on 32-bit platform, 64 bits on 64 bit platform. Thus, maximum number of flags that can be signaled using ypollet_t is 64 (or 32 depending on platform). One bit is used for internal purposes so there are just 63 bits left to signal individual threads. Now, there's an option to use simple socketpair for signaling between threads instead of ypollset_t. Doing so would add ~2us to the latency, however, it would allow us to move arbitrary number of bits between threads. Actually this option is already implemented. When you init 0MQ library with ZMQ_POLL flag, what happens is that that ypollset_t notification mechanism is replaced by a socket-based notification mechanism (fd_signaler_t). Currently, fd_signealer_t passes single byte between threads, meaning that there can be at most 256 threads. This can be changed to 2 bytes giving max. 65536 threads. So, the hack to make it work for up to 256 threads is: 1. Use ZMQ_POLL flag when initialising the context. 2. Remove the check in zmq_init that complains about more than 63 threads. As for the systematic solution I am thinking of ditching the lock-free polling (ypollset_t) altogether. That way we can trade 2us of extra latency for: 1. No need for ZMQ_POLL flag. 2. Arbitrary number of threads can be used. Thoughts? Martin _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
