On Thu, Dec 1, 2016 at 11:30 AM, Lineker Tomazeli <line...@gmail.com> wrote:

> Hi Luca,
>
> Thanks for answering. See my comments below.
>
> Lineker Tomazeli
> tomazeli.net
>
> On Thu, Dec 1, 2016 at 12:19 PM, Luca Boccassi <luca.bocca...@gmail.com>
> wrote:
>
>> On Thu, 2016-12-01 at 10:23 -0500, Lineker Tomazeli wrote:
>> > Hi guys,
>> >
>> > I have been reading zeromq list for a while now but is the first time
>> I'm
>> > asking something. So nice to meet you all :)
>> >
>> > In the zeromq documentation/book is very clear that we shouldn't be
>> > handling zeromq sockets from multiple threads without doing a memory
>> > barrier.
>> >
>> > *First question:*
>> >
>> > What is the recommended pattern to have one thread responsible for
>> > connecting, binding and polling messages from sockets. And have multiple
>> > thread sending messages.
>> >
>> > My first approach was to have a threadsafe producer/consumer queue where
>> > sender threads would push on the queue and the "zeromq" thread would
>> pick
>> > from queue and send through the wire. This works but I find it redundant
>> > and memory inefficient.
>> >
>> > My second approach (on which I used netmq) was to schedule a "send a
>> > message" task on the Poller. This guaranteed that the task ran in the
>> same
>> > thread as the poller.
>> >
>> >
>> > What would be the correct approach if I use clrzmq4 ?
>>
>> Why do you want to manage the socket on one thread and use from another?
>> What about just passing the endpoint string to each thread, so that they
>> can manage their own socket?
>>
>
> yes, I want to have one thread managing the socket and multiple thread
> doing work. When this multiple threads doing work need to send messages
> they say, "hey thread managing zeromq socket, here is message please
> deliver to me"
>
> having each thread manage their own socket is not feasible since these
> threads come and go when needed (thread pool). I don't want each of them to
> start connecting sockets if they only need to send one message.,
>
>
>> > *Second question :*
>> >
>> > What is the correct approach for a "zeromq" thread to notify other
>> threads
>> > that I new message was received ?
>> >  - raise an event on another thread?
>> >  - use a threadsafe producer/consumer queue to dump receiving message ?
>>
>> For inter-thread communication you could use an inproc:// endpoint,
>> which is essentially a lock-less in-memory queue. You can build any
>> model you want with it as a communication channel between threads. And
>> given they all take zmq_msg_t, you can just pass them along directly.
>>
>> If the tasks to distribute are all equal in cost/time, a simple and
>> common pattern is to use a round-robin type of socket to distribute
>> work. There are a lot of examples and suggestions in the zguide.
>>
>
> So would it be ok (performance, overhead and etc) for each thread that
> needs to send a message to connect using inproc://  to the "sender thread"
>
> worker thread ------inproc:// ---->
> worker thread ------inproc:// ---->     sender thread -----tcp:// ----->
> server
> worker thread ------inproc:// ---->
>
>
It's generally a bad idea to create and destroy lots of sockets, and also
create and destroy lots of threads, you'll have throughput problems due to
this, and as Luca mentioned, you'll run out of file descriptors. It can
take seconds to create a thread on a highly loaded system.

You described a good model in your first solution. Consider having a thread
worker pool which has long lived threads using some kind of semaphore to
allocate work to the work units. if you have a single socket where you get
the work, just use one thread to read all the data off that socket, then
package it up into some kind of message, and send it to the worker threads
either via ZMQ in-proc sockets, or some other mechanism. The workers each
have a socket and use it to send their response when they're done. Don't
kill those threads. Your problem description sounds like a classic
producer/consumer queue. Your producers are the long-lived threads which
take in ZMQ connections, and your consumers are your worker threads, and
many data structures exist which can do this. This may be less memory
efficient than a zero-copy approach, but it works fantastically, because
copying objects by value removes contention which you must solve by
locking.

When we were using ZMQ in production, I had a ZMQ Router taking in 100,000
- 200,000 PUB/SUB messages per second on a single ZMQ SUB socket using a
single thread (!!) and fanning out the resulting work to a pile of worker
threads, each of which did some work, and sent the result elsewhere. My
messages were a few hundred bytes in size, that's a 100's of megabytes
coming in per second, and my working memory was < 500MB. Memory is really
cheap these days :) ZMQ honestly blew my mind in how well it worked with
many publishers generating so much flow to a handful of subscribers.


>
>> > *Third question:*
>> >
>> > Zeromq is truly a great and fun framework to work with. I was wondering
>> why
>> > there not so much buzz about it? Do you know companies using in
>> production?
>>
>> Many companies do. We use it at Brocade in the vRouter, for example.
>>
>> --
>> Kind regards,
>> Luca Boccassi
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev@lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev@lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
_______________________________________________
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Reply via email to