my experience is: i start with 1 and always switch to 2. the main reason is with 1, you never know how big the queue is and eventually this will screw you.
On Jan 29, 2014, at 7:17 AM, Emil Stenqvist <[email protected]> wrote: > Hi > > I'm building a frontend service that will receive syslog messages (up to > 300,000 msgs/s, or 20MB/s of data), attach a timestamp to them and distribute > them evenly to a pool of workers for processing and storage. Once a worker > has received an event it doesn't send a reply, so from the frontend's > viewpoint it's send-and-forget. (That is — we can accept losing a few > messages in case of worker crashes.) > > The service is designed so that one thread listens for syslog data, which it > then sends to another thread that deals with workers. I've chosen the > PUSH-PULL pattern for frontend-to-worker communication. > > However, I also want the frontend to queue up incoming messages in case there > are no workers available, for which I'm pondering these two options: > > 1. Using ZeroMQ for queueing: e.g. an in-process DEALER-ROUTER socket > combination and where I adjust the HWM value to keep receiving and storing > messages while waiting for new workers. > 2. Implement the queue structure myself and just use ZeroMQ for shuffling > data between threads > > Option 1 seems justified here: > > http://lists.zeromq.org/pipermail/zeromq-dev/2010-March/002838.html > > where Martin says that they've seen in-memory queues of up to 200GB. Is this > a "good" approach? I'd like to see a) how many messages are in the queue, and > possibly b) start dropping old messages once the HWM is reached. Is there any > way to do this with ZeroMQ? Or should I handle this queueing behavior outside > of ZeroMQ? > > I'd appreciate all pointers and/or hints! > > Cheers > Emil > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev ----------------------- Andrew Hume 949-707-1964 (VO and best) 732-420-0907 (NJ) [email protected]
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
