Hi I'm building a frontend service that will receive syslog messages (up to 300,000 msgs/s, or 20MB/s of data), attach a timestamp to them and distribute them evenly to a pool of workers for processing and storage. Once a worker has received an event it doesn't send a reply, so from the frontend's viewpoint it's send-and-forget. (That is -- we can accept losing a few messages in case of worker crashes.)
The service is designed so that one thread listens for syslog data, which it then sends to another thread that deals with workers. I've chosen the PUSH-PULL pattern for frontend-to-worker communication. However, I also want the frontend to queue up incoming messages in case there are no workers available, for which I'm pondering these two options: 1. Using ZeroMQ for queueing: e.g. an in-process DEALER-ROUTER socket combination and where I adjust the HWM value to keep receiving and storing messages while waiting for new workers. 2. Implement the queue structure myself and just use ZeroMQ for shuffling data between threads Option 1 seems justified here: http://lists.zeromq.org/pipermail/zeromq-dev/2010-March/002838.html where Martin says that they've seen in-memory queues of up to 200GB. Is this a "good" approach? I'd like to see a) how many messages are in the queue, and possibly b) start dropping old messages once the HWM is reached. Is there any way to do this with ZeroMQ? Or should I handle this queueing behavior outside of ZeroMQ? I'd appreciate all pointers and/or hints! Cheers Emil
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
