Re: [zeromq-dev] Event-driven publish-subscribe model
Hello, I understand I have to use zmq_poll which is essentially event-driven. So that solves problem (1) and (3). However, I still need some comments on point (2) below. Thanks! On 26 March 2012 12:52, Rajalakshmi Iyer r...@blismobile.com wrote: Hello, I am working on a requirement whereby a process (say producer) needs to send out one-way messages to a variable number of processes (say consumers). The publish-subscribe model seemed good for this because the consumers will subscribe to messages from the producer. I tried using ZeroMQ to achieve this. However, I have a few problems with it: 1. The consumers have to continuously poll for messages. I would have the consumers to be notified when there is a new message. 2. There is a possibility of the producer queue being filled up. I would have liked the producer to remove messages from the queue based on some condition (say remove messages older than 5 seconds, or remove messages that have been read 5 times). I see ZMQ recommends using a HWM, but I would like more selective removal of elements from the buffer. 3. Since the consumers are polling and the messages are not removed from the queue, the consumers see duplicate messages till a new message comes in. I want the consumer to be notified only once per new message. I understand I may be using a wrong model (publish-subscribe may not be suitable). I have thought about using request-reply, but that doesn't work since the producer does not want to keep track of the number of consumers. Can anyone suggest a good alternative? Thanks in advance! Raj -- Twitter: @Blismobile http://twitter.com/#!/blismobile BlisMobile Media 32 Percy Street, London W1T 2DE www.blismobile.com [image: BlisMobile] http://www.blismobile.com/[image: Follow on Twitter]http://twitter.com/#!/blismobile[image: Blis Website] http://www.blismobile.com/ This communication is from BlisMobile Media, which is a trading name of Breeze Tech (UK) Ltd, a company registered in England and Wales with registered number 06455773. Its registered office is 32 Percy Street, London W1T 2DE, United Kingdom. This communication contains information that is confidential and may also be privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s), please (1) notify i...@blismobile.com by forwarding this email and delete all copies from your system and (2) note that disclosure, distribution, copying or use of this communication is strictly prohibited. Email communications cannot be guaranteed to be secure or free from error or viruses. All emails sent to or from a Blismobile email account are securely archived and stored by an external supplier. This email does not constitute a contractual agreement; such agreements are in specified contractual or Insertion Order (IO) form only and exclusively contain all the terms to which Breeze Tech )UK) Ltd will be bound. To the extent permitted by law, Breeze Tech (UK) Ltd does not accept any liability for use of or reliance on the contents of this email by any person save by the intended recipient(s) to the extent agreed in a contract or Insertion Order. Opinions, conclusions and other information in this email which have not been delivered by way of the business of Breeze Tech (UK) Ltd are neither given nor endorsed by it. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Event-driven publish-subscribe model
On Mar 26, 2012, at 10:24 AM, Rajalakshmi Iyer wrote: Hello, I understand I have to use zmq_poll which is essentially event-driven. So that solves problem (1) and (3). However, I still need some comments on point (2) below. Thanks! On 26 March 2012 12:52, Rajalakshmi Iyer r...@blismobile.com wrote: Hello, I am working on a requirement whereby a process (say producer) needs to send out one-way messages to a variable number of processes (say consumers). The publish-subscribe model seemed good for this because the consumers will subscribe to messages from the producer. I tried using ZeroMQ to achieve this. However, I have a few problems with it: The consumers have to continuously poll for messages. I would have the consumers to be notified when there is a new message. There is a possibility of the producer queue being filled up. I would have liked the producer to remove messages from the queue based on some condition (say remove messages older than 5 seconds, or remove messages that have been read 5 times). I see ZMQ recommends using a HWM, but I would like more selective removal of elements from the buffer. Since the consumers are polling and the messages are not removed from the queue, the consumers see duplicate messages till a new message comes in. I want the consumer to be notified only once per new message. I understand I may be using a wrong model (publish-subscribe may not be suitable). I have thought about using request-reply, but that doesn't work since the producer does not want to keep track of the number of consumers. Can anyone suggest a good alternative? Raj, I recommend that you read this posting: http://unprotocols.org/blog:15 That is a far better mechanism than the built-in HWM mechanism. Since it isn't built in to the library, you would have to add this logic at the application level. Since it *requires* two-way communication between the client and server, you cannot use the PUB/SUB sockets. You would need to use DEALER/ROUTER sockets to handle this communication. Yes, this will be a little more work but it is necessary for the kind of flexibility that your #2 issue requires. I hope this helps. cr PS - we would *love* it if someone made a patch to libzmq that added credit-based flow control as an option for all socket types. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Event-driven publish-subscribe model
On Mon, Mar 26, 2012 at 4:42 PM, Chuck Remes li...@chuckremes.com wrote: That is a far better mechanism than the built-in HWM mechanism. Since it isn't built in to the library, you would have to add this logic at the application level. Since it *requires* two-way communication between the client and server, you cannot use the PUB/SUB sockets. You would need to use DEALER/ROUTER sockets to handle this communication. Yes, this will be a little more work but it is necessary for the kind of flexibility that your #2 issue requires. I think you can still use PUB/SUB here, if you're doing at-least-one kind of flow control you just need credit from *someone*, not everyone, and you could do that with a simple pull or rep socket - as soon as anyone adds credit, then you push the message out the pub. This is actually pretty much just a send-on-ack sliding window, but with multiple clients. Doesn't need to be a library level, that's an easy bit of code in application land. By doing the checking in app land you also get the opportunity to be clever - for example, rolling up messages if they supersede each other, that kind of thing. Ian ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Event-driven publish-subscribe model
On Mar 26, 2012, at 11:08 AM, Ian Barber wrote: On Mon, Mar 26, 2012 at 4:42 PM, Chuck Remes li...@chuckremes.com wrote: That is a far better mechanism than the built-in HWM mechanism. Since it isn't built in to the library, you would have to add this logic at the application level. Since it *requires* two-way communication between the client and server, you cannot use the PUB/SUB sockets. You would need to use DEALER/ROUTER sockets to handle this communication. Yes, this will be a little more work but it is necessary for the kind of flexibility that your #2 issue requires. I think you can still use PUB/SUB here, if you're doing at-least-one kind of flow control you just need credit from *someone*, not everyone, and you could do that with a simple pull or rep socket - as soon as anyone adds credit, then you push the message out the pub. This is actually pretty much just a send-on-ack sliding window, but with multiple clients. Doesn't need to be a library level, that's an easy bit of code in application land. By doing the checking in app land you also get the opportunity to be clever - for example, rolling up messages if they supersede each other, that kind of thing. Yes, this is true. I answered assuming that the OP wanted to work with just a single set of sockets rather than having a side channel for communicating the credit flows. Either way will work, so it's up to the OP to figure out which way makes more sense to him. cr ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Event-driven publish-subscribe model
On Mon, Mar 26, 2012 at 5:53 PM, Chuck Remes li...@chuckremes.com wrote: Yes, this is true. I answered assuming that the OP wanted to work with just a single set of sockets rather than having a side channel for communicating the credit flows. Either way will work, so it's up to the OP to figure out which way makes more sense to him. cr Ack. I've found that I tend to lean to separating message flows over different topologies as I've done more stuff with ZMQ - it seems to fall out of a lot of different requirements. Definitely does cause more config management though to have to deal with extra sockets though, so a fair point. If 3.1 was an option and you wanted a tasty one socket set hack, you can communicate credit via subscribe messages back up stream :) Ian ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Event-driven publish-subscribe model
On Mar 26, 2012, at 11:58 AM, Ian Barber wrote: If 3.1 was an option and you wanted a tasty one socket set hack, you can communicate credit via subscribe messages back up stream :) Absolutely. I would love to add this kind of back channel communication so that we could have credit-based flow control built-in directly to libzmq for all socket types. I'm hoping that some enterprising individual runs with this idea as my C C++ skills are probably not up to the task of such an endeavor. cr ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
