[zeromq-dev] PUB-SUB with asynchronous response
Hello, I am working on an auction server which needs to forward each incoming request to several bidding services and conduct an auction based on the responses received from the bidders in real-time. One option is to employ asynchronous REQ-REP using ROUTER-DEALER against each bidder for every incoming request. As the number of bidder types increases, this option will cause the auction server to run out of sockets. Also note that each bidder, is actually a group of auto-scaling bidder instances behind a load balancer. Can ROUTER-DEALER work with a 3rd party load balancer in between? Another option is to use PUB-SUB, where the incoming request is published by the auction server and the bidders subscribe to the same, except the bidders now need to respond with their bids. One could potentially employ a cache to save the bids from all bidders and have the auction server query this cache. But that means that the auction server needs to necessarily wait for a max timeout before querying this cache, even though bidders would have responded way before the timeout. Are there any established ZeroMQ patterns that aim to solve such cases? Any advice is greatly appreciated. Thanks! -- This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Blis Ltd, a company registered in England and Wales with registered number 06455773. Its registered office is 5th Floor, 85 Tottenham Court Road, London, W1T 4TQ, United Kingdom. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Need help with a multi-threaded ZeroMQ application
Hello, I need help with the following use-case : A multi-threaded web service application runs ~ 1000 threads and each of those 1000 threads can potentially send a request to another service via a ZeroMQ publisher socket. An external service running a ZeroMQ subscriber socket subscribes to these messages, thus establishing an inverted PUB-SUB pattern. Now, within the web service application, what is the best option for creating the ZeroMQ publisher socket - - One ZeroMQ publisher socket per thread? - A pool of ZeroMQ sockets with memory barrier (mutexes etc) that allows for fewer tcp connections to the external service? - A single ZeroMQ publisher socket and each of the 1000 threads sending their messages using inproc? Note that not all of the 1000 threads will be publishing messages all the time, but there would still be significant publishing traffic. Would a single ZeroMQ socket be able to handle that? Any advice would be greatly helpful. Thanks! -- This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of BlisMedia Ltd, a company registered in England and Wales with registered number 06455773. Its registered office is 3rd Floor, 101 New Cavendish St, London, W1W 6XH, United Kingdom. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Fwd: Inverted PUB-SUB doesn't work
It worked after replacing a blocking recv with a poll + recv. Thanks! On Fri, Feb 6, 2015 at 7:38 AM, Rajalakshmi Iyer r...@blismedia.com wrote: :) Quite funny. However, even after adding - receiver.setsockopt(ZMQ_SUBSCRIBE, , 0); the result is the same. The subscriber is still not seeing the messages. Any ideas? On Thu, Feb 5, 2015 at 7:31 PM, Gregg Irwin gr...@pointillistic.com wrote: Hi Rajalakshmi, RI I have only just subscribed to the zeromq-dev mailing list. :) I mean in your code. You need to set the socket option with a filter for what messages you want it to subscribe to. From http://api.zeromq.org/4-1:zmq-socket: Initially a ZMQ_SUB socket is not subscribed to any messages, use the ZMQ_SUBSCRIBE option of zmq_setsockopt(3) to specify which messages to subscribe to. -- Gregg ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- @BlisMedia http://twitter.com/BlisMedia www.blismedia.com http://blismedia.com This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of BlisMedia Ltd, a company registered in England and Wales with registered number 06455773. Its registered office is 3rd Floor, 101 New Cavendish St, London, W1W 6XH, United Kingdom. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Fwd: Inverted PUB-SUB doesn't work
:) Quite funny. However, even after adding - receiver.setsockopt(ZMQ_SUBSCRIBE, , 0); the result is the same. The subscriber is still not seeing the messages. Any ideas? On Thu, Feb 5, 2015 at 7:31 PM, Gregg Irwin gr...@pointillistic.com wrote: Hi Rajalakshmi, RI I have only just subscribed to the zeromq-dev mailing list. :) I mean in your code. You need to set the socket option with a filter for what messages you want it to subscribe to. From http://api.zeromq.org/4-1:zmq-socket: Initially a ZMQ_SUB socket is not subscribed to any messages, use the ZMQ_SUBSCRIBE option of zmq_setsockopt(3) to specify which messages to subscribe to. -- Gregg ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- @BlisMedia http://twitter.com/BlisMedia www.blismedia.com http://blismedia.com This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of BlisMedia Ltd, a company registered in England and Wales with registered number 06455773. Its registered office is 3rd Floor, 101 New Cavendish St, London, W1W 6XH, United Kingdom. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Fwd: Inverted PUB-SUB doesn't work
I have only just subscribed to the zeromq-dev mailing list. Thanks! On Thu, Feb 5, 2015 at 6:54 PM, Gregg Irwin gr...@pointillistic.com wrote: Hi Rajalakshmi, I don't see where you're subscribing to any messages. -- Gregg ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- @BlisMedia http://twitter.com/BlisMedia www.blismedia.com http://blismedia.com This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of BlisMedia Ltd, a company registered in England and Wales with registered number 06455773. Its registered office is 3rd Floor, 101 New Cavendish St, London, W1W 6XH, United Kingdom. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Fwd: Inverted PUB-SUB doesn't work
Hello,
I have a use case where multiple publishers need to send messages to a
single subscriber. I have tried implementing the inverted PUB-SUB pattern
as follows -
#include zmq.hpp
#include iostream
int main()
{
try
{
int major, minor, patch;
zmq_version(major, minor, patch);
std::cout ZeroMQ version major . minor .
patch std::endl;
zmq::context_t ctx(1);
zmq::socket_t sender(ctx, ZMQ_PUB);
zmq::socket_t receiver(ctx, ZMQ_SUB);
receiver.bind(tcp://127.0.0.1:5557);
std::cout Subscriber bind std::endl;
sender.connect(tcp://127.0.0.1:5557);
std::cout Publisher connect std::endl;
zmq::message_t sendMsg;
memcpy(sendMsg.data(), (char*)(Hello!), 6);
sender.send(sendMsg);
std::cout Publisher send std::endl;
zmq::message_t recvMsg;
receiver.recv(recvMsg);
std::string message = std::string(static_castchar*(recvMsg.data()),
recvMsg.size());
std::cout Subscriber recv: message std::endl;
} catch (zmq::error_t err)
{
std::cout Error : err.what() std::endl;
}
}
The output of this is -
ZeroMQ version 4.0.4
Subscriber bind
Publisher connect
Publisher send
As can be seen, the subscriber is unable to receive the message sent by the
publisher and the recv() call blocks.
Any ideas on what might be going wrong would be useful.
Thanks!
--
@BlisMedia http://twitter.com/BlisMedia
www.blismedia.com http://blismedia.com
This email and any attachments to it may be confidential and are intended
solely
for the use of the individual to whom it is addressed. Any views or opinions
expressed are solely those of the author and do not necessarily represent
those of BlisMedia Ltd, a company registered in England and Wales with
registered number 06455773. Its registered office is 3rd Floor, 101 New
Cavendish St, London, W1W 6XH, United Kingdom.
If you are not the intended recipient of this email, you must neither take
any action based upon its contents, nor copy or show it to anyone. Please
contact the sender if you believe you have received this email in error.
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Event-driven publish-subscribe model
Hello, I am working on a requirement whereby a process (say producer) needs to send out one-way messages to a variable number of processes (say consumers). The publish-subscribe model seemed good for this because the consumers will subscribe to messages from the producer. I tried using ZeroMQ to achieve this. However, I have a few problems with it: 1. The consumers have to continuously poll for messages. I would have the consumers to be notified when there is a new message. 2. There is a possibility of the producer queue being filled up. I would have liked the producer to remove messages from the queue based on some condition (say remove messages older than 5 seconds, or remove messages that have been read 5 times). I see ZMQ recommends using a HWM, but I would like more selective removal of elements from the buffer. 3. Since the consumers are polling and the messages are not removed from the queue, the consumers see duplicate messages till a new message comes in. I want the consumer to be notified only once per new message. I understand I may be using a wrong model (publish-subscribe may not be suitable). I have thought about using request-reply, but that doesn't work since the producer does not want to keep track of the number of consumers. Can anyone suggest a good alternative? Thanks in advance! Raj -- Twitter: @Blismobile http://twitter.com/#!/blismobile BlisMobile Media 32 Percy Street, London W1T 2DE www.blismobile.com [image: BlisMobile] http://www.blismobile.com/[image: Follow on Twitter]http://twitter.com/#!/blismobile[image: Blis Website] http://www.blismobile.com/ This communication is from BlisMobile Media, which is a trading name of Breeze Tech (UK) Ltd, a company registered in England and Wales with registered number 06455773. Its registered office is 32 Percy Street, London W1T 2DE, United Kingdom. This communication contains information that is confidential and may also be privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s), please (1) notify i...@blismobile.com by forwarding this email and delete all copies from your system and (2) note that disclosure, distribution, copying or use of this communication is strictly prohibited. Email communications cannot be guaranteed to be secure or free from error or viruses. All emails sent to or from a Blismobile email account are securely archived and stored by an external supplier. This email does not constitute a contractual agreement; such agreements are in specified contractual or Insertion Order (IO) form only and exclusively contain all the terms to which Breeze Tech )UK) Ltd will be bound. To the extent permitted by law, Breeze Tech (UK) Ltd does not accept any liability for use of or reliance on the contents of this email by any person save by the intended recipient(s) to the extent agreed in a contract or Insertion Order. Opinions, conclusions and other information in this email which have not been delivered by way of the business of Breeze Tech (UK) Ltd are neither given nor endorsed by it. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Event-driven publish-subscribe model
Hello, I understand I have to use zmq_poll which is essentially event-driven. So that solves problem (1) and (3). However, I still need some comments on point (2) below. Thanks! On 26 March 2012 12:52, Rajalakshmi Iyer r...@blismobile.com wrote: Hello, I am working on a requirement whereby a process (say producer) needs to send out one-way messages to a variable number of processes (say consumers). The publish-subscribe model seemed good for this because the consumers will subscribe to messages from the producer. I tried using ZeroMQ to achieve this. However, I have a few problems with it: 1. The consumers have to continuously poll for messages. I would have the consumers to be notified when there is a new message. 2. There is a possibility of the producer queue being filled up. I would have liked the producer to remove messages from the queue based on some condition (say remove messages older than 5 seconds, or remove messages that have been read 5 times). I see ZMQ recommends using a HWM, but I would like more selective removal of elements from the buffer. 3. Since the consumers are polling and the messages are not removed from the queue, the consumers see duplicate messages till a new message comes in. I want the consumer to be notified only once per new message. I understand I may be using a wrong model (publish-subscribe may not be suitable). I have thought about using request-reply, but that doesn't work since the producer does not want to keep track of the number of consumers. Can anyone suggest a good alternative? Thanks in advance! Raj -- Twitter: @Blismobile http://twitter.com/#!/blismobile BlisMobile Media 32 Percy Street, London W1T 2DE www.blismobile.com [image: BlisMobile] http://www.blismobile.com/[image: Follow on Twitter]http://twitter.com/#!/blismobile[image: Blis Website] http://www.blismobile.com/ This communication is from BlisMobile Media, which is a trading name of Breeze Tech (UK) Ltd, a company registered in England and Wales with registered number 06455773. Its registered office is 32 Percy Street, London W1T 2DE, United Kingdom. This communication contains information that is confidential and may also be privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s), please (1) notify i...@blismobile.com by forwarding this email and delete all copies from your system and (2) note that disclosure, distribution, copying or use of this communication is strictly prohibited. Email communications cannot be guaranteed to be secure or free from error or viruses. All emails sent to or from a Blismobile email account are securely archived and stored by an external supplier. This email does not constitute a contractual agreement; such agreements are in specified contractual or Insertion Order (IO) form only and exclusively contain all the terms to which Breeze Tech )UK) Ltd will be bound. To the extent permitted by law, Breeze Tech (UK) Ltd does not accept any liability for use of or reliance on the contents of this email by any person save by the intended recipient(s) to the extent agreed in a contract or Insertion Order. Opinions, conclusions and other information in this email which have not been delivered by way of the business of Breeze Tech (UK) Ltd are neither given nor endorsed by it. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] ZeroMQ 2.1.11 - poll returns errno 11 on ZMQ_SUB sockets
Hello,
I am trying a simple PUBLISH/SUBSCRIBE setup with ZeroMQ 2.1.11.
Here is the server code:
#include zmq.hpp
#include iostream
using namespace std;
int main(void)
{
try
{
zmq::context_t context(1);
zmq::socket_t publisher(context, ZMQ_PUB);
publisher.bind(tcp://*:5556);
sleep(5);
while(1)
{
char* command = Command;
zmq::message_t msg(command, strlen(command), NULL, NULL);
if (!publisher.send(msg))
{
cout Unable to send message endl;
}
sleep(2);
}
}
catch (zmq::error_t e)
{
cout ZMQ Exception : e.what() endl;
}
return 0;
}
And this is the client code:
#include zmq.hpp
#include iostream
using namespace std;
int main()
{
try
{
zmq::context_t context(1);
cout Serving commands ... endl;
zmq::socket_t subscriber(context, ZMQ_SUB);
subscriber.connect(tcp://localhost:5556);
sleep(2);
while(1)
{
zmq::pollitem_t items[] = {
{(void*)subscriber, 0, ZMQ_POLLIN, 0}
};
int rc = zmq::poll(items, 1, 1);
cout zmq_poll(rc= errno ) endl;
if (items[0].revents ZMQ_POLLIN)
{
zmq::message_t command;
if (!subscriber.recv(command, ZMQ_NOBLOCK))
{
cout Failed to receive command endl;
}
cout Received command: (char*)command.data()
endl;
}
}
}
catch (zmq::error_t e)
{
cout ZMQ Exception : e.what() endl;
}
return 0;
}
The server is sending commands every 2 seconds over the ZMQ_PUB socket. The
client does zmq_poll on the ZMQ_SUB socket in a loop. However, the client
always sees errno 11 on a zmq_poll (which seems to indicate Resource
Temporarily Unavailable).
I have tried to figure out the cause from other posts in the mailing lists.
They seemed to indicate adding a sleep after the bind/connect phase before
starting to send/recv data. Even that has not helped.
Apologies, if this has been answered earlier, but it would be really great
if someone can help me out here.
Thanks,
Raj.
--
Twitter: @Blismobile http://twitter.com/#!/blismobile
BlisMobile Media
32 Percy Street,
London W1T 2DE
www.blismobile.com
[image: BlisMobile] http://www.blismobile.com/[image: Follow on
Twitter]http://twitter.com/#!/blismobile[image:
Blis Website] http://www.blismobile.com/
This communication is from BlisMobile Media, which is a trading name of
Breeze Tech (UK) Ltd, a company registered in England and Wales with
registered number 06455773. Its registered office is 32 Percy Street,
London W1T 2DE, United Kingdom.
This communication contains information that is confidential and may also
be privileged. It is for the exclusive use of the intended recipient(s). If
you are not the intended recipient(s), please (1) notify i...@blismobile.com by
forwarding this email and delete all copies from your system and (2) note
that disclosure, distribution, copying or use of this communication is
strictly prohibited. Email communications cannot be guaranteed to be secure
or free from error or viruses. All emails sent to or from a Blismobile
email account are securely archived and stored by an external supplier. This
email does not constitute a contractual agreement; such agreements are in
specified contractual or Insertion Order (IO) form only
and exclusively contain all the terms to which Breeze Tech )UK) Ltd will be
bound. To the extent permitted by law, Breeze Tech (UK) Ltd does not accept
any liability for use of or reliance on the contents of this email by any
person save by the intended recipient(s) to the extent agreed in a contract
or Insertion Order.
Opinions, conclusions and other information in this email which have not
been delivered by way of the business of Breeze Tech (UK) Ltd are neither
given nor endorsed by it.
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] ZeroMQ 2.1.11 - poll returns errno 11 on ZMQ_SUB sockets
Just an update,
I have also tried adding:
subscriber.setsockopt(ZMQ_SUBSCRIBE, , 0);
so the subscriber will subscribe to all messages.
But this did not help either.
Thanks,
Raj
On 26 March 2012 17:14, Rajalakshmi Iyer r...@blismobile.com wrote:
Hello,
I am trying a simple PUBLISH/SUBSCRIBE setup with ZeroMQ 2.1.11.
Here is the server code:
#include zmq.hpp
#include iostream
using namespace std;
int main(void)
{
try
{
zmq::context_t context(1);
zmq::socket_t publisher(context, ZMQ_PUB);
publisher.bind(tcp://*:5556);
sleep(5);
while(1)
{
char* command = Command;
zmq::message_t msg(command, strlen(command), NULL, NULL);
if (!publisher.send(msg))
{
cout Unable to send message endl;
}
sleep(2);
}
}
catch (zmq::error_t e)
{
cout ZMQ Exception : e.what() endl;
}
return 0;
}
And this is the client code:
#include zmq.hpp
#include iostream
using namespace std;
int main()
{
try
{
zmq::context_t context(1);
cout Serving commands ... endl;
zmq::socket_t subscriber(context, ZMQ_SUB);
subscriber.connect(tcp://localhost:5556);
sleep(2);
while(1)
{
zmq::pollitem_t items[] = {
{(void*)subscriber, 0, ZMQ_POLLIN, 0}
};
int rc = zmq::poll(items, 1, 1);
cout zmq_poll(rc= errno ) endl;
if (items[0].revents ZMQ_POLLIN)
{
zmq::message_t command;
if (!subscriber.recv(command, ZMQ_NOBLOCK))
{
cout Failed to receive command endl;
}
cout Received command: (char*)command.data()
endl;
}
}
}
catch (zmq::error_t e)
{
cout ZMQ Exception : e.what() endl;
}
return 0;
}
The server is sending commands every 2 seconds over the ZMQ_PUB socket.
The client does zmq_poll on the ZMQ_SUB socket in a loop. However, the
client always sees errno 11 on a zmq_poll (which seems to indicate Resource
Temporarily Unavailable).
I have tried to figure out the cause from other posts in the mailing
lists. They seemed to indicate adding a sleep after the bind/connect phase
before starting to send/recv data. Even that has not helped.
Apologies, if this has been answered earlier, but it would be really great
if someone can help me out here.
Thanks,
Raj.
--
Twitter: @Blismobile http://twitter.com/#!/blismobile
BlisMobile Media
32 Percy Street,
London W1T 2DE
www.blismobile.com
[image: BlisMobile] http://www.blismobile.com/[image: Follow on
Twitter]http://twitter.com/#!/blismobile[image:
Blis Website] http://www.blismobile.com/
This communication is from BlisMobile Media, which is a trading name of
Breeze Tech (UK) Ltd, a company registered in England and Wales with
registered number 06455773. Its registered office is 32 Percy Street,
London W1T 2DE, United Kingdom.
This communication contains information that is confidential and may also
be privileged. It is for the exclusive use of the intended recipient(s). If
you are not the intended recipient(s), please (1) notify i...@blismobile.com by
forwarding this email and delete all copies from your system and (2) note
that disclosure, distribution, copying or use of this communication is
strictly prohibited. Email communications cannot be guaranteed to be secure
or free from error or viruses. All emails sent to or from a Blismobile
email account are securely archived and stored by an external supplier. This
email does not constitute a contractual agreement; such agreements are in
specified contractual or Insertion Order (IO) form only
and exclusively contain all the terms to which Breeze Tech )UK) Ltd will be
bound. To the extent permitted by law, Breeze Tech (UK) Ltd does not accept
any liability for use of or reliance on the contents of this email by any
person save by the intended recipient(s) to the extent agreed in a contract
or Insertion Order.
Opinions, conclusions and other information in this email which have not
been delivered by way of the business of Breeze Tech (UK) Ltd are neither
given nor endorsed by it.
___
zeromq-dev mailing list
zeromq-dev
