Re: [zeromq-dev] CZMQ guaranteed delivery
This minimal test confirms it is the delay of the JSON handler that causes the problem (HWM): https://github.com/drbitboy/zmq_push_pull_test With minimal processing on the server side this code's client pushed over 100M (small) messages with none missed by the pulling server. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] CZMQ guaranteed delivery
Johnny Depp writes: > What is the meaning of 'HWM'? "High water mark". ZeroMQ also calls this "reaching mute state". In "man zmq_socket" you can see for each socket what the behavior is. -Brett. signature.asc Description: PGP signature ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Flow control with router socket
Thank you guys for the answers! That's plenty of information to get me going. Regards, Gyorgy Szekely On Thu, May 14, 2020 at 6:13 PM Brett Viren wrote: > I was going to reply similarly. Once a message is outside a socket, > it's out of ZeroMQ's hands and the responsibility of the application. > > This aspect is somewhat fresh in my own learning. The core issue here > (in my mind) is that the broker (router) severs any direct communication > between producer and consumer. Thus any back pressure from consumer to > producer must be communicated through the broker application. It may > sound obvious to say it like this but it took me a bit to internalize > this basic idea. > > If it's any help to look at some examples, I've recently implemented a > form of credit-based flow control following the ZeroMQ guide > description. It adds a runtime choice to use ROUTER/DEALER or > SERVER/CLIENT. The implementation is somewhat enmeshed in other choices > in my library but it could still serve as a general example. > > Actually, I implmented it twice, once in PyZMQ and once in cppzmq. > Barring some bug, the two should inter-operate. > > PyZMQ: > > https://github.com/brettviren/zio/tree/master/python/zio/flow > > cppzmq: > > https://github.com/brettviren/zio/blob/master/inc/zio/flow.hpp > https://github.com/brettviren/zio/blob/master/src/flow.cpp > > (flow.cpp is kind of a mess due to me still trying to understand > boost.sml which I use for the FSM. I welcome any critiques if anyone > cares to offer them.) > > One caveat, this code is Free to use but I don't consider it released. > I may still end up making drastic changes. > > But, to understand (this implementation of) credit-based flow control > this documentation and diagram may be more useful than the code: > > https://brettviren.github.io/zio/flow.html > > > And, one last comment: The Python implementation includes a "broker" > that does a little dance in order to hook up the "producer" and the > "consumer" ends in a way that allows each to think they are simple > clients and thus they can reuse the same "flow protocol" code in a > symmetric manner regadless of which role each endpoint plays. I have > some rambling about how the broker does this dance: > > https://brettviren.github.io/zio/flow-broker.html > > There are likely other, and simpler ways to do this. I'd be interested > to hear any ideas on that. The main issue is the broker needs to spawn > a "back end" consumer (producer) to service the "front end" producer > (consumer). The broker has no a'priori info about its front end clients > and what they may want from a back end so this spawning must be done as > a dymamic reaction to the first message from a "front end". That and my > desire for symmetric "flow protocol" code leads to this somewhat > contorted dance. > > > Cheers, > -Brett. > > > Kevin Sapper writes: > > > Hi Gyorgy, > > > > back-pressure is something very specific to your application. ZeroMQ > itself only implements blocking or dropping depending on the socket types > > you're using. > > > > To implement a credit-based flow have a look into the guide: > http://zguide.zeromq.org/page:all#Transferring-Files > > > > //Kevin > > > > Am Do., 14. Mai 2020 um 09:55 Uhr schrieb Gyorgy Szekely < > hodito...@gmail.com>: > > > > Hi All, > > > > I have a message router that routes messages between different types of > producers and consumers. Socket types are router (message > > router), dealer (producer, consumer). Currently if a consumer is not > fast enough messages start to queue up (dealer input queue, router > > output queue), eventually the router starts to drop messages to avoid > blocking. > > > > I would like to implement some kind of flow control. With plain TCP I > can rely on the built-in flow control: write blocks if consumer is > > overloaded, and producers can be sanctioned by not reading their socket. > > > > With ZMQ router/dealer I can detect if a consumer is slow by receiving > EAGAIN on send, but as far as I understand I can't "slow down" a > > specific producer, because router socket does fair queuing. So I have > to do application layer "stop sending" and "continue" messages and > > send them to specific producers... > > > > Is there any better way to do this? I would rather not reinvent the > wheel, TCP already has a sophisticated mechanism for this. > > > > Regards, > > Gyorgy Szekely > > > > ___ > > zeromq-dev mailing list > > zeromq-dev@lists.zeromq.org > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > ___ > > zeromq-dev mailing list > > zeromq-dev@lists.zeromq.org > > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev >
Re: [zeromq-dev] CZMQ guaranteed delivery
Brett, Thank you for the fast answer. The client/server is actually a push/pull pattern. What is the meaning of 'HWM'? Johnny From: Brett Viren Sent: Thursday, May 14, 2020 6:52 PM To: Johnny Depp Cc: zeromq-dev@lists.zeromq.org Subject: Re: [zeromq-dev] CZMQ guaranteed delivery Hi Johnny, Johnny Depp writes: > zstr_send(client, json_string); This returns an int which -1 will indicate some kind of error. zstr_send() wraps a lot of functionality so -1 can mean one of a few things went wrong. In any case, checking and acting on it would be part of guaranteeing delivery. > If the client sends 1000 messages, the server handles them with aplomb > and all is well. However, if the client sends 1 messages, about > 1000 are lost. The client pauses from time to time while sending, but > the server loses messages. I guess the change of behavior at 1000 is a likely indication of filling a socket buffer up to the default HWM of 1000. > I undrstand that while handling json messages, there is a delay. But > regardless of that, even if this would be a simple send and receive, > how can I guarantee delivery? Is there another mechanism or pattern > that I should employ? Adding more server reading threads would help > with this? Is there a way of buffering? You can increase the HWM. But, that tight send loop likely will always run faster than the recv loop. I guess HWM will always eventually be reached in this test regardless of the HWM. You don't mention the socket types being used. If sender "client" means "CLIENT" then the send loop should block on HWM which by itself should guarantee eventual delivery. So, I guess some other socket is being used. -Brett. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] CZMQ guaranteed delivery
Hi Johnny, Johnny Depp writes: > zstr_send(client, json_string); This returns an int which -1 will indicate some kind of error. zstr_send() wraps a lot of functionality so -1 can mean one of a few things went wrong. In any case, checking and acting on it would be part of guaranteeing delivery. > If the client sends 1000 messages, the server handles them with aplomb > and all is well. However, if the client sends 1 messages, about > 1000 are lost. The client pauses from time to time while sending, but > the server loses messages. I guess the change of behavior at 1000 is a likely indication of filling a socket buffer up to the default HWM of 1000. > I undrstand that while handling json messages, there is a delay. But > regardless of that, even if this would be a simple send and receive, > how can I guarantee delivery? Is there another mechanism or pattern > that I should employ? Adding more server reading threads would help > with this? Is there a way of buffering? You can increase the HWM. But, that tight send loop likely will always run faster than the recv loop. I guess HWM will always eventually be reached in this test regardless of the HWM. You don't mention the socket types being used. If sender "client" means "CLIENT" then the send loop should block on HWM which by itself should guarantee eventual delivery. So, I guess some other socket is being used. -Brett. signature.asc Description: PGP signature ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] CZMQ guaranteed delivery
Hi,
Here is my environment: RHEL/CENTOS 7, CZMQ version 4.2.1, ZMQ version 5.2.3,
GCC version 4.8.5.
I have a classic (as in examples) setting of a client-server:
client:
for (i= 0; i< 1000; i++_ {
zstr_send(client, json_string);
zsys_debug("Sent: %d %s", strlen(json_string), json_string);
}
server:
while(!zsys_interrupted) {
zsock_t *which = (zsock_t *) zpoller_wait (poller, 500);
if (which != NULL) {
char *message = zstr_recv (which);
zsys_debug("Message: %s", message);
handle_json(message);
zstr_free ();
}
}
If the client sends 1000 messages, the server handles them with aplomb and all
is well. However, if the client sends 1 messages, about 1000 are lost. The
client pauses from time to time while sending, but the server loses messages.
I undrstand that while handling json messages, there is a delay. But regardless
of that, even if this would be a simple send and receive, how can I guarantee
delivery? Is there another mechanism or pattern that I should employ? Adding
more server reading threads would help with this? Is there a way of buffering?
Thank you,
Johnny
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] How to limit the range for zmq_bind() in C++?
Hi Mike, "Alexander, Michael" writes: > I have been using zmq_bind(socket, “tcp://lo:*”) to bind to a random > unused port. I can find examples using other languages to use > random_port to limit the range of ports (e.g. only assign a port > between 3 and 31000). Is there a way to do this in C++? Maybe it's heavy handed, but I do this by manually looping over a port range, continuing until I no longer get an error from cppzmq's zmq::socket_t::bind(): https://github.com/brettviren/zio/blob/master/src/port.cpp#L44 Actually, now I check cppzmq and I'm not sure why I did this because I see cppzmq simply passes through the address to zmq_bind(). So, if "tcp://lo:*" works with libzmq, I think it should work with cppzmq. -Brett. signature.asc Description: PGP signature ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Flow control with router socket
I was going to reply similarly. Once a message is outside a socket, it's out of ZeroMQ's hands and the responsibility of the application. This aspect is somewhat fresh in my own learning. The core issue here (in my mind) is that the broker (router) severs any direct communication between producer and consumer. Thus any back pressure from consumer to producer must be communicated through the broker application. It may sound obvious to say it like this but it took me a bit to internalize this basic idea. If it's any help to look at some examples, I've recently implemented a form of credit-based flow control following the ZeroMQ guide description. It adds a runtime choice to use ROUTER/DEALER or SERVER/CLIENT. The implementation is somewhat enmeshed in other choices in my library but it could still serve as a general example. Actually, I implmented it twice, once in PyZMQ and once in cppzmq. Barring some bug, the two should inter-operate. PyZMQ: https://github.com/brettviren/zio/tree/master/python/zio/flow cppzmq: https://github.com/brettviren/zio/blob/master/inc/zio/flow.hpp https://github.com/brettviren/zio/blob/master/src/flow.cpp (flow.cpp is kind of a mess due to me still trying to understand boost.sml which I use for the FSM. I welcome any critiques if anyone cares to offer them.) One caveat, this code is Free to use but I don't consider it released. I may still end up making drastic changes. But, to understand (this implementation of) credit-based flow control this documentation and diagram may be more useful than the code: https://brettviren.github.io/zio/flow.html And, one last comment: The Python implementation includes a "broker" that does a little dance in order to hook up the "producer" and the "consumer" ends in a way that allows each to think they are simple clients and thus they can reuse the same "flow protocol" code in a symmetric manner regadless of which role each endpoint plays. I have some rambling about how the broker does this dance: https://brettviren.github.io/zio/flow-broker.html There are likely other, and simpler ways to do this. I'd be interested to hear any ideas on that. The main issue is the broker needs to spawn a "back end" consumer (producer) to service the "front end" producer (consumer). The broker has no a'priori info about its front end clients and what they may want from a back end so this spawning must be done as a dymamic reaction to the first message from a "front end". That and my desire for symmetric "flow protocol" code leads to this somewhat contorted dance. Cheers, -Brett. Kevin Sapper writes: > Hi Gyorgy, > > back-pressure is something very specific to your application. ZeroMQ itself > only implements blocking or dropping depending on the socket types > you're using. > > To implement a credit-based flow have a look into the guide: > http://zguide.zeromq.org/page:all#Transferring-Files > > //Kevin > > Am Do., 14. Mai 2020 um 09:55 Uhr schrieb Gyorgy Szekely > : > > Hi All, > > I have a message router that routes messages between different types of > producers and consumers. Socket types are router (message > router), dealer (producer, consumer). Currently if a consumer is not fast > enough messages start to queue up (dealer input queue, router > output queue), eventually the router starts to drop messages to avoid > blocking. > > I would like to implement some kind of flow control. With plain TCP I can > rely on the built-in flow control: write blocks if consumer is > overloaded, and producers can be sanctioned by not reading their socket. > > With ZMQ router/dealer I can detect if a consumer is slow by receiving > EAGAIN on send, but as far as I understand I can't "slow down" a > specific producer, because router socket does fair queuing. So I have to do > application layer "stop sending" and "continue" messages and > send them to specific producers... > > Is there any better way to do this? I would rather not reinvent the wheel, > TCP already has a sophisticated mechanism for this. > > Regards, > Gyorgy Szekely > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev signature.asc Description: PGP signature ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Flow control with router socket
Hi Gyorgy, back-pressure is something very specific to your application. ZeroMQ itself only implements blocking or dropping depending on the socket types you're using. To implement a credit-based flow have a look into the guide: http://zguide.zeromq.org/page:all#Transferring-Files //Kevin Am Do., 14. Mai 2020 um 09:55 Uhr schrieb Gyorgy Szekely < hodito...@gmail.com>: > Hi All, > > I have a message router that routes messages between different types of > producers and consumers. Socket types are router (message router), dealer > (producer, consumer). Currently if a consumer is not fast enough messages > start to queue up (dealer input queue, router output queue), eventually the > router starts to drop messages to avoid blocking. > > I would like to implement some kind of flow control. With plain TCP I can > rely on the built-in flow control: write blocks if consumer is overloaded, > and producers can be sanctioned by not reading their socket. > > With ZMQ router/dealer I can detect if a consumer is slow by receiving > EAGAIN on send, but as far as I understand I can't "slow down" a specific > producer, because router socket does fair queuing. So I have to do > application layer "stop sending" and "continue" messages and send them to > specific producers... > > Is there any better way to do this? I would rather not reinvent the wheel, > TCP already has a sophisticated mechanism for this. > > Regards, >Gyorgy Szekely > > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] How to limit the range for zmq_bind() in C++?
I have been using zmq_bind(socket, "tcp://lo:*") to bind to a random unused port. I can find examples using other languages to use random_port to limit the range of ports (e.g. only assign a port between 3 and 31000). Is there a way to do this in C++? Thanks, Mike Alexander ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Flow control with router socket
Hi All, I have a message router that routes messages between different types of producers and consumers. Socket types are router (message router), dealer (producer, consumer). Currently if a consumer is not fast enough messages start to queue up (dealer input queue, router output queue), eventually the router starts to drop messages to avoid blocking. I would like to implement some kind of flow control. With plain TCP I can rely on the built-in flow control: write blocks if consumer is overloaded, and producers can be sanctioned by not reading their socket. With ZMQ router/dealer I can detect if a consumer is slow by receiving EAGAIN on send, but as far as I understand I can't "slow down" a specific producer, because router socket does fair queuing. So I have to do application layer "stop sending" and "continue" messages and send them to specific producers... Is there any better way to do this? I would rather not reinvent the wheel, TCP already has a sophisticated mechanism for this. Regards, Gyorgy Szekely ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
