On 10/07/2012 11:00 AM, Chuck Remes wrote:
On Jul 10, 2012, at 10:31 AM, Edwin Amsler wrote:
On 10/07/2012 1:59 AM, Paul Colomiets wrote:
Hi Edwin,
The behavior is intentional for pub/sub sockets. If you'd have only
one subscriber you could use push/pull. Push sockets block when reach
high water mark, so are Req sockets.
The pub/sub sockets can't reliably block in general case because there
could be multiple subscribers, only one of which reaches high water
mark. Partially this comes from implementation: when you do zmq_send()
I never want them to block. That'd kind of break my program's state
machine. What I'd like, is to know when the high water mark was
reached with a send() error, and and EAGAIN errno so that I can
/choose/ whether or not I'd like to throw away the message. Right
now, the message is just silently cast away further down the stack as
far as I can tell.
I need to stress a particular detail here. The socket has a separate
HWM for *each* connected subscriber because each one has its own
outgoing queue. If you were to receive EAGAIN (or an error return
code) from zmq_send() when a subset of subscribers hit their HWM, your
published message would still be received by subscribers that have not
fallen behind. It does *not* drop the message for all subscribers,
just those that have entered the HWM exceptional state.
I don't think you can (even with a patch specific to your use case)
rely on this behavior to achieve your goal.
Well, that's depressing. I'm using "epgm://" here, and I know that
OpenPGM itself has no concept of connected clients. Are there still
individual queues for clients in this case?
zeromq starts to push message to the pipe for each connected
subscribers, if one of the pipes full, there is no way to rollback
messages already put into pipes, if it would it's not clear whether
single slow subscriber should stop the publisher. Also pgm sockets
can't have backpressure AFAIU.
Is there a way to detect if a pipe is full, and return a send() error
then? As I said, I feel that pub sockets should never block (that's
why I changed from using OpenPGM alone to ZeroMQ actually, though
I've found yet more reasons to stay).
Not at this time. However, there has been some work done by others to
let developers receive notification for specific internal library
events. I'm certain it would be possible to add this type of event as
a notification.
Also, PUB sockets never block.
You can try to implement the behavior as a socket option (it can't be
default behavior), but you have to be aware of the problems above (And
I don't know if core developers are willing to accept the patch).
That's more what I'm worried about. I'm quite confident there's no
way to handle this yet, and I feel that I'm not the only one who'd
like to have some sense of how fast a pub socket is truthfully
sending data. The implementation on the API side should be solid
since the send() error, EAGAIN method is how all ZMQ_NOBLOCK sockets
work already. It's just that send() doesn't error when a PUB socket
starts throwing away data (I know that's the case for sure on Windows
at least).
What we really need is a better mechanism for handling this type of
situation than HWM. One particular thought is described here:
http://unprotocols.org/blog:15
This can be implemented today using DEALER/ROUTER sockets.
You may also be interested in the Suicidal Snail pattern described in
the guide.
http://zguide.zeromq.org/page:all#Slow-Subscriber-Detection-Suicidal-Snail-Pattern
If you haven't read the guide yet, you should do so before getting too
much deeper into your problem space. It will save you a lot of time.
Good advice. I made it about three chapters in. I really need to go
digest it though.
cr
_______________________________________________
zeromq-dev mailing list
[email protected]
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
[email protected]
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
