Re: [zeromq-dev] PUB/SUB unreliabiliity

2014-06-16 Thread Gerry Steele
The issue I'm seeing is not indicative of having no subscribers and nothing to do with sending backed up data to late subscribers. That I'd something pub sub should never do. In my testing I ruled that out. Big chunks of messages go missing mid flow and then pick up again. There is no literature

Re: [zeromq-dev] PUB/SUB unreliabiliity

2014-06-16 Thread Pieter Hintjens
On Mon, Jun 16, 2014 at 9:10 AM, Gerry Steele gerry.ste...@gmail.com wrote: Big chunks of messages go missing mid flow and then pick up again. There is no literature that indicates that is expected behaviour. Right. The two plausible causes for this are (a) HWM overflows, and (b) temporary

Re: [zeromq-dev] How to view current SUBs and their filters from PUB side?

2014-06-16 Thread Goswin von Brederlow
No. I mean http://api.zeromq.org/4-0:zmq-socket-monitor MfG Goswin On Thu, Jun 12, 2014 at 04:55:58PM +0800, ? wrote: Hi MfG, Do you mean use zmq_proxy() to monitor inbound messages of a XPUB socket? - Zhichang 2014-06-12 16:06 GMT+08:00 Goswin von Brederlow

Re: [zeromq-dev] PUB/SUB unreliabiliity

2014-06-16 Thread Gerry Steele
Thanks Peter. I can't try this out till I get home but it is looking like hwm overflows. If you run the utilities you notice the drops start happening after precisely 1000 events in the first instance (which Is the default hwm). There was another largely ignored thread about this recently

Re: [zeromq-dev] PUB/SUB unreliabiliity

2014-06-16 Thread Pieter Hintjens
Gerry, can you provide a minimal test case that shows the behavior? Thanks. On Mon, Jun 16, 2014 at 12:49 PM, Gerry Steele gerry.ste...@gmail.com wrote: Thanks Peter. I can't try this out till I get home but it is looking like hwm overflows. If you run the utilities you notice the drops start

Re: [zeromq-dev] PUB/SUB unreliabiliity

2014-06-16 Thread Gerry Steele
In the patent email I have links to the minimal examples on gist.github.com Happy to open an issue and commit them later on if that's what you need. Thanks On 16 Jun 2014 14:43, Pieter Hintjens p...@imatix.com wrote: Gerry, can you provide a minimal test case that shows the behavior? Thanks.

Re: [zeromq-dev] PUB/SUB unreliabiliity

2014-06-16 Thread Pieter Hintjens
OK, just to double check, you're using ZeroMQ 4.0.x? In your test case (which I'm belatedly looking at), you use a uint64_t for the hwm values; it should be int. Probably not significant. On Mon, Jun 16, 2014 at 6:20 PM, Gerry Steele gerry.ste...@gmail.com wrote: In the patent email I have links

Re: [zeromq-dev] PUB/SUB unreliabiliity

2014-06-16 Thread Gerry Steele
Hi Pieter, you have struck on something there. Converting it to int seems to yield the correct behaviour. I guess the way setsockopt works type coercion doesn't happen. Embarrassing! But at least we got to the bottom of it. I was able to send billions of events without incurring loss.

Re: [zeromq-dev] zmq_ctx_term vs zmq_ctx_destroy vs zmq_ctx_shutdown

2014-06-16 Thread Indradhanush Gupta
On Mon, Jun 16, 2014 at 3:46 AM, MinRK benjami...@gmail.com wrote: On Sun, Jun 15, 2014 at 11:55 AM, Indradhanush Gupta indradhanush.gu...@gmail.com wrote: On Mon, Jun 16, 2014 at 12:17 AM, MinRK benjami...@gmail.com wrote: To add further confusion, PyZMQ distinguishes term from

Re: [zeromq-dev] PUB/SUB unreliabiliity

2014-06-16 Thread Thomas Rodgers
set_sockopt accepts arguments by void* and casts them, later to int* iif sizeof(int) == optvalen. There's no proper narrowing conversion, and no compiler warning. On Mon, Jun 16, 2014 at 12:36 PM, Gerry Steele gerry.ste...@gmail.com wrote: Hi Pieter, you have struck on something there.

Re: [zeromq-dev] PUB/SUB unreliabiliity

2014-06-16 Thread Pieter Hintjens
Hmm, it does check the size of the passed argument, and if that's wrong, returns an error (which you do check for). On Mon, Jun 16, 2014 at 7:36 PM, Gerry Steele gerry.ste...@gmail.com wrote: Hi Pieter, you have struck on something there. Converting it to int seems to yield the correct

[zeromq-dev] Deferreds in ZeroMQ

2014-06-16 Thread Indradhanush Gupta
Is there any way in which I could get a function to return deferreds in zeromq? Has anyone done it before? Thanks, -- Indradhanush Gupta ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] PUB/SUB unreliabiliity

2014-06-16 Thread Pieter Hintjens
And indeed, this code prints -1 as the return code: void *context = zmq_ctx_new (); void *publisher = zmq_socket (context, ZMQ_PUB); uint64_t rhwm = 0; int rc = zmq_setsockopt (publisher, ZMQ_SNDHWM, rhwm, sizeof (rhwm)); printf (RC=%d\n, rc); -Pieter On Mon, Jun 16, 2014 at

Re: [zeromq-dev] PUB/SUB unreliabiliity

2014-06-16 Thread Thomas Rodgers
GCC4.7(x86_64) - sizeof(int) = 4, sizeof(uint64_t) = 8 On Mon, Jun 16, 2014 at 1:03 PM, Pieter Hintjens p...@imatix.com wrote: Hmm, it does check the size of the passed argument, and if that's wrong, returns an error (which you do check for). On Mon, Jun 16, 2014 at 7:36 PM, Gerry Steele

Re: [zeromq-dev] PUB/SUB unreliabiliity

2014-06-16 Thread Gerry Steele
Thanks, there was also an error in my error handling thus why it was never flagged. I imagine its the same in my app code. uint64_t came from the cli argument handling lib thus why it was used over int. A lesson learned there. On 16 June 2014 19:13, Pieter Hintjens p...@imatix.com wrote: And

Re: [zeromq-dev] PUB/SUB unreliabiliity

2014-06-16 Thread Michi Henning
On 17.06.2014, at 8:11, Pieter Hintjens p...@imatix.com wrote: Indeed... I've had this hit more than once, a zmq_setsockopt that caused an error that wasn't being handled, with weird and expensive results down the line. Kind of makes you appreciate assertions more. CZMQ does this -- if a

Re: [zeromq-dev] PUB/SUB unreliabiliity

2014-06-16 Thread Thomas Rodgers
If so, debug/checked builds only please. On Mon, Jun 16, 2014 at 5:11 PM, Pieter Hintjens p...@imatix.com wrote: Indeed... I've had this hit more than once, a zmq_setsockopt that caused an error that wasn't being handled, with weird and expensive results down the line. Kind of makes you

Re: [zeromq-dev] PUB ZMQ_SNDHWM doesn't work

2014-06-16 Thread 余志昌
Hi Brian, From the later thread PUB/SUB unreliabiliity, I noticed that 4.0.x has resolved the issue PUB SNDHWM doesn't work. I verified it's true! Thanks, Zhichang 2014-06-12 18:22 GMT+08:00 Brian Knox bk...@digitalocean.com: Setting the HWM on a PUB socket to 0 would have the opposite