Re: [zeromq-dev] trying to understand zeromq high water mark behaviour
On Mon, Jan 27, 2014 at 09:06:42PM +, Shaobo Hou wrote: Thanks Pieter, I had another look at the output, and it does look like maybe the TCP buffer is involved somehow, the messages received by the slow worker were all from the beginning of the test sequence, so presumably they were sent out before the sockets started blocking due to HWM. Shaobo One thing that was mentioned was that one should be carefull about the timing. When you send out jobs and then one worker connects then it will grab all messages it can get. Then when the second worker connects there might be no work left. So you probably want to start all the worker and have them send an ACK once they are connected and only then start queuing up jobs or so. MfG Goswin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] trying to understand zeromq high water mark behaviour
On Mon, Jan 27, 2014 at 07:15:03AM -0600, Pieter Hintjens wrote: Hi, The dealer socket distributes messages equally to all its peers, and the HWM isn't relevant here. Even busy workers will receive messages. You want some form of load balancing. The Guide has examples of this (load balancing brokers). -Pieter Is it completly irelevant? I naively thought the dealer would send equally to all sockets untill they block. And on the other side the socket would read ahead until its recv HWM is met. So at least recv HWM messages get queued up and then whatever fits into the send and recv buffers the kernel has for the sockets and whatever remains of the last message is buffered in zmq internally. Or is that completly wrong? MfG Goswin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] trying to understand zeromq high water mark behaviour
It's not irrelevant but first, it defaults to 1000 and secondly there are buffers both at the sender and the receiver, as well as in TCP. On Jan 27, 2014 4:37 PM, Goswin von Brederlow goswin-...@web.de wrote: On Mon, Jan 27, 2014 at 07:15:03AM -0600, Pieter Hintjens wrote: Hi, The dealer socket distributes messages equally to all its peers, and the HWM isn't relevant here. Even busy workers will receive messages. You want some form of load balancing. The Guide has examples of this (load balancing brokers). -Pieter Is it completly irelevant? I naively thought the dealer would send equally to all sockets untill they block. And on the other side the socket would read ahead until its recv HWM is met. So at least recv HWM messages get queued up and then whatever fits into the send and recv buffers the kernel has for the sockets and whatever remains of the last message is buffered in zmq internally. Or is that completly wrong? MfG Goswin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] trying to understand zeromq high water mark behaviour
Thanks Pieter, I had another look at the output, and it does look like maybe the TCP buffer is involved somehow, the messages received by the slow worker were all from the beginning of the test sequence, so presumably they were sent out before the sockets started blocking due to HWM. Shaobo On 27 January 2014 16:01, Pieter Hintjens piet...@gmail.com wrote: It's not irrelevant but first, it defaults to 1000 and secondly there are buffers both at the sender and the receiver, as well as in TCP. On Jan 27, 2014 4:37 PM, Goswin von Brederlow goswin-...@web.de wrote: On Mon, Jan 27, 2014 at 07:15:03AM -0600, Pieter Hintjens wrote: Hi, The dealer socket distributes messages equally to all its peers, and the HWM isn't relevant here. Even busy workers will receive messages. You want some form of load balancing. The Guide has examples of this (load balancing brokers). -Pieter Is it completly irelevant? I naively thought the dealer would send equally to all sockets untill they block. And on the other side the socket would read ahead until its recv HWM is met. So at least recv HWM messages get queued up and then whatever fits into the send and recv buffers the kernel has for the sockets and whatever remains of the last message is buffered in zmq internally. Or is that completly wrong? MfG Goswin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
