Re: [zeromq-dev] general strategy for sending data over epgm
On 3 December 2012 17:19, Justin Karneges jus...@affinix.com wrote: Slightly OT, but I wonder if RATE might be a useful feature to allow for all PUB sockets (and maybe just all sockets), rather than restricted to pgm only. It seems to me that you want to use RATE when you need to slow down sending but can't depend on receiver feedback. This is a problem at the pattern level, not the transport level. Note that the RATE limits what is forwarded onto the wire, not what you are feeding into 0mq. If you send fast enough with no high watermark you will consume a significant amount of memory to adverse affects. An application rate limited is recommended but may be more suitable as an optional layer above 0mq API. -- Steve-o ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] general strategy for sending data over epgm
On 3 December 2012 15:39, mlist user mlist.user8...@gmail.com wrote:
Hi list,
First post here. I would like to figure out how you guys manage to
send data at certain frequency.
For e.g., if I send small'ish {k,v} over epgm over 100mb link with
0.60ms RTT, i need to sleep only 2 second per send.
But, this is not applicable to large {k, v} over epgm. I need to sleep
more than 12 second per send!
Can you confirm reliability is working? Wireshark can be used to monitor
the protocol. If you are sending too fast, e.g. 60% capacity of the
network you may see packet loss and thus NAKs from the receivers and RDATA
packets from the senders. Monitor the network capacity to ensure that both
the original data and repair data are not saturating the network.
--
Steve-o
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] general strategy for sending data over epgm
i'm not sure what the issue is.
if the rate is small and the messages are large, then yes, you need to sleep
large amounts.
On Dec 3, 2012, at 1:39 PM, mlist user wrote:
Hi list,
First post here. I would like to figure out how you guys manage to
send data at certain frequency.
For e.g., if I send small'ish {k,v} over epgm over 100mb link with
0.60ms RTT, i need to sleep only 2 second per send.
But, this is not applicable to large {k, v} over epgm. I need to sleep
more than 12 second per send!
Config:
epgm on zmq 3.2.2 / pyzmq (trunk).
Thanks!
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
---
Andrew Hume
623-551-2845 (VO and best)
973-236-2014 (NJ)
and...@research.att.com
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] general strategy for sending data over epgm
Hi Andrew, On 4 December 2012 07:47, Andrew Hume and...@research.att.com wrote: i'm not sure what the issue is. if the rate is small and the messages are large, then yes, you need to sleep large amounts. How would you figure out you need to sleep X second? [ ... ] ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] general strategy for sending data over epgm
On Monday, December 03, 2012 09:59:12 PM Ian Barber wrote: On Mon, Dec 3, 2012 at 8:50 PM, mlist user mlist.user8...@gmail.com wrote: Hi Andrew, On 4 December 2012 07:47, Andrew Hume and...@research.att.com wrote: i'm not sure what the issue is. if the rate is small and the messages are large, then yes, you need to sleep large amounts. How would you figure out you need to sleep X second? Are you setting the RATE sockopt yourself and then trying to manage the pace at which you send on top of that? Slightly OT, but I wonder if RATE might be a useful feature to allow for all PUB sockets (and maybe just all sockets), rather than restricted to pgm only. It seems to me that you want to use RATE when you need to slow down sending but can't depend on receiver feedback. This is a problem at the pattern level, not the transport level. Justin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] general strategy for sending data over epgm
Hi Ian, On 4 December 2012 08:59, Ian Barber ian.bar...@gmail.com wrote: On Mon, Dec 3, 2012 at 8:50 PM, mlist user mlist.user8...@gmail.com wrote: Hi Andrew, On 4 December 2012 07:47, Andrew Hume and...@research.att.com wrote: i'm not sure what the issue is. if the rate is small and the messages are large, then yes, you need to sleep large amounts. How would you figure out you need to sleep X second? Are you setting the RATE sockopt yourself and then trying to manage the pace at which you send on top of that? I was. Then I stopped using RATE. Now, I am again. Thanks. [ ... ] ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] general strategy for sending data over epgm
On 03/12/12 23:19, Justin Karneges wrote: Slightly OT, but I wonder if RATE might be a useful feature to allow for all PUB sockets (and maybe just all sockets), rather than restricted to pgm only. It seems to me that you want to use RATE when you need to slow down sending but can't depend on receiver feedback. This is a problem at the pattern level, not the transport level. Spot on! Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] general strategy for sending data over epgm
Hi Justin,
On 4 December 2012 09:19, Justin Karneges jus...@affinix.com wrote:
On Monday, December 03, 2012 09:59:12 PM Ian Barber wrote:
On Mon, Dec 3, 2012 at 8:50 PM, mlist user mlist.user8...@gmail.com wrote:
Hi Andrew,
On 4 December 2012 07:47, Andrew Hume and...@research.att.com wrote:
i'm not sure what the issue is.
if the rate is small and the messages are large, then yes, you need to
sleep
large amounts.
How would you figure out you need to sleep X second?
Are you setting the RATE sockopt yourself and then trying to manage the
pace at which you send on top of that?
Slightly OT, but I wonder if RATE might be a useful feature to allow for all
PUB sockets (and maybe just all sockets), rather than restricted to pgm only.
It seems to me that you want to use RATE when you need to slow down sending
but can't depend on receiver feedback. This is a problem at the pattern level,
not the transport level.
I have modified my src.
Currently testing a single {key:'foo', value:49mb /dev/zero} and
have the following:
mcast_sender.py:
self._sock.setsockopt(zmq.SNDHWM, 200 * 1024)
self._sock.setsockopt(zmq.SNDBUF, 64*1024)
self._sock.setsockopt(zmq.RATE, 1000)
(no sleeps)
mcast_receiver.py:
s.setsockopt(zmq.RCVHWM, 200*1024)
s.setsockopt(zmq.RCVBUF, 64*1024)
s.setsockopt(zmq.RATE, 1000)
(no sleeps)
The multicast is happening at full throttle (mtu: 1452):
10:07:42.641075 IP 172.16.255.3.33096 224.0.0.1.: UDP, length 1452
I think it's looking good so far but removing those sleeps makes the
code much more predictable.
Thanks
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
