[zeromq-dev] Cannot get epgm pubsub to work
To try out ZeroMQ, I've installed pyzmq and zeromq 2.2.0. When I run the example publisher.py, I am able to see UDP packages via tcpdump, e.g., $ python examples/pubsub/publisher.py epgm://eth1;239.0.1.1:34415 3 5 [root@vagrant-centos-5 ~]# tcpdump -i eth1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes 20:30:10.144813 IP 192.168.15.11.47356 239.0.1.1.34415: UDP, length 36 However, the example subscribe never receives any messages: $ python examples/pubsub/subscriber.py epgm://eth1;239.0.1.1:34415 1 any idea what can be causing this? -- --- John L Cheng ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Cannot get epgm pubsub to work
On 13 December 2012 15:33, John Cheng johnlich...@gmail.com wrote: However, the example subscribe never receives any messages: Separate hosts are required on the network, PGM is not supported localhost, use IPC or ITC instead. -- Steve-o ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] general strategy for sending data over epgm
On 3 December 2012 17:19, Justin Karneges jus...@affinix.com wrote: Slightly OT, but I wonder if RATE might be a useful feature to allow for all PUB sockets (and maybe just all sockets), rather than restricted to pgm only. It seems to me that you want to use RATE when you need to slow down sending but can't depend on receiver feedback. This is a problem at the pattern level, not the transport level. Note that the RATE limits what is forwarded onto the wire, not what you are feeding into 0mq. If you send fast enough with no high watermark you will consume a significant amount of memory to adverse affects. An application rate limited is recommended but may be more suitable as an optional layer above 0mq API. -- Steve-o ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] general strategy for sending data over epgm
On 3 December 2012 15:39, mlist user mlist.user8...@gmail.com wrote:
Hi list,
First post here. I would like to figure out how you guys manage to
send data at certain frequency.
For e.g., if I send small'ish {k,v} over epgm over 100mb link with
0.60ms RTT, i need to sleep only 2 second per send.
But, this is not applicable to large {k, v} over epgm. I need to sleep
more than 12 second per send!
Can you confirm reliability is working? Wireshark can be used to monitor
the protocol. If you are sending too fast, e.g. 60% capacity of the
network you may see packet loss and thus NAKs from the receivers and RDATA
packets from the senders. Monitor the network capacity to ensure that both
the original data and repair data are not saturating the network.
--
Steve-o
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Sub problem
Hi, I am still observing a case were a connected ZeroMQ Sub is still not receiving any data anymore after a certain period of idle time without any data. We are willing to test this with exact numbers (our estimation is that less than 4 hours no data results in a dead client) but in the mean time I was reasoning about problem. My hypothesis of this 'dead client' is that the socket is for some reason in a disconnected state, after it reconnects it does not apply the 'original' socket options. This cannot be reproduced using normal code, a socket in ZeroMQ can't be closed and then reconnected or rebinded. Obviously we are going to see if we can reproduce the problem, and tcpdumping the connection to find out if the client is really receiving new data, or that the client is truly 'disconnected'. Would anyone be willing to take a peak if there is any basis for this reasoning? Stefan ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Sub problem
Hi Stefan, Are you connecting SUB to PUB via some firewall or proxy, or is this a direct connection? Also, is the publisher silent for long periods of time? The reason I ask is that we've seen connections going 'stale' sometimes; TCP not reporting an error at the client side but not transmitting anything. It's due to some proxy in the middle getting confused, presumably. If this is happening, consider sending keep-alive messages from PUB to SUB, which the SUB can discard (but must subscribe to, in 3.2). -Pieter On Thu, Dec 13, 2012 at 11:19 PM, Stefan de Konink ste...@konink.de wrote: Hi, I am still observing a case were a connected ZeroMQ Sub is still not receiving any data anymore after a certain period of idle time without any data. We are willing to test this with exact numbers (our estimation is that less than 4 hours no data results in a dead client) but in the mean time I was reasoning about problem. My hypothesis of this 'dead client' is that the socket is for some reason in a disconnected state, after it reconnects it does not apply the 'original' socket options. This cannot be reproduced using normal code, a socket in ZeroMQ can't be closed and then reconnected or rebinded. Obviously we are going to see if we can reproduce the problem, and tcpdumping the connection to find out if the client is really receiving new data, or that the client is truly 'disconnected'. Would anyone be willing to take a peak if there is any basis for this reasoning? Stefan ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Titanic SP with encrypted data transfer
On Thu, Dec 13, 2012 at 11:42 PM, Jovan Kostovski chomb...@gmail.com wrote: I know that ZeroMQ supports TLS shared keys encryption... It doesn't, yet, unfortunately. If you need encryption, you will need to either do it at a lower layer (VPN), which is usually quite nasty, or else modify the TSP protocol to do encryption using something like SASL, which is also nasty. You can also encrypt per message, using pre-shared keys, which is the least nasty option IMO. I do hope we find a better general solution at some stage; you are not the only person who hits this. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Sub problem
Hi Pieter, On 12/14/12 00:20, Pieter Hintjens wrote: Are you connecting SUB to PUB via some firewall or proxy, or is this a direct connection? The same (local) iptables as before, just accept/drop, without any connection tracking what so ever. Also, is the publisher silent for long periods of time? Typically not (messages every second), but in this case the 'original' publisher had an issue, which lead to a longer silence. The reason I ask is that we've seen connections going 'stale' sometimes; TCP not reporting an error at the client side but not transmitting anything. It's due to some proxy in the middle getting confused, presumably. If this is happening, consider sending keep-alive messages from PUB to SUB, which the SUB can discard (but must subscribe to, in 3.2). I was considering this, and using zmq_poll to actually timeout and reconnect. Would you be interested in a tcpdump of a (failed) test? Stefan ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Sub problem
On Fri, Dec 14, 2012 at 12:23 AM, Stefan de Konink ste...@konink.de wrote: The same (local) iptables as before, just accept/drop, without any connection tracking what so ever. I recall last time you found the problem didn't happen when you connected directly, only when you went via iptables? So it's definitely doing *something*. If this is happening, consider sending keep-alive messages from PUB to SUB, which the SUB can discard (but must subscribe to, in 3.2). I was considering this, and using zmq_poll to actually timeout and reconnect. Would you be interested in a tcpdump of a (failed) test? No, but thanks for the offer. Yes, definitely use zmq_poll to timeout, and reconnect. However if TCP reports an error the SUB socket will reconnect automatically. What you have to do (and this is pretty standard across any protocol) is do your own keep-alive to keep the I'm really not messing with your connections, promised!* proxy in the middle out of trouble. PUB socket sends a null message once a second; SUB gets that (must subscribe to it!) and discards it. If SUB doesn't get null message after X seconds, decides that PUB is dead so looks for backup. -Pieter * It really is messing with your connections. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Titanic SP with encrypted data transfer
On Friday, December 14, 2012 12:23:19 AM Pieter Hintjens wrote: On Thu, Dec 13, 2012 at 11:42 PM, Jovan Kostovski chomb...@gmail.com wrote: I know that ZeroMQ supports TLS shared keys encryption... It doesn't, yet, unfortunately. If you need encryption, you will need to either do it at a lower layer (VPN), which is usually quite nasty, or else modify the TSP protocol to do encryption using something like SASL, which is also nasty. You can also encrypt per message, using pre-shared keys, which is the least nasty option IMO. Another idea is to gateway through a secure protocol such as HTTPS or XMPP when crossing hostile networks. This isn't really a ZeroMQ solution, but if the majority of your sockets aren't at risk and you're just trying to protect a couple of them that go over the internet, it's probably the best choice in terms of secureness vs nastiness. :) It is, however, quite some extra coding. Justin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
