Re: [collectd] zeormq architecture inquiry
So I talked to some people in the #zeromq channel on freenode, and have been reading through the zeromq guide. Seems like at scale, with tons of nodes, you would need pretty fast disk I/O to handle all the writes to RRD. Using ZeroMQ, you could dish out to different collectd worker machines to handle the data of certain kinds, example CPU on one worker and memory on another, which would allow you to distribute your disk i/o amongst smaller and cheaper nodes, possibly VMs. That's a bit complex for right now, but an idea worth thinking about in the future. This would involve a zeromq process that routed requests to various worker machines. I think for my initial attempt at this, I will have one collectd instance set to subscribe, and each node set to publish to my subscriber's endpoint. This should allow nodes to come/go as they please, hopefully this goes well, as the flexibility in zeromq makes a great way to distribute work. Thanks, Allan On Mon, Aug 29, 2011 at 11:58 AM, Florian Forster o...@collectd.org wrote: Hi Allan, On Sun, Aug 28, 2011 at 08:11:14PM -0400, Allan Feid wrote: I was wondering if anyone has any experience doing this, or can provide some guidance on what zeromq architecture would scale best. I was wondering the same thing. For a normal (intra-)DC setup, I think it the best option would be a central subscriber that binds to a local address and many publishers which connect to that address. For an inter-DC setup, I'd probably go with aggregators on a DC level and repeat the same pattern, i.e. publish to a remote address and, on the global level, subscribe to a local address. Disclaimer: I hardly have any ZeroMQ experience either, so this is basically how I think stuff should work, but I might be totally wrong about it … Best regards, —octo -- Florian octo Forster Hacker in training GnuPG: 0x0C705A15 http://octo.it/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) iQIcBAEBAgAGBQJOW7cQAAoJEMPSHpbi2Mmg9PAP/AtwcBZdyS7qDPZC57dodFnV Kaj3Z5k5TJqzVE7+n5JeVYrMkCUelHKI7LTovQvOZ8m/ViYz3tOugfAoRr4pU3M5 GYflMXWAW5yES0+srZsTxChu22n717zwnL3DjqM4byZxL2b0S4h5I7v5pyWqqYSM LmZOOXwy/H8mWXcl1qsMgFBeV0s8jQnITq/yRjx+sQcMDk3Pj094AmLPSNh3tLrC mipOfWyXWBcsFtr7x9K+sqY6PFxmpZEJYU5JEJG+pptRdNmOvPcfGxjlkB4b1MQ3 SlpCLHsWY28/e3fwQ0RVK0dc6+PqzqRLocCRxphbdqRYlZ3EJ9m4Zq+rMtrHra+r i4SpmptQQgRvp7vftv1tE3vYh7jCa7a3li7KhOA7axS1IHQh4zasN/c5/CGnXDLE RvaW4g71xOOoWe63SRABEOUiY+AHRS5+ZiY/+bprscofbd+HmE7CiFK/ZGEMnuG+ eRX+L4sm7dhYckGe7AnokW5fVKOzJd9kJZ36nhEZOUFpLiGIDjoieRK9YZtVeVBF VIZV75WWElT+zFOUcG8tFulR4QrYlsRzoAs6AnvAMRZ4YjfaaIE5IS7Q+GbeQCcQ YuzsXm6E6h/ifx+gBtChpUpAEc+qanS1q1Y8RHCwP7RBEr8mxAYgS4v/YKBV62M9 7Pckd9P+aN6HuHRL/PUN =6w1f -END PGP SIGNATURE- ___ collectd mailing list collectd@verplant.org http://mailman.verplant.org/listinfo/collectd
Re: [collectd] zeormq architecture inquiry
Hi Allan, if you're after distributing the load, you might want to take a look at the AMQP plugin, too. It has these topic branches, where subscribers can subscribe to matching messages only. This allows you to distribute the load in a consistent manner and is re-configurable during runtime. It's possible to only subscribe to certain messages in ZeroMQ as well, but I *think* they match the message content for this. Since the plugin uses the binary protocol, that might not work for us. I might be wrong, though, and there is some label that you can attach to messages, which would make this kind of filtering possible with ZeroMQ, too. Regards, —octo -- Florian octo Forster Hacker in training GnuPG: 0x0C705A15 http://octo.it/ signature.asc Description: Digital signature ___ collectd mailing list collectd@verplant.org http://mailman.verplant.org/listinfo/collectd
Re: [collectd] zeormq architecture inquiry
Hi guys, Just a little note: The current implementation of ZMQ filter on the subscriber, not on the publisher... Seem that ZMQ v3 is addressing the problem. my 2cents On Mon, Aug 29, 2011 at 9:22 AM, Florian Forster o...@collectd.org wrote: Hi Allan, if you're after distributing the load, you might want to take a look at the AMQP plugin, too. It has these topic branches, where subscribers can subscribe to matching messages only. This allows you to distribute the load in a consistent manner and is re-configurable during runtime. It's possible to only subscribe to certain messages in ZeroMQ as well, but I *think* they match the message content for this. Since the plugin uses the binary protocol, that might not work for us. I might be wrong, though, and there is some label that you can attach to messages, which would make this kind of filtering possible with ZeroMQ, too. Regards, —octo -- Florian octo Forster Hacker in training GnuPG: 0x0C705A15 http://octo.it/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) iQIcBAEBAgAGBQJOW7y2AAoJEMPSHpbi2Mmg0ysP/ijjRkTLWdvFCxs+sBBXtvU3 liEIgNf90s3Jgzr33bgx+7m/BQ0JgDT7VrNvH3ek8/OdePGJ15Lnnqr6T5mDz2gD Tj0uysiFCyQCIZPiw5v23x+iNLhwkeS/RC9HFBTPTG3I+s4jG46EvVvBfjlosEYA g+LslGIZ7yG02PN/WKxehYUEK3nnWlz14uHyFwPoQfUGWObpQiiKzerM5R0uL/qn rwyK+IFlmAgzU0m1cbaGq3KCyqOGDHzreW6rC1fql8IFC1ca2ugFA3F6ttYUzRvV yP1RYwydWwA6WYjYyyExWsSTQTWhIyx0JWnxZslodhdA2pEHU+OGsciuPoBnCW5c 35GAozEiAuRFe2QKFPnhDkLyVf8Wh/agyWfCSR8Q1i2UVVrhWUFRIaOCzC/+e8od UamMKv3KkuC931M6YDbLNW4W/E71rQB9sAxCOItVmOpISaW5nCMiw59dknZwN2I4 WsdmTYzMdQZhmjB28/i1fo0vZVuWHH/dK0zAoaWwutanthlVFKrSxcnmfkjzQNj1 WlHqXDvW6h7crksOeWQamrUqplHgP0sKSWsTI2z4zytCGL3+KpdeXe/0FBqPou7J qHqFM2nTKdRNpL8fgcRm9nTz0VyB0vmpmrIQ8WgOjpwfNz9APUrW5s1RECovb+ow 4wLyW448ylCs6WC7HXwE =HGPq -END PGP SIGNATURE- ___ collectd mailing list collectd@verplant.org http://mailman.verplant.org/listinfo/collectd -- François-Xavier Bourlet ___ collectd mailing list collectd@verplant.org http://mailman.verplant.org/listinfo/collectd
Re: [collectd] zeormq architecture inquiry
Thanks Francois, the way the code is right now, the subscriber is the node doing the bind, and the publisher connects to the subscriber. This would mean that your central collectd process would be able to do the filtering since it is the subscriber. As far as AMQP goes, we have noticed scaling problems with the broker model, where this box becomes complex to scale out and seems to be the bottle neck. That is why ZeroMQ is appealing for this kind of situation. On Mon, Aug 29, 2011 at 12:42 PM, Francois-Xavier Bourlet f...@dotcloud.comwrote: Hi guys, Just a little note: The current implementation of ZMQ filter on the subscriber, not on the publisher... Seem that ZMQ v3 is addressing the problem. my 2cents On Mon, Aug 29, 2011 at 9:22 AM, Florian Forster o...@collectd.orgwrote: Hi Allan, if you're after distributing the load, you might want to take a look at the AMQP plugin, too. It has these topic branches, where subscribers can subscribe to matching messages only. This allows you to distribute the load in a consistent manner and is re-configurable during runtime. It's possible to only subscribe to certain messages in ZeroMQ as well, but I *think* they match the message content for this. Since the plugin uses the binary protocol, that might not work for us. I might be wrong, though, and there is some label that you can attach to messages, which would make this kind of filtering possible with ZeroMQ, too. Regards, —octo -- Florian octo Forster Hacker in training GnuPG: 0x0C705A15 http://octo.it/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) iQIcBAEBAgAGBQJOW7y2AAoJEMPSHpbi2Mmg0ysP/ijjRkTLWdvFCxs+sBBXtvU3 liEIgNf90s3Jgzr33bgx+7m/BQ0JgDT7VrNvH3ek8/OdePGJ15Lnnqr6T5mDz2gD Tj0uysiFCyQCIZPiw5v23x+iNLhwkeS/RC9HFBTPTG3I+s4jG46EvVvBfjlosEYA g+LslGIZ7yG02PN/WKxehYUEK3nnWlz14uHyFwPoQfUGWObpQiiKzerM5R0uL/qn rwyK+IFlmAgzU0m1cbaGq3KCyqOGDHzreW6rC1fql8IFC1ca2ugFA3F6ttYUzRvV yP1RYwydWwA6WYjYyyExWsSTQTWhIyx0JWnxZslodhdA2pEHU+OGsciuPoBnCW5c 35GAozEiAuRFe2QKFPnhDkLyVf8Wh/agyWfCSR8Q1i2UVVrhWUFRIaOCzC/+e8od UamMKv3KkuC931M6YDbLNW4W/E71rQB9sAxCOItVmOpISaW5nCMiw59dknZwN2I4 WsdmTYzMdQZhmjB28/i1fo0vZVuWHH/dK0zAoaWwutanthlVFKrSxcnmfkjzQNj1 WlHqXDvW6h7crksOeWQamrUqplHgP0sKSWsTI2z4zytCGL3+KpdeXe/0FBqPou7J qHqFM2nTKdRNpL8fgcRm9nTz0VyB0vmpmrIQ8WgOjpwfNz9APUrW5s1RECovb+ow 4wLyW448ylCs6WC7HXwE =HGPq -END PGP SIGNATURE- ___ collectd mailing list collectd@verplant.org http://mailman.verplant.org/listinfo/collectd -- François-Xavier Bourlet ___ collectd mailing list collectd@verplant.org http://mailman.verplant.org/listinfo/collectd
Re: [collectd] zeormq architecture inquiry
Hello, Allan Feid wrote, Hello, I've been looking into using collectd combined with zeromq for transport. This seems like a fairly good way to scale out monitoring infrastructure as each node being monitored can send out requests over zeromq to one or more locations with little overhead. I was wondering if anyone has any experience doing this, or can provide some guidance on what zeromq architecture would scale best. With my limited zeromq knowledge, I'm thinking that each node being monitored would publish to one or more collector nodes, which could then reroute the information down to a single point where graphing and alerting thresholds could be handled. The collector nodes would likely be in each datacenter and running collectd to write to RRD files. These instances of collectd, could also be publishing or pushing data down to a single aggregation system. I guess my real question here, is what zeromq pattern would make most sense for a distributed collectd monitoring system. Any insight or suggestions are greatly appreciated. My team initially started out with collectd but we ran into a requirement where we needed to have the app itself emit events in real time. It was not clear if the collectd libraries could be used within Java, Perl/Ruby etc applications. After a bit of grepping around, we settled on MonDemand with LWES (UDP+Multicast) as the transport. ZeroMQ was an option, but we liked what we saw in MonDemand. If the need arises in the future, we plan to replace LWES with a MQ system. lwes.sf.net mondemand.sf.net Hth. -- http://shankerbalan.net/ ___ collectd mailing list collectd@verplant.org http://mailman.verplant.org/listinfo/collectd
