I have 100s of routers peering to a pool of about 50 collectors currently. Each router peers to one collector and its this 1:1 relationship that sometimes changes. The IP’s do not change often, maybe a handful of times a year but when they do, its a real pain to go touch the routers. Its also a pain to manage the necessary rules of which router points to which collector at the network level. I prefer clean and consistent router configs so pushing these rules outside the network makes automation a lot easier.
> On Sep 5, 2017, at 2:22 PM, Aaron Finney <aaron.fin...@openx.com> wrote: > > I would think you'd just peer every collector to every device in a full mesh, > unless i'm missing something obvious. Having peering sessions going up and > down constantly between the network devices and one of n collectors behind a > load balancer does not seem feasible. > > On Tue, Sep 5, 2017 at 10:46 AM, Paul Mabey <p...@mabey.net > <mailto:p...@mabey.net>> wrote: > Right…..routers export flow to the VIP, as well as “think” they are BGPing > with the VIP. The LB then has a static rule that forwards both BGP/flow to > the correct collector. The goal being that if the collector IP changes for > some reason, I don’t have to go touch the router configs. > > >> On Sep 5, 2017, at 11:35 AM, Aaron Finney <aaron.fin...@openx.com >> <mailto:aaron.fin...@openx.com>> wrote: >> >> I'm not sure I follow - do you mean setting up BGP peering of the collectors >> to your source devices using the collector VIP as the neighbor address? >> >> On Sep 5, 2017 10:11 AM, "Paul Mabey" <p...@mabey.net >> <mailto:p...@mabey.net>> wrote: >> Has anyone had success is pushing BGP sessions through an LB along with >> netflow? Interested in the solution below but would like to have BGP aligned >> with netflow as well. >> >>> On Sep 4, 2017, at 9:48 AM, Aaron Finney <aaron.fin...@openx.com >>> <mailto:aaron.fin...@openx.com>> wrote: >>> >>> Great to hear, nice work! >>> >>> Aaron >>> >>> On Sep 4, 2017 1:55 AM, "Yann Belin" <y.belin...@gmail.com >>> <mailto:y.belin...@gmail.com>> wrote: >>> Hi all, >>> >>> Updating on this, in case someone is interested. >>> >>> Consul was indeed the way to go: >>> >>> * nginx is doing the actual UDP load balancing, based on source IP >>> hash (to optimize aggregation). >>> * consul keeps track of nfacctd collectors, of their health, and of >>> the health of their dependencies (rabbitmq in my case). >>> * consul-template uses the information provided by consul (servers + >>> health) to generate nginx configuration files, and reloads nginx >>> service if needed; if a collector becomes unhealthy (e.g. rabbitmq >>> crashes), it will be removed from nginx configuration and will stop >>> receiving flows. >>> >>> The great thing with consul is that you can write your own checks. For >>> now my checks are relatively basic (process + port binding checks) but >>> I am working on a more advanced one for rabbitmq (e.g. queue length / >>> ram usage). I'm still thinking about more advanced ways to check >>> nfacctd health, if anyone has a suggestion. >>> >>> Cheers, >>> >>> Yann >>> >>> >>> On Mon, Aug 21, 2017 at 4:02 PM, Aaron Finney <aaron.fin...@openx.com >>> <mailto:aaron.fin...@openx.com>> wrote: >>> > Hi Yann >>> > >>> > We use Consul for this, it works very well. >>> > >>> > https://www.consul.io <https://www.consul.io/> >>> > >>> > >>> > Aaron >>> > >>> > >>> > >>> > On Aug 21, 2017 6:44 AM, "Yann Belin" <y.belin...@gmail.com >>> > <mailto:y.belin...@gmail.com>> wrote: >>> > >>> > Hello, >>> > >>> > I have been looking into solutions to achieve reliable load balancing >>> > of my incoming flows across multiple nfacctd servers / daemons. >>> > >>> > Basic load balancing is relatively easy (see Nginx configuration >>> > below), but *reliable* load balancing (only sending flows to servers >>> > that have a running nfacctd daemon) is quite more complicated. For >>> > instance, Nginx normally monitors UDP responses from the remote >>> > servers to determine if those servers are health, but this approach >>> > will not work in the case of netflow or ipfix. >>> > >>> > Did anybody already managed to solve this? Or has a suggestion perhaps? >>> > >>> > Thanks in advance! >>> > >>> > *-*-*-*-*-*-*-* >>> > stream { >>> > upstream ipfix_traffic { >>> > hash $binary_remote_addr; >>> > server 10.20.10.10:9055 <http://10.20.10.10:9055/>; >>> > server 10.20.10.20:9055 <http://10.20.10.20:9055/>; >>> > } >>> > >>> > server { >>> > listen 9055 udp; >>> > proxy_responses 0; >>> > proxy_pass ipfix_traffic; >>> > proxy_bind $remote_addr transparent; >>> > error_log /var/log/nginx/ipfix_traffic.error.log; >>> > } >>> > } >>> > *-*-*-*-*-*-*-* >>> > >>> > Kind regards, >>> > >>> > Yann >>> > >>> > _______________________________________________ >>> > pmacct-discussion mailing list >>> > http://www.pmacct.net/#mailinglists <http://www.pmacct.net/#mailinglists> >>> > >>> > >>> > >>> > _______________________________________________ >>> > pmacct-discussion mailing list >>> > http://www.pmacct.net/#mailinglists <http://www.pmacct.net/#mailinglists> >>> >>> _______________________________________________ >>> pmacct-discussion mailing list >>> http://www.pmacct.net/#mailinglists <http://www.pmacct.net/#mailinglists> >>> >>> _______________________________________________ >>> pmacct-discussion mailing list >>> http://www.pmacct.net/#mailinglists <http://www.pmacct.net/#mailinglists> >> >> _______________________________________________ >> pmacct-discussion mailing list >> http://www.pmacct.net/#mailinglists <http://www.pmacct.net/#mailinglists> >> _______________________________________________ >> pmacct-discussion mailing list >> http://www.pmacct.net/#mailinglists <http://www.pmacct.net/#mailinglists> > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists <http://www.pmacct.net/#mailinglists> > > > > -- > Aaron Finney > Network Engineer | OpenX > 888 East Walnut Street, 2nd Floor | Pasadena, CA 91101 > o: +1 (626) 466-1141 x6035 | aaron.fin...@openx.com > <mailto:aaron.fin...@openx.com> > Advertising Age Best Places to Work > <http://openx.com/press-releases/openx-named-as-one-of-advertising-ages-top-fifty-best-places-to-work-for-2015/> > Deloitte's Technology Fast 500™ > <http://openx.com/press-releases/openx-ranked-3rd-fastest-growing-software-company-north-america-5th-fastest-overall-deloittes-2013-technology-fast-500/> > www.openx.com <http://www.openx.com/>| Twitter > <http://twitter.com/openx>| Facebook <http://www.facebook.com/OpenX>| > LinkedIn <http://www.linkedin.com/company/openx/products>| YouTube > <http://www.youtube.com/user/openxvideos> > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists
_______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists