Hi all,

Updating on this, in case someone is interested.

Consul was indeed the way to go:

* nginx is doing the actual UDP load balancing, based on source IP
hash (to optimize aggregation).
* consul keeps track of nfacctd collectors, of their health, and of
the health of their dependencies (rabbitmq in my case).
* consul-template uses the information provided by consul (servers +
health) to generate nginx configuration files, and reloads nginx
service if needed; if a collector becomes unhealthy (e.g. rabbitmq
crashes), it will be removed from nginx configuration and will stop
receiving flows.

The great thing with consul is that you can write your own checks. For
now my checks are relatively basic (process + port binding checks) but
I am working on a more advanced one for rabbitmq (e.g. queue length /
ram usage). I'm still thinking about more advanced ways to check
nfacctd health, if anyone has a suggestion.

Cheers,

Yann


On Mon, Aug 21, 2017 at 4:02 PM, Aaron Finney <aaron.fin...@openx.com> wrote:
> Hi Yann
>
> We use Consul for this, it works very well.
>
> https://www.consul.io
>
>
> Aaron
>
>
>
> On Aug 21, 2017 6:44 AM, "Yann Belin" <y.belin...@gmail.com> wrote:
>
> Hello,
>
> I have been looking into solutions to achieve reliable load balancing
> of my incoming flows across multiple nfacctd servers / daemons.
>
> Basic load balancing is relatively easy (see Nginx configuration
> below), but *reliable* load balancing (only sending flows to servers
> that have a running nfacctd daemon) is quite more complicated. For
> instance, Nginx normally monitors UDP responses from the remote
> servers to determine if those servers are health, but this approach
> will not work in the case of netflow or ipfix.
>
> Did anybody already managed to solve this? Or has a suggestion perhaps?
>
> Thanks in advance!
>
> *-*-*-*-*-*-*-*
> stream {
>     upstream ipfix_traffic {
>         hash $binary_remote_addr;
>         server 10.20.10.10:9055;
>         server 10.20.10.20:9055;
>     }
>
>     server {
>         listen 9055 udp;
>         proxy_responses 0;
>         proxy_pass ipfix_traffic;
>         proxy_bind $remote_addr transparent;
>         error_log /var/log/nginx/ipfix_traffic.error.log;
>     }
> }
> *-*-*-*-*-*-*-*
>
> Kind regards,
>
> Yann
>
> _______________________________________________
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
>
>
> _______________________________________________
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists

_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Reply via email to