Hi!..  so i have the load spread between a 3 machines and 2 ports per
box. the biggest thing in the netflow data is the ordering for me. I
guess where i am still curious is would either of those settings be
causing the complete drop in the service where it starts and stops
every 5 minutes on the dot? I am going to play around with the times
on it to see if it is one of those settings. I will eventually have to
increase this to about 2-4m flows per second so maybe the replicator
is the best way forward.

On Fri, Nov 17, 2017 at 9:47 AM, Paolo Lucente <pa...@pmacct.net> wrote:
>
> Hi Anthony,
>
> I map the word 'message' to 'flow' and not to NetFlow packet, please
> correct me if this assumption is wrong. 55m flows/min makes it roughly
> 1m flows/sec. I would not recommend stretching a single nfacctd daemon
> beyond beyond 200K flows/sec and the beauty of NetFlow, being UDP, is
> that it can be easily scaled horizontally. For a start, details and
> complexity may vary from use-case to use-case, I would hence recommend
> to look in the following direction: point all NetFlow to a single IP/
> port where a nfacctd in replicator mode is listening. You should test
> it being able to absorb the full feed on your CPU resources. Then you
> replicate to nfacctd collectors downstream parts of the full feed, ie.
> you can instantiate with some headroom around 6-8 nfacctd collectors.
> You can balance the incoming NetFlow packets using round-robin or
> assigning flow exporters to flow collectors or with some hashing. Here
> is how to start with it:
>
> https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L1384-L1445
>
> Of course you can do the same with your load-balancer of preference.
>
> Paolo
>
> On Thu, Nov 16, 2017 at 01:16:48PM -0500, Anthony Caiafa wrote:
>> Hi! So my usecase may be slightly larger than most. I am processing 1:1
>> netflow data for a larger infrastructure. We are receiving about 55million
>> messages a minute which isn’t much but through pmacct it seems to not like
>> it so much. I have pmacct scheduled with nomad running across a few
>> machines and 2 designated ports accepting the flow traffic and outputting
>> those to kafka.
>>
>> About every 5m or so pmacct dies and restarts basically dropping all
>> traffic for a short period of time. The two configurations i have that are
>> doing anything every 5 minutes are:
>>
>> kafka_refresh_time[name]: 300
>> kafka_history[name]: 5m
>>
>>
>> So i am not sure if its one of these or not since the logs only indicate
>> that it lost a connection to kafka and thats about it.
>
>> _______________________________________________
>> pmacct-discussion mailing list
>> http://www.pmacct.net/#mailinglists
>
>
> _______________________________________________
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists

_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Reply via email to