I second Catalin's answer; once you have the data in Kafka, you can have multiple consumers working with it for different purposes. Our solution sends raw flow data to Kafka topics with a 24 hour retention time, then we have spark streaming jobs that do near-real-time processing for anomaly detection and initial enrichment/aggregation before sending to RiakTS for longer-term storage. We then have a series of ETLs that send enriched data to other Kafka topics for other reporting tools to work with, and others that reduce the data to lower levels of granularity over time (e.g. 5 min resolution after 30 days with 180 day retention).
For presentation, we eventually plan to use different tools for different views. We have Grafana <= RRDs <= in-house collector <= spark-streaming consumer for showing relatively simple flow data (ASN/IP/bandwidth) for individual partners now; we also have an in-house alerting system built on top of Grafana. We're still looking at tools to display reports we're generating on deeper analysis of data enriched with other business metrics, and possibly a 2nd "dashboard" type view which will more clearly identify things like peering opportunities and traffic management optimizations, but that's still TBD. Aaron On Mon, Jun 12, 2017 at 3:05 AM, Catalin Petrescu <[email protected]> wrote: > Hi Sami, > > Looks like you need “tee plugin” http://wiki.pmacct.net/OfficialExamples > has more details on how to configure that. > > For the second question, i guess depends on what you want to achieve. For > us pmacct>kafka>influxdb and grafana works best , other have used > elasticsearch and kibana or plain old mysql. > > Regards, > > Catalin > > > On Sun, Jun 11, 2017 at 1:37 AM, Sami <[email protected]> wrote: > >> Hello, >> I have been looking for a while on the net to find out a tool allowing me >> to properly proxy/duplicate Netflow traffic and nfacctd seems to do the job >> well but now i wanted to get the most of it. >> This is my setup: >> Multiple NetFlow flows -> Server running nfacctd/pmacctd (Let's call it a >> proxy)-> multiple destinations >> >> What i want to do now is to log NetFlow traffic on files (.csv/.log ..), >> do you have any sample configuration for this? >> >> >> Another question not related to the previous is: >> How do you visualize accounted traffic via dashboards? (ES + Kibana or >> you extract data directly from MySQL via some Dashboarding reporting tool?) >> >> Sami >> >> >> _______________________________________________ >> pmacct-discussion mailing list >> http://www.pmacct.net/#mailinglists >> > > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists >
_______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
