I second Catalin's answer; once you have the data in Kafka, you can have
multiple consumers working with it for different purposes. Our solution
sends raw flow data to Kafka topics with a 24 hour retention time, then we
have spark streaming jobs that do near-real-time processing for anomaly
detection and initial enrichment/aggregation before sending to RiakTS for
longer-term storage. We then have a series of ETLs that send enriched data
to other Kafka topics for other reporting tools to work with, and others
that reduce the data to lower levels of granularity over time (e.g. 5 min
resolution after 30 days with 180 day retention).

For presentation, we eventually plan to use different tools for different
views. We have Grafana <= RRDs <= in-house collector <= spark-streaming
consumer for showing relatively simple flow data (ASN/IP/bandwidth) for
individual partners now; we also have an in-house alerting system built on
top of Grafana. We're still looking at tools to display reports we're
generating on deeper analysis of data enriched with other business metrics,
and possibly a 2nd "dashboard" type view which will more clearly identify
things like peering opportunities and traffic management optimizations, but
that's still TBD.


On Mon, Jun 12, 2017 at 3:05 AM, Catalin Petrescu <cpmar...@gmail.com>

> Hi Sami,
> Looks like you need “tee plugin”  http://wiki.pmacct.net/OfficialExamples
> has more details on how to configure that.
> For the second question, i guess depends on what you want to achieve. For
> us pmacct>kafka>influxdb and grafana works best , other have used
> elasticsearch and kibana or plain old mysql.
> Regards,
> Catalin
> On Sun, Jun 11, 2017 at 1:37 AM, Sami <sam...@protonmail.com> wrote:
>> Hello,
>> I have been looking for a while on the net to find out a tool allowing me
>> to properly proxy/duplicate Netflow traffic and nfacctd seems to do the job
>> well but now i wanted to get the most of it.
>> This is my setup:
>> Multiple NetFlow flows -> Server running nfacctd/pmacctd (Let's call it a
>> proxy)-> multiple destinations
>> What i want to do now is to log NetFlow traffic on files (.csv/.log ..),
>> do you have any sample configuration for this?
>> Another question not related to the previous is:
>> How do you visualize accounted traffic via dashboards? (ES + Kibana or
>> you extract data directly from MySQL via some Dashboarding reporting tool?)
>> Sami
>> _______________________________________________
>> pmacct-discussion mailing list
>> http://www.pmacct.net/#mailinglists
> _______________________________________________
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
pmacct-discussion mailing list

Reply via email to