On 25/08/2021 09:44, Peter Haag wrote:
On 18.08.21 11:49, Brian Candler wrote:
On 18/08/2021 09:45, Peter Haag wrote:
nfsen is *very* old code, and actually I wanted to rewrite a new version, which
got delayed and finally aborted for a number of reasons.
I could consider to give it a go again, but I am still uncertain if it would
still be used these days. Feedback would be appreciated.
If there would be a new NfSen incarnation, I would write it in Golang as it
provides far more flexibility and speed. I am not a exceptionally gifted
GUI/Web developer - maybe someone would be willing to support. However, also
for a new NfSen, I could prefer KISS
- easy of use and - hopefully - fast.
A maintained replacement for NFSen would be extremely welcome!
I think a careful design can help to prevent making the new interface unusable
and integrate
those assets, which users on a large scale would like.
Actually I like the idea using Prometheus, as a lot of things are already
integrated.
Using Prometheus would naturally result in using Grafana as frontend. The drill
down and listing flows needs to be implemented somehow. Up to now I have no
experience in
developing grafana apps - I need to study this first.
Would you be comfortable with Grafana graphs and the way to zoom in to spot the
points of
interest?
Yes I think so. Selecting time ranges in Grafana is OK, although
zooming out once you've already selected a range is a bit painful. The
key thing of course would be mapping the time range to the nfdump
command line. Checking how grafana deals with loki logs may be helpful
here. Maybe it's even possible to integrate with the "Explore"
functionality. (*)
I would still ideally like to be able to select a single point of
interest and get the netflow logs from that "bucket" - i.e. the "select
single" function in nfsen. It makes it easy to move back and forth
around a point of interest.
(*) Aside: a completely different approach would be to convert netflow
records into JSON and write them to loki, with certain fields extracted
as labels. This would then use loki's query language for analyzing
netflow, and make most of nfdump redundant. I don't think loki's query
engine is featureful enough to do this (e.g. grouping by subnet); I also
suspect that even compressed JSON would be less efficient storage-wise
than nfdump's format.
Creating profile as possible so far in NfSen could be a challenge for
Prometheus, as importing
historical data ( which filtering for a profile is nothing else ) seems not
what Prometheus was
intended for.
Prometheus has fairly recently (v2.24.0) gained the ability to backfill
historical data:
https://github.com/prometheus/prometheus/releases/tag/v2.24.0
https://github.com/prometheus/prometheus/pull/8230/files
Any remarks opinions and ideas are welcome in order to draft ideas for a modern
interface and
workflow with netflow data.
Meanwhile I try to write a nfdump netflow_exporter for Prometheus to play and
experiment with.
It's almost done.
I would put the exporter on a new repo on Github, so discussion could continue
there as an issue,
or here on the ML.
Sounds excellent, I look forward to playing with this!
Regards,
Brian.
Regards
- Peter
I see two aspects to this:
1. the web UI for selecting time ranges and drilling down to netflow queries
2. the creation and storage of graphs of flow data (i.e. summaries by protocol,
and user-defined channels)
As regards (1): whilst the UI of NFSen is very much web-1.0, it is
straightforward, practical and usable in real world scenarios.
Attempts to "modernise" the UI, as in nfsen-ng, have made it unusable. Just
try selecting time ranges: things zoom horizontally, making it very
difficult to change to a wider time range. In my opinion, it's a complete
fail. Selecting time ranges is the main /raison d'être/ of NFSen.
As regards (2): I would be very happy to see rrdtool dropped entirely and
replaced by something modern, like Prometheus. This could be very simple to
implement: the nfsen backend simply filters and counts flow records, and
exposes the counters as prometheus metrics. Then prometheus scrapes them.
This could be done at a finer granularity than today, e.g. 1 minute or better. (In fact,
since prometheus considers timeseries "stale" after 5 minutes
of no data, it's recommended that you scrape *at least* every 2 minutes. And
the cost of doing so is much lower than rrdtool). Apart from being
efficient and scalable, the Prometheus ecosystem opens up a whole range of
possibilities around dashboards and alerting - and it means there's no need
to re-implement alerts in NFSen.
nfsen-ng missed this opportunity too.
Regards,
Brian.
_______________________________________________
Nfsen-discuss mailing list
Nfsen-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
_______________________________________________
Nfsen-discuss mailing list
Nfsen-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
