Hi Paolo,
Thanks for your feedback.
We are now filtering the template messages from the accounting of netflow
messages, but still the number of flows are l kind of 3 times more than the
number of json messages.
To troubleshoot, we did focus on one specific router, now comparing the
number of flow received for (router ; ifindex) pair with the number of json
messages for the same (router ; ifindex) pair
And we have the kind of comparaison:
exporter IP address ! ifIndex ! netflow_count ! json_count
xxxx.xxxx.xxxx.xxxx ! 464 ! 91144 !
19491xxxx.xxxx.xxxx.xxxx ! 820 ! 3900 !
919
xxxx.xxxx.xxxx.xxxx ! 959 ! 11219 ! 1918
xxxx.xxxx.xxxx.xxxx ! 756 ! 280 ! 59
xxxx.xxxx.xxxx.xxxx ! 757 ! 293 !
56Obviously I am not asking to troubleshoot, but I would like again
confirmation that we should expect from kafka plugin to translate each flow
record matching {router, ifindex) into json and sent to kafka.
Thanks again
Wilfrid
-----Original Message-----
From: Paolo Lucente <[email protected]>
Sent: Friday, 21 October 2022 15:37
To: [email protected]; Grassot, Wilfrid <[email protected]>
Subject: Re: [pmacct-discussion] kafka plugin and number of json messages vs
number of netflow record
CAUTION: External email. Do not click links or open attachments unless you
recognize the sender and know the content is safe.
Hi Wilfrid,
To say whether some aggregation is taking place or not, you should look at
the template of the incoming NetFlow records. You can achieve this with
Wireshark / tshark or via pmacct, either running it in debug mode - you will
find the templates in the log file - or defining a nfacctd_templates_file.
In general, i would expect less JSON records output to Kafka than incoming
NetFlow records because of the templates - which are really service messages
to make the protocol work and hence do not make it to the database.
Paolo
On 21/10/22 09:13, Grassot, Wilfrid wrote:
> Hi Paolo
>
> We are collecting netflow records of several routers interfaces.
>
> Now we are testing the kafka plugin of nfactt using json as format output.
>
> kafka_topic[l3vpn]: pmacct_netflow
>
> aggregate[l3vpn]: tcpflags, proto, src_host, src_port, dst_host,
> dst_port, src_as, dst_as, peer_src_as, peer_dst_as, peer_src_ip,
> peer_dst_ip, in_iface, src_net,
>
> dst_net, tos, timestamp_start, timestamp_end
>
> kafka_broker_host[l3vpn]:
> kafka-node-1.interstellar.prv:9092,kafka-node-2.interstellar.prv:9092,
> kafka-node-3.interstellar.prv:9092
>
> kafka_output[l3vpn]: json
>
> kafka_topic[l3vpn]: pmacct_netflow
>
> Is this setup converting 1 for 1 a netflow record to a json message ?
>
> I am asking because the backend engineers are noticing a lot of
> difference between the number of netflow records received and the
> number of the json messages kafka is receiving.
>
> Is there a kind of aggregation done by kafka plugin that would reduce
> the number of json messages sent to Kafka ?
>
> Thank you in advance.
>
> Wilfrid Grassot
>
> **
>
>
> _______________________________________________
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
