Hi Wilfrid,
Can you please check whether you are dropping any NetFlow packets:
https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L3065-#L3071 .
Also, as i was saying in the previous email, to be sure no aggregation
is taking place, we should look at the templates and compare them with
the aggregation method you defined in pmacct: any chance you can share a
sample (produced as i was suggesting but also a Wireshark screenshot
would work OK, if it's easier for you) here or via unicast email?
Paolo
On 24/10/22 06:16, Grassot, Wilfrid wrote:
Hi Paolo,
Thanks for your feedback.
We are now filtering the template messages from the accounting of netflow
messages, but still the number of flows are l kind of 3 times more than the
number of json messages.
To troubleshoot, we did focus on one specific router, now comparing the
number of flow received for (router ; ifindex) pair with the number of json
messages for the same (router ; ifindex) pair
And we have the kind of comparaison:
exporter IP address ! ifIndex ! netflow_count ! json_count
xxxx.xxxx.xxxx.xxxx ! 464 ! 91144 !
19491xxxx.xxxx.xxxx.xxxx ! 820 ! 3900 !
919
xxxx.xxxx.xxxx.xxxx ! 959 ! 11219 ! 1918
xxxx.xxxx.xxxx.xxxx ! 756 ! 280 ! 59
xxxx.xxxx.xxxx.xxxx ! 757 ! 293 !
56Obviously I am not asking to troubleshoot, but I would like again
confirmation that we should expect from kafka plugin to translate each flow
record matching {router, ifindex) into json and sent to kafka.
Thanks again
Wilfrid
-----Original Message-----
From: Paolo Lucente <[email protected]>
Sent: Friday, 21 October 2022 15:37
To: [email protected]; Grassot, Wilfrid <[email protected]>
Subject: Re: [pmacct-discussion] kafka plugin and number of json messages vs
number of netflow record
CAUTION: External email. Do not click links or open attachments unless you
recognize the sender and know the content is safe.
Hi Wilfrid,
To say whether some aggregation is taking place or not, you should look at
the template of the incoming NetFlow records. You can achieve this with
Wireshark / tshark or via pmacct, either running it in debug mode - you will
find the templates in the log file - or defining a nfacctd_templates_file.
In general, i would expect less JSON records output to Kafka than incoming
NetFlow records because of the templates - which are really service messages
to make the protocol work and hence do not make it to the database.
Paolo
On 21/10/22 09:13, Grassot, Wilfrid wrote:
Hi Paolo
We are collecting netflow records of several routers interfaces.
Now we are testing the kafka plugin of nfactt using json as format output.
kafka_topic[l3vpn]: pmacct_netflow
aggregate[l3vpn]: tcpflags, proto, src_host, src_port, dst_host,
dst_port, src_as, dst_as, peer_src_as, peer_dst_as, peer_src_ip,
peer_dst_ip, in_iface, src_net,
dst_net, tos, timestamp_start, timestamp_end
kafka_broker_host[l3vpn]:
kafka-node-1.interstellar.prv:9092,kafka-node-2.interstellar.prv:9092,
kafka-node-3.interstellar.prv:9092
kafka_output[l3vpn]: json
kafka_topic[l3vpn]: pmacct_netflow
Is this setup converting 1 for 1 a netflow record to a json message ?
I am asking because the backend engineers are noticing a lot of
difference between the number of netflow records received and the
number of the json messages kafka is receiving.
Is there a kind of aggregation done by kafka plugin that would reduce
the number of json messages sent to Kafka ?
Thank you in advance.
Wilfrid Grassot
**
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
