Hi Paolo,
Thanks for your feedback.
We are now filtering the template messages from the accounting of netflow
messages, but still the number of flows are l kind of 3 times more
than the
number of json messages.
To troubleshoot, we did focus on one specific router, now comparing the
number of flow received for (router ; ifindex) pair with the number
of json
messages for the same (router ; ifindex) pair
And we have the kind of comparaison:
exporter IP address ! ifIndex ! netflow_count ! json_count
xxxx.xxxx.xxxx.xxxx ! 464 ! 91144 !
19491xxxx.xxxx.xxxx.xxxx ! 820 ! 3900 ! 919
xxxx.xxxx.xxxx.xxxx ! 959 ! 11219 ! 1918
xxxx.xxxx.xxxx.xxxx ! 756 ! 280 ! 59
xxxx.xxxx.xxxx.xxxx ! 757 ! 293 !
56Obviously I am not asking to troubleshoot, but I would like again
confirmation that we should expect from kafka plugin to translate each
flow
record matching {router, ifindex) into json and sent to kafka.
Thanks again
Wilfrid
-----Original Message-----
From: Paolo Lucente <[email protected]>
Sent: Friday, 21 October 2022 15:37
To: [email protected]; Grassot, Wilfrid
<[email protected]>
Subject: Re: [pmacct-discussion] kafka plugin and number of json
messages vs
number of netflow record
CAUTION: External email. Do not click links or open attachments
unless you
recognize the sender and know the content is safe.
Hi Wilfrid,
To say whether some aggregation is taking place or not, you should
look at
the template of the incoming NetFlow records. You can achieve this with
Wireshark / tshark or via pmacct, either running it in debug mode -
you will
find the templates in the log file - or defining a
nfacctd_templates_file.
In general, i would expect less JSON records output to Kafka than
incoming
NetFlow records because of the templates - which are really service
messages
to make the protocol work and hence do not make it to the database.
Paolo
On 21/10/22 09:13, Grassot, Wilfrid wrote:
Hi Paolo
We are collecting netflow records of several routers interfaces.
Now we are testing the kafka plugin of nfactt using json as format
output.
kafka_topic[l3vpn]: pmacct_netflow
aggregate[l3vpn]: tcpflags, proto, src_host, src_port, dst_host,
dst_port, src_as, dst_as, peer_src_as, peer_dst_as, peer_src_ip,
peer_dst_ip, in_iface, src_net,
dst_net, tos, timestamp_start, timestamp_end
kafka_broker_host[l3vpn]:
kafka-node-1.interstellar.prv:9092,kafka-node-2.interstellar.prv:9092,
kafka-node-3.interstellar.prv:9092
kafka_output[l3vpn]: json
kafka_topic[l3vpn]: pmacct_netflow
Is this setup converting 1 for 1 a netflow record to a json message ?
I am asking because the backend engineers are noticing a lot of
difference between the number of netflow records received and the
number of the json messages kafka is receiving.
Is there a kind of aggregation done by kafka plugin that would reduce
the number of json messages sent to Kafka ?
Thank you in advance.
Wilfrid Grassot
**
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
