Hi Wilfrid,
To say whether some aggregation is taking place or not, you should look
at the template of the incoming NetFlow records. You can achieve this
with Wireshark / tshark or via pmacct, either running it in debug mode -
you will find the templates in the log file - or defining a
nfacctd_templates_file.
In general, i would expect less JSON records output to Kafka than
incoming NetFlow records because of the templates - which are really
service messages to make the protocol work and hence do not make it to
the database.
Paolo
On 21/10/22 09:13, Grassot, Wilfrid wrote:
Hi Paolo
We are collecting netflow records of several routers interfaces.
Now we are testing the kafka plugin of nfactt using json as format output.
kafka_topic[l3vpn]: pmacct_netflow
aggregate[l3vpn]: tcpflags, proto, src_host, src_port, dst_host,
dst_port, src_as, dst_as, peer_src_as, peer_dst_as, peer_src_ip,
peer_dst_ip, in_iface, src_net,
dst_net, tos, timestamp_start, timestamp_end
kafka_broker_host[l3vpn]:
kafka-node-1.interstellar.prv:9092,kafka-node-2.interstellar.prv:9092,kafka-node-3.interstellar.prv:9092
kafka_output[l3vpn]: json
kafka_topic[l3vpn]: pmacct_netflow
Is this setup converting 1 for 1 a netflow record to a json message ?
I am asking because the backend engineers are noticing a lot of
difference between the number of netflow records received and the number
of the json messages kafka is receiving.
Is there a kind of aggregation done by kafka plugin that would reduce
the number of json messages sent to Kafka ?
Thank you in advance.
Wilfrid Grassot
**
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
