Hi Paolo, Thanks for your feedback. We are now filtering the template messages from the accounting of netflow messages, but still the number of flows are l kind of 3 times more than the number of json messages.
To troubleshoot, we did focus on one specific router, now comparing the number of flow received for (router ; ifindex) pair with the number of json messages for the same (router ; ifindex) pair And we have the kind of comparaison: exporter IP address ! ifIndex ! netflow_count ! json_count xxxx.xxxx.xxxx.xxxx ! 464 ! 91144 ! 19491xxxx.xxxx.xxxx.xxxx ! 820 ! 3900 ! 919 xxxx.xxxx.xxxx.xxxx ! 959 ! 11219 ! 1918 xxxx.xxxx.xxxx.xxxx ! 756 ! 280 ! 59 xxxx.xxxx.xxxx.xxxx ! 757 ! 293 ! 56Obviously I am not asking to troubleshoot, but I would like again confirmation that we should expect from kafka plugin to translate each flow record matching {router, ifindex) into json and sent to kafka. Thanks again Wilfrid -----Original Message----- From: Paolo Lucente <pa...@pmacct.net> Sent: Friday, 21 October 2022 15:37 To: pmacct-discussion@pmacct.net; Grassot, Wilfrid <wgras...@pccwglobal.com> Subject: Re: [pmacct-discussion] kafka plugin and number of json messages vs number of netflow record CAUTION: External email. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Wilfrid, To say whether some aggregation is taking place or not, you should look at the template of the incoming NetFlow records. You can achieve this with Wireshark / tshark or via pmacct, either running it in debug mode - you will find the templates in the log file - or defining a nfacctd_templates_file. In general, i would expect less JSON records output to Kafka than incoming NetFlow records because of the templates - which are really service messages to make the protocol work and hence do not make it to the database. Paolo On 21/10/22 09:13, Grassot, Wilfrid wrote: > Hi Paolo > > We are collecting netflow records of several routers interfaces. > > Now we are testing the kafka plugin of nfactt using json as format output. > > kafka_topic[l3vpn]: pmacct_netflow > > aggregate[l3vpn]: tcpflags, proto, src_host, src_port, dst_host, > dst_port, src_as, dst_as, peer_src_as, peer_dst_as, peer_src_ip, > peer_dst_ip, in_iface, src_net, > > dst_net, tos, timestamp_start, timestamp_end > > kafka_broker_host[l3vpn]: > kafka-node-1.interstellar.prv:9092,kafka-node-2.interstellar.prv:9092, > kafka-node-3.interstellar.prv:9092 > > kafka_output[l3vpn]: json > > kafka_topic[l3vpn]: pmacct_netflow > > Is this setup converting 1 for 1 a netflow record to a json message ? > > I am asking because the backend engineers are noticing a lot of > difference between the number of netflow records received and the > number of the json messages kafka is receiving. > > Is there a kind of aggregation done by kafka plugin that would reduce > the number of json messages sent to Kafka ? > > Thank you in advance. > > Wilfrid Grassot > > ** > > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists