Hi Paolo,

Thanks for your feedback.
We are now filtering the template messages from the accounting of netflow 
messages, but still the number of flows are l kind of 3 times more than the 
number of json messages.

To troubleshoot, we did focus on one specific router, now comparing the 
number of flow received  for (router ; ifindex) pair with the number of json 
messages for the same  (router ; ifindex) pair
And we have the kind of comparaison:

exporter IP address     !  ifIndex      !  netflow_count        !  json_count

xxxx.xxxx.xxxx.xxxx     !  464          !    91144              !   
19491xxxx.xxxx.xxxx.xxxx    !  820          !    3900                       !   
919

xxxx.xxxx.xxxx.xxxx     !  959          !    11219              !   1918

xxxx.xxxx.xxxx.xxxx     !  756          !    280                        !    59

xxxx.xxxx.xxxx.xxxx     !  757          !    293                        !    
56Obviously I am not asking to troubleshoot, but I would like again 
confirmation that we should expect from kafka plugin to translate each flow 
record matching {router, ifindex)  into json and sent to kafka.

Thanks again

Wilfrid







-----Original Message-----
From: Paolo Lucente <pa...@pmacct.net>
Sent: Friday, 21 October 2022 15:37
To: pmacct-discussion@pmacct.net; Grassot, Wilfrid <wgras...@pccwglobal.com>
Subject: Re: [pmacct-discussion] kafka plugin and number of json messages vs 
number of netflow record

CAUTION:  External email. Do not click links or open attachments unless you 
recognize the sender and know the content is safe.

Hi Wilfrid,

To say whether some aggregation is taking place or not, you should look at 
the template of the incoming NetFlow records. You can achieve this with 
Wireshark / tshark or via pmacct, either running it in debug mode - you will 
find the templates in the log file - or defining a nfacctd_templates_file.

In general, i would expect less JSON records output to Kafka than incoming 
NetFlow records because of the templates - which are really service messages 
to make the protocol work and hence do not make it to the database.

Paolo


On 21/10/22 09:13, Grassot, Wilfrid wrote:
> Hi Paolo
>
> We are  collecting netflow records of several routers interfaces.
>
> Now we are testing the kafka plugin of nfactt using json as format output.
>
> kafka_topic[l3vpn]: pmacct_netflow
>
> aggregate[l3vpn]: tcpflags, proto, src_host, src_port, dst_host,
> dst_port, src_as, dst_as, peer_src_as, peer_dst_as, peer_src_ip,
> peer_dst_ip, in_iface, src_net,
>
> dst_net, tos, timestamp_start, timestamp_end
>
> kafka_broker_host[l3vpn]:
> kafka-node-1.interstellar.prv:9092,kafka-node-2.interstellar.prv:9092,
> kafka-node-3.interstellar.prv:9092
>
> kafka_output[l3vpn]: json
>
> kafka_topic[l3vpn]: pmacct_netflow
>
> Is this setup converting 1 for 1 a netflow record to a json message ?
>
> I am asking because the backend engineers are noticing a lot of
> difference between the number of netflow records received and the
> number of the json messages kafka is receiving.
>
> Is there a kind of aggregation done by kafka plugin that would reduce
> the number of json messages sent to Kafka ?
>
> Thank you in advance.
>
> Wilfrid Grassot
>
> **
>
>
> _______________________________________________
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists

_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Reply via email to