Paolo, Thank you for your answer.
My bad, my question have been badly expressed, as I meant to ask about specifically "tee plugins" sending replicated data straight to kafka. https://github.com/pmacct/pmacct/commit/f660b083ccccb505c969c623e15b1dbb9e 27ffac and related parameter tee_kafka_config_file https://github.com/pmacct/pmacct/blob/master/ChangeLog#L579-L582 I cannot see where tee plugin specifies the kafka topic to send netflow to, and how Kafka can handle received netflow replicated from tee plugin. Somewhere there must be a plugin on Kafka side that handle the conversion of netflow to send it to a topic in a data format retrievable from a topic. I hope my question a bit clearer. Thanks for your usual support and for your pmacct swiss knife Wilfrid -----Original Message----- From: Paolo Lucente <pa...@pmacct.net> Sent: Tuesday, 11 May 2021 04:52 To: pmacct-discussion@pmacct.net; Grassot, Wilfrid <wgras...@pccwglobal.com> Subject: Re: [pmacct-discussion] Tee and Kafka plugins CAUTION: External email. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Wilfrid, Your understanding is correct although replication and collection are two separate pieces. You can have 1) a nfacctd replicator, that is binary NetFlow to binary NetFlow, where you could fan-out and filter pieces of your original export (to different collector, apps, etc.) and 2) a nfacctd collector that is parsing binary NetFlow and sending into Kafka; this piece should be business as usual and encodings you can pick are JSON or Apache Avro. You can find examples and some introductory elaboration around the replication (tee) plugin here: https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2105-#L2167 Paolo On 10/5/21 12:28, Grassot, Wilfrid wrote: > Hi Paolo, > > I understand that we can configure tee to replicate filtered datagram > with pre-tag.map to a kafka broker. > > I am novice to Kafka, but does it mean the data sent to Kafka is in > json format ? If not which other data format is it sent to Kafka ? > > I am not sure either how would look like the configuration nfacct with > the tee plugin to replicate netflow packet to a kafka broker, would > you have an example ? > > Many thanks . > > Wilfrid > > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists > _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
