Hi Wilfrid,
Ah, i see & thanks for your kind words on the pmacct project.
You can define Kafka brokers and topics as part of the tee_receivers
map:
https://github.com/pmacct/pmacct/blob/master/examples/tee_receivers.lst.example#L29-#L32
. But you can have only one set of configuration for all brokers for one
single tee plugin (where you can define SSL stuff, buffers, etc.), hence
the separate option tee_kafka_config_file (action item for me, i will
augment the CONFIG-KEYS doc to specifically say all of this).
The above is all for the sender end. For the receiving end, please see
CONFIG-KEYS for nfacctd_kafka_* options. In this case binary NetFlow
data is shuffled as-is into Kafka (prepending some socket info so that
the collector can reckon which router originally sent the datagram) and
unpacked at the receiving end by nfacctd - all of this is "non
standard", i mean, there is no such a thing as using Kafka as transport
for NetFlow so this is is a pmacct-to-pmacct thing. Essentially, big
picture is:
router -> nfacctd (tee) -> Kafka -> nfacctd (collector)
Paolo
On 12/5/21 10:52, Grassot, Wilfrid wrote:
Paolo,
Thank you for your answer.
My bad, my question have been badly expressed, as I meant to ask about
specifically "tee plugins" sending replicated data straight to kafka.
https://github.com/pmacct/pmacct/commit/f660b083ccccb505c969c623e15b1dbb9e
27ffac
and related parameter tee_kafka_config_file
https://github.com/pmacct/pmacct/blob/master/ChangeLog#L579-L582
I cannot see where tee plugin specifies the kafka topic to send netflow
to, and how Kafka can handle received netflow replicated from tee plugin.
Somewhere there must be a plugin on Kafka side that handle the conversion
of netflow to send it to a topic in a data format retrievable from a
topic.
I hope my question a bit clearer.
Thanks for your usual support and for your pmacct swiss knife
Wilfrid
-----Original Message-----
From: Paolo Lucente <pa...@pmacct.net>
Sent: Tuesday, 11 May 2021 04:52
To: pmacct-discussion@pmacct.net; Grassot, Wilfrid
<wgras...@pccwglobal.com>
Subject: Re: [pmacct-discussion] Tee and Kafka plugins
CAUTION: External email. Do not click links or open attachments unless
you recognize the sender and know the content is safe.
Hi Wilfrid,
Your understanding is correct although replication and collection are two
separate pieces. You can have 1) a nfacctd replicator, that is binary
NetFlow to binary NetFlow, where you could fan-out and filter pieces of
your original export (to different collector, apps, etc.) and
2) a nfacctd collector that is parsing binary NetFlow and sending into
Kafka; this piece should be business as usual and encodings you can pick
are JSON or Apache Avro.
You can find examples and some introductory elaboration around the
replication (tee) plugin here:
https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2105-#L2167
Paolo
On 10/5/21 12:28, Grassot, Wilfrid wrote:
Hi Paolo,
I understand that we can configure tee to replicate filtered datagram
with pre-tag.map to a kafka broker.
I am novice to Kafka, but does it mean the data sent to Kafka is in
json format ? If not which other data format is it sent to Kafka ?
I am not sure either how would look like the configuration nfacct with
the tee plugin to replicate netflow packet to a kafka broker, would
you have an example ?
Many thanks .
Wilfrid
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
