I tried, follow my config:
kafka_topic: netflow
kafka_broker_host: 192.168100.105
kafka_broker_port: 9092
kafka_refresh_time: 1
#daemonize: true
plugins: kafka
nfacctd_port: 9995
post_tag: 1
aggregate: tag, peer_src_ip, src_host, dst_host, timestamp_start,
timestamp_end, src_port, dst_port, proto
I kept the peer_src_ip, but the tag one is not being posted to Kafka.
{'event_type': 'purge', 'peer_ip_src': '172.18.0.2', 'ip_src':
'192.168.1.100', 'ip_dst': 'x.46.x.245', 'port_src': 51184, 'port_dst':
443, 'ip_proto': 'tcp', 'timestamp_start': '2020-04-14 14:15:39.000000',
'timestamp_end': '2020-04-14 14:15:54.000000', 'packets': 5, 'bytes': 260,
'writer_id': 'default_kafka/75091'}
Did I miss anything ?
Thanks !
On Tue, Apr 14, 2020 at 10:26 AM Paolo Lucente <pa...@pmacct.net> wrote:
>
> I may have skipped the important detail you need to add the 'tag' key to
> your 'aggregate' line in the config, my bad. This is in addition to, say,
> 'post_tag: 1' to identify collector 1. Let me know how it goes.
>
> Paolo
>
> On Tue, Apr 14, 2020 at 10:18:55AM -0400, Emanuel dos Reis Rodrigues wrote:
> > Thank you man, I did this test but I did not see the id being pushed
> along
> > with the Netflow info to Kafka topic. Is there the place the information
> > would show up ?
> >
> >
> > On Tue, Apr 14, 2020 at 9:15 AM Paolo Lucente <pa...@pmacct.net> wrote:
> >
> > >
> > > Hi Emanuel,
> > >
> > > Apologies i did not get you wanted and ID for the collector. The
> > > simplest way of achieving that is 'post_tag' as you just have to supply
> > > a number as ID; pre_tag_map expects a map and may be better to be
> > > reserved for more complex use-cases.
> > >
> > > Paolo
> > >
> > > On Mon, Apr 13, 2020 at 03:35:52PM -0400, Emanuel dos Reis Rodrigues
> wrote:
> > > > Thank you for your help. Appreciate it !
> > > >
> > > > See, I did use it for testing after I sent this email. However, the
> ip
> > > > showed there was the IP from my nfacctd machine, the collector
> itself.
> > > Not
> > > > the exporter.
> > > >
> > > > peer_src_ip : IP address or identificator of
> > > telemetry
> > > > exporting device
> > > >
> > > > In fact, it may have todo with the fact I currently have an SSH
> tunnel
> > > with
> > > > socat with the remote machine in order to collect the data. This may
> be
> > > the
> > > > reason why which is definitively not a ordinary condition. :)
> > > >
> > > > I am wondering if I could use this one to include a different tag on
> it
> > > > process/collector, but have not yet figured out how. Any thoughts ?
> > > >
> > > > label : String label, ie. as result of
> > > > pre_tag_map evaluation
> > > >
> > > >
> > > > Thank you again.
> > > >
> > > > On Mon, Apr 13, 2020 at 9:07 AM Paolo Lucente <pa...@pmacct.net>
> wrote:
> > > >
> > > > >
> > > > > Hi Emanuel,
> > > > >
> > > > > I think you are looking for (i admit, non-intuitive) 'peer_src_ip'
> > > > > primitive:
> > > > >
> > > > > $ nfacctd -a | grep peer_src_ip
> > > > > peer_src_ip : IP address or identificator of
> > > > > telemetry exporting device
> > > > >
> > > > > Without the grep you can see all supported primitives by the
> nfacctd
> > > > > release you are using along with a text explanation.
> > > > >
> > > > > Paolo
> > > > >
> > > > > On Sun, Apr 12, 2020 at 06:55:26PM -0400, Emanuel dos Reis
> Rodrigues
> > > wrote:
> > > > > > Hello guys,
> > > > > >
> > > > > > I implemented nfacctd acting as a Netflow collector using
> pmacct. It
> > > is
> > > > > > working perfectly and writing the flows to a Kafka topic which I
> > > have an
> > > > > > application processing it.
> > > > > >
> > > > > > Following is my configuration:
> > > > > >
> > > > > > kafka_topic: netflow
> > > > > > kafka_broker_host: Kafka-host
> > > > > > kafka_broker_port: 9092
> > > > > > kafka_refresh_time: 1
> > > > > > daemonize: true
> > > > > > plugins: kafka
> > > > > > pcap_interface: enp0s8
> > > > > > nfacctd_ip: 192.168.1.100
> > > > > > nfacctd_port: 9995
> > > > > > aggregate: src_host, dst_host, timestamp_start, timestamp_end,
> > > src_port,
> > > > > > dst_port, proto
> > > > > >
> > > > > > Currently, there is only one Netflow exporter sending data to
> this
> > > > > > demon and I would like to add another exporter. The problem is
> that
> > > I am
> > > > > > not finding a way to differentiate the flows coming from
> different
> > > > > > exporters.
> > > > > >
> > > > > > Let's say I have the exporter A currently sending data to nfacctd
> > > running
> > > > > > at port 9995 and the data is being written to Kafka topic
> Netflow.
> > > > > >
> > > > > > Now I want a new exporter B to start sending data to nfacctd port
> > > 9996
> > > > > which
> > > > > > will be running as a separate demon ( just because I though so,
> not
> > > sure
> > > > > > yet if it is a necessary approach) and writing the data to the
> > > > > > same Netflow topic in Kafka.
> > > > > >
> > > > > > When the data comes from Kafka to my application, I cannot tell
> from
> > > > > > which exporter the data came from. I would need some sort of
> > > > > identification
> > > > > > in order to make this differentiation. It is important for me,
> > > because my
> > > > > > application may treat differently Netflow traffic coming from
> these
> > > > > > two Netflow exporters.
> > > > > >
> > > > > > Thanks in advance.
> > > > > >
> > > > > > Emanuel
> > > > >
> > > > > > _______________________________________________
> > > > > > pmacct-discussion mailing list
> > > > > > http://www.pmacct.net/#mailinglists
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > pmacct-discussion mailing list
> > > > > http://www.pmacct.net/#mailinglists
> > > > >
> > >
>
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
