Hey Emanuel,

The config is correct and I did try your same config and that does work
for me, ie.:

$ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic 
pmacct.flows
{"event_type": "purge", "tag": 1, [ .. ]}

What version of the software are you using? Is it 1.7.4p1 (latest
stable) or master code from GitHub? If so, is it possible an old running
nfacctd process is reading the data instead of the newly configured one?

Paolo 

On Wed, Apr 15, 2020 at 12:17:43AM -0400, Emanuel dos Reis Rodrigues wrote:
> I tried, follow my config:
> 
> kafka_topic: netflow
> kafka_broker_host: 192.168100.105
> kafka_broker_port: 9092
> kafka_refresh_time: 1
> #daemonize: true
> plugins: kafka
> nfacctd_port: 9995
> post_tag: 1
> aggregate: tag, peer_src_ip, src_host, dst_host, timestamp_start,
> timestamp_end, src_port, dst_port, proto
> 
> 
> I kept the peer_src_ip, but the tag one is not being posted to Kafka.
> 
> {'event_type': 'purge', 'peer_ip_src': '172.18.0.2', 'ip_src':
> '192.168.1.100', 'ip_dst': 'x.46.x.245', 'port_src': 51184, 'port_dst':
> 443, 'ip_proto': 'tcp', 'timestamp_start': '2020-04-14 14:15:39.000000',
> 'timestamp_end': '2020-04-14 14:15:54.000000', 'packets': 5, 'bytes': 260,
> 'writer_id': 'default_kafka/75091'}
> 
> Did I miss anything ?
> 
> 
> Thanks !
> 
> 
> 
> On Tue, Apr 14, 2020 at 10:26 AM Paolo Lucente <pa...@pmacct.net> wrote:
> 
> >
> > I may have skipped the important detail you need to add the 'tag' key to
> > your 'aggregate' line in the config, my bad. This is in addition to, say,
> > 'post_tag: 1' to identify collector 1. Let me know how it goes.
> >
> > Paolo
> >
> > On Tue, Apr 14, 2020 at 10:18:55AM -0400, Emanuel dos Reis Rodrigues wrote:
> > > Thank you man, I did this test but I did not see the id being pushed
> > along
> > > with the Netflow info to Kafka topic. Is there the place the information
> > > would show up ?
> > >
> > >
> > > On Tue, Apr 14, 2020 at 9:15 AM Paolo Lucente <pa...@pmacct.net> wrote:
> > >
> > > >
> > > > Hi Emanuel,
> > > >
> > > > Apologies i did not get you wanted and ID for the collector. The
> > > > simplest way of achieving that is 'post_tag' as you just have to supply
> > > > a number as ID; pre_tag_map expects a map and may be better to be
> > > > reserved for more complex use-cases.
> > > >
> > > > Paolo
> > > >
> > > > On Mon, Apr 13, 2020 at 03:35:52PM -0400, Emanuel dos Reis Rodrigues
> > wrote:
> > > > > Thank you for your help. Appreciate it !
> > > > >
> > > > > See, I did use it for testing after I sent this email. However, the
> > ip
> > > > > showed there was the IP from my nfacctd machine, the collector
> > itself.
> > > > Not
> > > > > the exporter.
> > > > >
> > > > > peer_src_ip                      : IP address or identificator of
> > > > telemetry
> > > > > exporting device
> > > > >
> > > > > In fact, it may have todo with the fact I currently have an SSH
> > tunnel
> > > > with
> > > > > socat with the remote machine in order to collect the data. This may
> > be
> > > > the
> > > > > reason why which is definitively not a ordinary condition. :)
> > > > >
> > > > > I am wondering if I could use this one to include a different tag on
> > it
> > > > > process/collector, but have not yet figured out how. Any thoughts ?
> > > > >
> > > > > label                            : String label, ie. as result of
> > > > > pre_tag_map evaluation
> > > > >
> > > > >
> > > > > Thank you again.
> > > > >
> > > > > On Mon, Apr 13, 2020 at 9:07 AM Paolo Lucente <pa...@pmacct.net>
> > wrote:
> > > > >
> > > > > >
> > > > > > Hi Emanuel,
> > > > > >
> > > > > > I think you are looking for (i admit, non-intuitive) 'peer_src_ip'
> > > > > > primitive:
> > > > > >
> > > > > > $ nfacctd -a | grep peer_src_ip
> > > > > > peer_src_ip                      : IP address or identificator of
> > > > > > telemetry exporting device
> > > > > >
> > > > > > Without the grep you can see all supported primitives by the
> > nfacctd
> > > > > > release you are using along with a text explanation.
> > > > > >
> > > > > > Paolo
> > > > > >
> > > > > > On Sun, Apr 12, 2020 at 06:55:26PM -0400, Emanuel dos Reis
> > Rodrigues
> > > > wrote:
> > > > > > > Hello guys,
> > > > > > >
> > > > > > > I implemented nfacctd acting as a Netflow collector using
> > pmacct. It
> > > > is
> > > > > > > working perfectly and writing the flows to a Kafka topic which I
> > > > have an
> > > > > > > application processing it.
> > > > > > >
> > > > > > > Following is my configuration:
> > > > > > >
> > > > > > > kafka_topic: netflow
> > > > > > > kafka_broker_host: Kafka-host
> > > > > > > kafka_broker_port: 9092
> > > > > > > kafka_refresh_time: 1
> > > > > > > daemonize: true
> > > > > > > plugins: kafka
> > > > > > > pcap_interface: enp0s8
> > > > > > > nfacctd_ip: 192.168.1.100
> > > > > > > nfacctd_port: 9995
> > > > > > > aggregate: src_host, dst_host, timestamp_start, timestamp_end,
> > > > src_port,
> > > > > > > dst_port, proto
> > > > > > >
> > > > > > > Currently, there is only one Netflow exporter sending data to
> > this
> > > > > > > demon and I would like to add another exporter. The problem is
> > that
> > > > I am
> > > > > > > not finding a way to differentiate the flows coming from
> > different
> > > > > > > exporters.
> > > > > > >
> > > > > > > Let's say I have the exporter A currently sending data to nfacctd
> > > > running
> > > > > > > at port 9995 and the data is being written to Kafka topic
> > Netflow.
> > > > > > >
> > > > > > > Now I want a new exporter B to start sending data to nfacctd port
> > > > 9996
> > > > > > which
> > > > > > > will be running as a separate demon ( just because I though so,
> > not
> > > > sure
> > > > > > > yet if it is a necessary approach)  and writing the data to the
> > > > > > > same Netflow topic in Kafka.
> > > > > > >
> > > > > > > When the data comes from Kafka to my application, I cannot tell
> > from
> > > > > > > which exporter the data came from. I would need some sort of
> > > > > > identification
> > > > > > > in order to make this differentiation. It is important for me,
> > > > because my
> > > > > > > application may treat differently Netflow traffic coming from
> > these
> > > > > > > two Netflow exporters.
> > > > > > >
> > > > > > > Thanks in advance.
> > > > > > >
> > > > > > > Emanuel
> > > > > >
> > > > > > > _______________________________________________
> > > > > > > pmacct-discussion mailing list
> > > > > > > http://www.pmacct.net/#mailinglists
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > pmacct-discussion mailing list
> > > > > > http://www.pmacct.net/#mailinglists
> > > > > >
> > > >
> >

_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Reply via email to