I am using:
NetFlow Accounting Daemon, nfacctd 1.7.2-git (20181018-00+c3)
Arguments:
'--enable-kafka' '--enable-jansson' 'JANSSON_CFLAGS=-I/usr/local/include/'
'JANSSON_LIBS=-L/usr/local/lib -ljansson' '--enable-l2' '--enable-ipv6'
'--enable-64bit' '--enable-traffic-bins' '--enable-bgp-bins'
'--enable-bmp-bins' '--enable-st-bins'
Libs:
libpcap version 1.5.3
rdkafka 0.11.4
jansson 2.12
I can upgrade it to a newer version and try again.
On Wed, Apr 15, 2020 at 8:59 AM Paolo Lucente <pa...@pmacct.net> wrote:
>
> Hey Emanuel,
>
> The config is correct and I did try your same config and that does work
> for me, ie.:
>
> $ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092
> --topic pmacct.flows
> {"event_type": "purge", "tag": 1, [ .. ]}
>
> What version of the software are you using? Is it 1.7.4p1 (latest
> stable) or master code from GitHub? If so, is it possible an old running
> nfacctd process is reading the data instead of the newly configured one?
>
> Paolo
>
> On Wed, Apr 15, 2020 at 12:17:43AM -0400, Emanuel dos Reis Rodrigues wrote:
> > I tried, follow my config:
> >
> > kafka_topic: netflow
> > kafka_broker_host: 192.168100.105
> > kafka_broker_port: 9092
> > kafka_refresh_time: 1
> > #daemonize: true
> > plugins: kafka
> > nfacctd_port: 9995
> > post_tag: 1
> > aggregate: tag, peer_src_ip, src_host, dst_host, timestamp_start,
> > timestamp_end, src_port, dst_port, proto
> >
> >
> > I kept the peer_src_ip, but the tag one is not being posted to Kafka.
> >
> > {'event_type': 'purge', 'peer_ip_src': '172.18.0.2', 'ip_src':
> > '192.168.1.100', 'ip_dst': 'x.46.x.245', 'port_src': 51184, 'port_dst':
> > 443, 'ip_proto': 'tcp', 'timestamp_start': '2020-04-14 14:15:39.000000',
> > 'timestamp_end': '2020-04-14 14:15:54.000000', 'packets': 5, 'bytes':
> 260,
> > 'writer_id': 'default_kafka/75091'}
> >
> > Did I miss anything ?
> >
> >
> > Thanks !
> >
> >
> >
> > On Tue, Apr 14, 2020 at 10:26 AM Paolo Lucente <pa...@pmacct.net> wrote:
> >
> > >
> > > I may have skipped the important detail you need to add the 'tag' key
> to
> > > your 'aggregate' line in the config, my bad. This is in addition to,
> say,
> > > 'post_tag: 1' to identify collector 1. Let me know how it goes.
> > >
> > > Paolo
> > >
> > > On Tue, Apr 14, 2020 at 10:18:55AM -0400, Emanuel dos Reis Rodrigues
> wrote:
> > > > Thank you man, I did this test but I did not see the id being pushed
> > > along
> > > > with the Netflow info to Kafka topic. Is there the place the
> information
> > > > would show up ?
> > > >
> > > >
> > > > On Tue, Apr 14, 2020 at 9:15 AM Paolo Lucente <pa...@pmacct.net>
> wrote:
> > > >
> > > > >
> > > > > Hi Emanuel,
> > > > >
> > > > > Apologies i did not get you wanted and ID for the collector. The
> > > > > simplest way of achieving that is 'post_tag' as you just have to
> supply
> > > > > a number as ID; pre_tag_map expects a map and may be better to be
> > > > > reserved for more complex use-cases.
> > > > >
> > > > > Paolo
> > > > >
> > > > > On Mon, Apr 13, 2020 at 03:35:52PM -0400, Emanuel dos Reis
> Rodrigues
> > > wrote:
> > > > > > Thank you for your help. Appreciate it !
> > > > > >
> > > > > > See, I did use it for testing after I sent this email. However,
> the
> > > ip
> > > > > > showed there was the IP from my nfacctd machine, the collector
> > > itself.
> > > > > Not
> > > > > > the exporter.
> > > > > >
> > > > > > peer_src_ip : IP address or identificator of
> > > > > telemetry
> > > > > > exporting device
> > > > > >
> > > > > > In fact, it may have todo with the fact I currently have an SSH
> > > tunnel
> > > > > with
> > > > > > socat with the remote machine in order to collect the data. This
> may
> > > be
> > > > > the
> > > > > > reason why which is definitively not a ordinary condition. :)
> > > > > >
> > > > > > I am wondering if I could use this one to include a different
> tag on
> > > it
> > > > > > process/collector, but have not yet figured out how. Any
> thoughts ?
> > > > > >
> > > > > > label : String label, ie. as result of
> > > > > > pre_tag_map evaluation
> > > > > >
> > > > > >
> > > > > > Thank you again.
> > > > > >
> > > > > > On Mon, Apr 13, 2020 at 9:07 AM Paolo Lucente <pa...@pmacct.net>
> > > wrote:
> > > > > >
> > > > > > >
> > > > > > > Hi Emanuel,
> > > > > > >
> > > > > > > I think you are looking for (i admit, non-intuitive)
> 'peer_src_ip'
> > > > > > > primitive:
> > > > > > >
> > > > > > > $ nfacctd -a | grep peer_src_ip
> > > > > > > peer_src_ip : IP address or identificator
> of
> > > > > > > telemetry exporting device
> > > > > > >
> > > > > > > Without the grep you can see all supported primitives by the
> > > nfacctd
> > > > > > > release you are using along with a text explanation.
> > > > > > >
> > > > > > > Paolo
> > > > > > >
> > > > > > > On Sun, Apr 12, 2020 at 06:55:26PM -0400, Emanuel dos Reis
> > > Rodrigues
> > > > > wrote:
> > > > > > > > Hello guys,
> > > > > > > >
> > > > > > > > I implemented nfacctd acting as a Netflow collector using
> > > pmacct. It
> > > > > is
> > > > > > > > working perfectly and writing the flows to a Kafka topic
> which I
> > > > > have an
> > > > > > > > application processing it.
> > > > > > > >
> > > > > > > > Following is my configuration:
> > > > > > > >
> > > > > > > > kafka_topic: netflow
> > > > > > > > kafka_broker_host: Kafka-host
> > > > > > > > kafka_broker_port: 9092
> > > > > > > > kafka_refresh_time: 1
> > > > > > > > daemonize: true
> > > > > > > > plugins: kafka
> > > > > > > > pcap_interface: enp0s8
> > > > > > > > nfacctd_ip: 192.168.1.100
> > > > > > > > nfacctd_port: 9995
> > > > > > > > aggregate: src_host, dst_host, timestamp_start,
> timestamp_end,
> > > > > src_port,
> > > > > > > > dst_port, proto
> > > > > > > >
> > > > > > > > Currently, there is only one Netflow exporter sending data to
> > > this
> > > > > > > > demon and I would like to add another exporter. The problem
> is
> > > that
> > > > > I am
> > > > > > > > not finding a way to differentiate the flows coming from
> > > different
> > > > > > > > exporters.
> > > > > > > >
> > > > > > > > Let's say I have the exporter A currently sending data to
> nfacctd
> > > > > running
> > > > > > > > at port 9995 and the data is being written to Kafka topic
> > > Netflow.
> > > > > > > >
> > > > > > > > Now I want a new exporter B to start sending data to nfacctd
> port
> > > > > 9996
> > > > > > > which
> > > > > > > > will be running as a separate demon ( just because I though
> so,
> > > not
> > > > > sure
> > > > > > > > yet if it is a necessary approach) and writing the data to
> the
> > > > > > > > same Netflow topic in Kafka.
> > > > > > > >
> > > > > > > > When the data comes from Kafka to my application, I cannot
> tell
> > > from
> > > > > > > > which exporter the data came from. I would need some sort of
> > > > > > > identification
> > > > > > > > in order to make this differentiation. It is important for
> me,
> > > > > because my
> > > > > > > > application may treat differently Netflow traffic coming from
> > > these
> > > > > > > > two Netflow exporters.
> > > > > > > >
> > > > > > > > Thanks in advance.
> > > > > > > >
> > > > > > > > Emanuel
> > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > pmacct-discussion mailing list
> > > > > > > > http://www.pmacct.net/#mailinglists
> > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > pmacct-discussion mailing list
> > > > > > > http://www.pmacct.net/#mailinglists
> > > > > > >
> > > > >
> > >
>
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
