Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-22 Thread Emanuel dos Reis Rodrigues
Thank you so much! On Wed, Apr 22, 2020, 12:01 PM Brian Solar wrote: > use the named configuration feature: > > kafka_topic[config_name]: netflow > kafka_broker_host[config_name]: 192.168100.105 > ... > ... > > > > ‐‐‐ Original Message ‐‐‐ > On Sunday, April 19, 2020 5:51 PM, Emanuel

Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-22 Thread Brian Solar
use the named configuration feature: kafka_topic[config_name]: netflow kafka_broker_host[config_name]: 192.168100.105 ... ... ‐‐‐ Original Message ‐‐‐ On Sunday, April 19, 2020 5:51 PM, Emanuel dos Reis Rodrigues wrote: > I see, I actually tried it before and the realized the write_id

Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-19 Thread Emanuel dos Reis Rodrigues
I see, I actually tried it before and the realized the write_id was changing based on the PID of nfacctd. Do you know what is the parameter to customize the writer_id ? Thanks ! Best Regards, Emanuel On Sun, Apr 19, 2020 at 11:42 AM Brian Solar wrote: > You already seem to have a solution,

Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-19 Thread Brian Solar
You already seem to have a solution, but to me the writer_id is what you want. Change the name of the process in your configuration file. ‐‐‐ Original Message ‐‐‐ On Wednesday, April 15, 2020 7:33 PM, Emanuel dos Reis Rodrigues wrote: > Hey, I just realize it worked. I think I was

Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-15 Thread Emanuel dos Reis Rodrigues
Hey, I just realize it worked. I think I was little behind on the messages parking on my kafka, now I can see the tag. Thank you so much for your help. On Wed, Apr 15, 2020 at 10:33 AM Emanuel dos Reis Rodrigues < emanueldosr...@gmail.com> wrote: > I am using: > > NetFlow Accounting Daemon,

Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-15 Thread Emanuel dos Reis Rodrigues
I am using: NetFlow Accounting Daemon, nfacctd 1.7.2-git (20181018-00+c3) Arguments: '--enable-kafka' '--enable-jansson' 'JANSSON_CFLAGS=-I/usr/local/include/' 'JANSSON_LIBS=-L/usr/local/lib -ljansson' '--enable-l2' '--enable-ipv6' '--enable-64bit' '--enable-traffic-bins' '--enable-bgp-bins'

Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-15 Thread Paolo Lucente
Hey Emanuel, The config is correct and I did try your same config and that does work for me, ie.: $ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic pmacct.flows {"event_type": "purge", "tag": 1, [ .. ]} What version of the software are you using? Is it 1.7.4p1

Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-14 Thread Emanuel dos Reis Rodrigues
I tried, follow my config: kafka_topic: netflow kafka_broker_host: 192.168100.105 kafka_broker_port: 9092 kafka_refresh_time: 1 #daemonize: true plugins: kafka nfacctd_port: 9995 post_tag: 1 aggregate: tag, peer_src_ip, src_host, dst_host, timestamp_start, timestamp_end, src_port, dst_port, proto

Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-14 Thread Paolo Lucente
I may have skipped the important detail you need to add the 'tag' key to your 'aggregate' line in the config, my bad. This is in addition to, say, 'post_tag: 1' to identify collector 1. Let me know how it goes. Paolo On Tue, Apr 14, 2020 at 10:18:55AM -0400, Emanuel dos Reis Rodrigues wrote: >

Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-14 Thread Emanuel dos Reis Rodrigues
Thank you man, I did this test but I did not see the id being pushed along with the Netflow info to Kafka topic. Is there the place the information would show up ? On Tue, Apr 14, 2020 at 9:15 AM Paolo Lucente wrote: > > Hi Emanuel, > > Apologies i did not get you wanted and ID for the

Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-14 Thread Paolo Lucente
Hi Emanuel, Apologies i did not get you wanted and ID for the collector. The simplest way of achieving that is 'post_tag' as you just have to supply a number as ID; pre_tag_map expects a map and may be better to be reserved for more complex use-cases. Paolo On Mon, Apr 13, 2020 at 03:35:52PM

Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-13 Thread Emanuel dos Reis Rodrigues
Thank you for your help. Appreciate it ! See, I did use it for testing after I sent this email. However, the ip showed there was the IP from my nfacctd machine, the collector itself. Not the exporter. peer_src_ip : IP address or identificator of telemetry exporting device

Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-13 Thread Paolo Lucente
Hi Emanuel, I think you are looking for (i admit, non-intuitive) 'peer_src_ip' primitive: $ nfacctd -a | grep peer_src_ip peer_src_ip : IP address or identificator of telemetry exporting device Without the grep you can see all supported primitives by the nfacctd release

[pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-12 Thread Emanuel dos Reis Rodrigues
Hello guys, I implemented nfacctd acting as a Netflow collector using pmacct. It is working perfectly and writing the flows to a Kafka topic which I have an application processing it. Following is my configuration: kafka_topic: netflow kafka_broker_host: Kafka-host kafka_broker_port: 9092