use the named configuration feature:
kafka_topic[config_name]: netflow
kafka_broker_host[config_name]: 192.168100.105
...
...
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Sunday, April 19, 2020 5:51 PM, Emanuel dos Reis Rodrigues
<emanueldosr...@gmail.com> wrote:
> I see, I actually tried it before and the realized the write_id was changing
> based on the PID of nfacctd. Do you know what is the parameter to customize
> the writer_id ?
>
> Thanks !
>
> Best Regards,
> Emanuel
>
> On Sun, Apr 19, 2020 at 11:42 AM Brian Solar <brian.so...@pm.me> wrote:
>
>> You already seem to have a solution, but to me the writer_id is what you
>> want. Change the name of the process in your configuration file.
>>
>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>> On Wednesday, April 15, 2020 7:33 PM, Emanuel dos Reis Rodrigues
>> <emanueldosr...@gmail.com> wrote:
>>
>>> Hey, I just realize it worked. I think I was little behind on the messages
>>> parking on my kafka, now I can see the tag.
>>>
>>> Thank you so much for your help.
>>>
>>> On Wed, Apr 15, 2020 at 10:33 AM Emanuel dos Reis Rodrigues
>>> <emanueldosr...@gmail.com> wrote:
>>>
>>>> I am using:
>>>>
>>>> NetFlow Accounting Daemon, nfacctd 1.7.2-git (20181018-00+c3)
>>>>
>>>> Arguments:
>>>> '--enable-kafka' '--enable-jansson'
>>>> 'JANSSON_CFLAGS=-I/usr/local/include/' 'JANSSON_LIBS=-L/usr/local/lib
>>>> -ljansson' '--enable-l2' '--enable-ipv6' '--enable-64bit'
>>>> '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
>>>> '--enable-st-bins'
>>>>
>>>> Libs:
>>>> libpcap version 1.5.3
>>>> rdkafka 0.11.4
>>>> jansson 2.12
>>>>
>>>> I can upgrade it to a newer version and try again.
>>>>
>>>> On Wed, Apr 15, 2020 at 8:59 AM Paolo Lucente <pa...@pmacct.net> wrote:
>>>>
>>>>> Hey Emanuel,
>>>>>
>>>>> The config is correct and I did try your same config and that does work
>>>>> for me, ie.:
>>>>>
>>>>> $ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092
>>>>> --topic pmacct.flows
>>>>> {"event_type": "purge", "tag": 1, [ .. ]}
>>>>>
>>>>> What version of the software are you using? Is it 1.7.4p1 (latest
>>>>> stable) or master code from GitHub? If so, is it possible an old running
>>>>> nfacctd process is reading the data instead of the newly configured one?
>>>>>
>>>>> Paolo
>>>>>
>>>>> On Wed, Apr 15, 2020 at 12:17:43AM -0400, Emanuel dos Reis Rodrigues
>>>>> wrote:
>>>>>> I tried, follow my config:
>>>>>>
>>>>>> kafka_topic: netflow
>>>>>> kafka_broker_host: 192.168100.105
>>>>>> kafka_broker_port: 9092
>>>>>> kafka_refresh_time: 1
>>>>>> #daemonize: true
>>>>>> plugins: kafka
>>>>>> nfacctd_port: 9995
>>>>>> post_tag: 1
>>>>>> aggregate: tag, peer_src_ip, src_host, dst_host, timestamp_start,
>>>>>> timestamp_end, src_port, dst_port, proto
>>>>>>
>>>>>>
>>>>>> I kept the peer_src_ip, but the tag one is not being posted to Kafka.
>>>>>>
>>>>>> {'event_type': 'purge', 'peer_ip_src': '172.18.0.2', 'ip_src':
>>>>>> '192.168.1.100', 'ip_dst': 'x.46.x.245', 'port_src': 51184, 'port_dst':
>>>>>> 443, 'ip_proto': 'tcp', 'timestamp_start': '2020-04-14 14:15:39.000000',
>>>>>> 'timestamp_end': '2020-04-14 14:15:54.000000', 'packets': 5, 'bytes':
>>>>>> 260,
>>>>>> 'writer_id': 'default_kafka/75091'}
>>>>>>
>>>>>> Did I miss anything ?
>>>>>>
>>>>>>
>>>>>> Thanks !
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 14, 2020 at 10:26 AM Paolo Lucente <pa...@pmacct.net> wrote:
>>>>>>
>>>>>> >
>>>>>> > I may have skipped the important detail you need to add the 'tag' key
>>>>>> > to
>>>>>> > your 'aggregate' line in the config, my bad. This is in addition to,
>>>>>> > say,
>>>>>> > 'post_tag: 1' to identify collector 1. Let me know how it goes.
>>>>>> >
>>>>>> > Paolo
>>>>>> >
>>>>>> > On Tue, Apr 14, 2020 at 10:18:55AM -0400, Emanuel dos Reis Rodrigues
>>>>>> > wrote:
>>>>>> > > Thank you man, I did this test but I did not see the id being pushed
>>>>>> > along
>>>>>> > > with the Netflow info to Kafka topic. Is there the place the
>>>>>> > > information
>>>>>> > > would show up ?
>>>>>> > >
>>>>>> > >
>>>>>> > > On Tue, Apr 14, 2020 at 9:15 AM Paolo Lucente <pa...@pmacct.net>
>>>>>> > > wrote:
>>>>>> > >
>>>>>> > > >
>>>>>> > > > Hi Emanuel,
>>>>>> > > >
>>>>>> > > > Apologies i did not get you wanted and ID for the collector. The
>>>>>> > > > simplest way of achieving that is 'post_tag' as you just have to
>>>>>> > > > supply
>>>>>> > > > a number as ID; pre_tag_map expects a map and may be better to be
>>>>>> > > > reserved for more complex use-cases.
>>>>>> > > >
>>>>>> > > > Paolo
>>>>>> > > >
>>>>>> > > > On Mon, Apr 13, 2020 at 03:35:52PM -0400, Emanuel dos Reis
>>>>>> > > > Rodrigues
>>>>>> > wrote:
>>>>>> > > > > Thank you for your help. Appreciate it !
>>>>>> > > > >
>>>>>> > > > > See, I did use it for testing after I sent this email. However,
>>>>>> > > > > the
>>>>>> > ip
>>>>>> > > > > showed there was the IP from my nfacctd machine, the collector
>>>>>> > itself.
>>>>>> > > > Not
>>>>>> > > > > the exporter.
>>>>>> > > > >
>>>>>> > > > > peer_src_ip : IP address or identificator of
>>>>>> > > > telemetry
>>>>>> > > > > exporting device
>>>>>> > > > >
>>>>>> > > > > In fact, it may have todo with the fact I currently have an SSH
>>>>>> > tunnel
>>>>>> > > > with
>>>>>> > > > > socat with the remote machine in order to collect the data. This
>>>>>> > > > > may
>>>>>> > be
>>>>>> > > > the
>>>>>> > > > > reason why which is definitively not a ordinary condition. :)
>>>>>> > > > >
>>>>>> > > > > I am wondering if I could use this one to include a different
>>>>>> > > > > tag on
>>>>>> > it
>>>>>> > > > > process/collector, but have not yet figured out how. Any
>>>>>> > > > > thoughts ?
>>>>>> > > > >
>>>>>> > > > > label : String label, ie. as result of
>>>>>> > > > > pre_tag_map evaluation
>>>>>> > > > >
>>>>>> > > > >
>>>>>> > > > > Thank you again.
>>>>>> > > > >
>>>>>> > > > > On Mon, Apr 13, 2020 at 9:07 AM Paolo Lucente <pa...@pmacct.net>
>>>>>> > wrote:
>>>>>> > > > >
>>>>>> > > > > >
>>>>>> > > > > > Hi Emanuel,
>>>>>> > > > > >
>>>>>> > > > > > I think you are looking for (i admit, non-intuitive)
>>>>>> > > > > > 'peer_src_ip'
>>>>>> > > > > > primitive:
>>>>>> > > > > >
>>>>>> > > > > > $ nfacctd -a | grep peer_src_ip
>>>>>> > > > > > peer_src_ip : IP address or identificator
>>>>>> > > > > > of
>>>>>> > > > > > telemetry exporting device
>>>>>> > > > > >
>>>>>> > > > > > Without the grep you can see all supported primitives by the
>>>>>> > nfacctd
>>>>>> > > > > > release you are using along with a text explanation.
>>>>>> > > > > >
>>>>>> > > > > > Paolo
>>>>>> > > > > >
>>>>>> > > > > > On Sun, Apr 12, 2020 at 06:55:26PM -0400, Emanuel dos Reis
>>>>>> > Rodrigues
>>>>>> > > > wrote:
>>>>>> > > > > > > Hello guys,
>>>>>> > > > > > >
>>>>>> > > > > > > I implemented nfacctd acting as a Netflow collector using
>>>>>> > pmacct. It
>>>>>> > > > is
>>>>>> > > > > > > working perfectly and writing the flows to a Kafka topic
>>>>>> > > > > > > which I
>>>>>> > > > have an
>>>>>> > > > > > > application processing it.
>>>>>> > > > > > >
>>>>>> > > > > > > Following is my configuration:
>>>>>> > > > > > >
>>>>>> > > > > > > kafka_topic: netflow
>>>>>> > > > > > > kafka_broker_host: Kafka-host
>>>>>> > > > > > > kafka_broker_port: 9092
>>>>>> > > > > > > kafka_refresh_time: 1
>>>>>> > > > > > > daemonize: true
>>>>>> > > > > > > plugins: kafka
>>>>>> > > > > > > pcap_interface: enp0s8
>>>>>> > > > > > > nfacctd_ip: 192.168.1.100
>>>>>> > > > > > > nfacctd_port: 9995
>>>>>> > > > > > > aggregate: src_host, dst_host, timestamp_start,
>>>>>> > > > > > > timestamp_end,
>>>>>> > > > src_port,
>>>>>> > > > > > > dst_port, proto
>>>>>> > > > > > >
>>>>>> > > > > > > Currently, there is only one Netflow exporter sending data to
>>>>>> > this
>>>>>> > > > > > > demon and I would like to add another exporter. The problem
>>>>>> > > > > > > is
>>>>>> > that
>>>>>> > > > I am
>>>>>> > > > > > > not finding a way to differentiate the flows coming from
>>>>>> > different
>>>>>> > > > > > > exporters.
>>>>>> > > > > > >
>>>>>> > > > > > > Let's say I have the exporter A currently sending data to
>>>>>> > > > > > > nfacctd
>>>>>> > > > running
>>>>>> > > > > > > at port 9995 and the data is being written to Kafka topic
>>>>>> > Netflow.
>>>>>> > > > > > >
>>>>>> > > > > > > Now I want a new exporter B to start sending data to nfacctd
>>>>>> > > > > > > port
>>>>>> > > > 9996
>>>>>> > > > > > which
>>>>>> > > > > > > will be running as a separate demon ( just because I though
>>>>>> > > > > > > so,
>>>>>> > not
>>>>>> > > > sure
>>>>>> > > > > > > yet if it is a necessary approach) and writing the data to
>>>>>> > > > > > > the
>>>>>> > > > > > > same Netflow topic in Kafka.
>>>>>> > > > > > >
>>>>>> > > > > > > When the data comes from Kafka to my application, I cannot
>>>>>> > > > > > > tell
>>>>>> > from
>>>>>> > > > > > > which exporter the data came from. I would need some sort of
>>>>>> > > > > > identification
>>>>>> > > > > > > in order to make this differentiation. It is important for
>>>>>> > > > > > > me,
>>>>>> > > > because my
>>>>>> > > > > > > application may treat differently Netflow traffic coming from
>>>>>> > these
>>>>>> > > > > > > two Netflow exporters.
>>>>>> > > > > > >
>>>>>> > > > > > > Thanks in advance.
>>>>>> > > > > > >
>>>>>> > > > > > > Emanuel
>>>>>> > > > > >
>>>>>> > > > > > > _______________________________________________
>>>>>> > > > > > > pmacct-discussion mailing list
>>>>>> > > > > > > http://www.pmacct.net/#mailinglists
>>>>>> > > > > >
>>>>>> > > > > >
>>>>>> > > > > > _______________________________________________
>>>>>> > > > > > pmacct-discussion mailing list
>>>>>> > > > > > http://www.pmacct.net/#mailinglists
>>>>>> > > > > >
>>>>>> > > >
>>>>>> >
>>
>> _______________________________________________
>> pmacct-discussion mailing list
>> http://www.pmacct.net/#mailinglists_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
