You already seem to have a solution, but to me the writer_id is what you want.
Change the name of the process in your configuration file.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, April 15, 2020 7:33 PM, Emanuel dos Reis Rodrigues
<emanueldosr...@gmail.com> wrote:
> Hey, I just realize it worked. I think I was little behind on the messages
> parking on my kafka, now I can see the tag.
>
> Thank you so much for your help.
>
> On Wed, Apr 15, 2020 at 10:33 AM Emanuel dos Reis Rodrigues
> <emanueldosr...@gmail.com> wrote:
>
>> I am using:
>>
>> NetFlow Accounting Daemon, nfacctd 1.7.2-git (20181018-00+c3)
>>
>> Arguments:
>> '--enable-kafka' '--enable-jansson' 'JANSSON_CFLAGS=-I/usr/local/include/'
>> 'JANSSON_LIBS=-L/usr/local/lib -ljansson' '--enable-l2' '--enable-ipv6'
>> '--enable-64bit' '--enable-traffic-bins' '--enable-bgp-bins'
>> '--enable-bmp-bins' '--enable-st-bins'
>>
>> Libs:
>> libpcap version 1.5.3
>> rdkafka 0.11.4
>> jansson 2.12
>>
>> I can upgrade it to a newer version and try again.
>>
>> On Wed, Apr 15, 2020 at 8:59 AM Paolo Lucente <pa...@pmacct.net> wrote:
>>
>>> Hey Emanuel,
>>>
>>> The config is correct and I did try your same config and that does work
>>> for me, ie.:
>>>
>>> $ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic
>>> pmacct.flows
>>> {"event_type": "purge", "tag": 1, [ .. ]}
>>>
>>> What version of the software are you using? Is it 1.7.4p1 (latest
>>> stable) or master code from GitHub? If so, is it possible an old running
>>> nfacctd process is reading the data instead of the newly configured one?
>>>
>>> Paolo
>>>
>>> On Wed, Apr 15, 2020 at 12:17:43AM -0400, Emanuel dos Reis Rodrigues wrote:
>>>> I tried, follow my config:
>>>>
>>>> kafka_topic: netflow
>>>> kafka_broker_host: 192.168100.105
>>>> kafka_broker_port: 9092
>>>> kafka_refresh_time: 1
>>>> #daemonize: true
>>>> plugins: kafka
>>>> nfacctd_port: 9995
>>>> post_tag: 1
>>>> aggregate: tag, peer_src_ip, src_host, dst_host, timestamp_start,
>>>> timestamp_end, src_port, dst_port, proto
>>>>
>>>>
>>>> I kept the peer_src_ip, but the tag one is not being posted to Kafka.
>>>>
>>>> {'event_type': 'purge', 'peer_ip_src': '172.18.0.2', 'ip_src':
>>>> '192.168.1.100', 'ip_dst': 'x.46.x.245', 'port_src': 51184, 'port_dst':
>>>> 443, 'ip_proto': 'tcp', 'timestamp_start': '2020-04-14 14:15:39.000000',
>>>> 'timestamp_end': '2020-04-14 14:15:54.000000', 'packets': 5, 'bytes': 260,
>>>> 'writer_id': 'default_kafka/75091'}
>>>>
>>>> Did I miss anything ?
>>>>
>>>>
>>>> Thanks !
>>>>
>>>>
>>>>
>>>> On Tue, Apr 14, 2020 at 10:26 AM Paolo Lucente <pa...@pmacct.net> wrote:
>>>>
>>>> >
>>>> > I may have skipped the important detail you need to add the 'tag' key to
>>>> > your 'aggregate' line in the config, my bad. This is in addition to, say,
>>>> > 'post_tag: 1' to identify collector 1. Let me know how it goes.
>>>> >
>>>> > Paolo
>>>> >
>>>> > On Tue, Apr 14, 2020 at 10:18:55AM -0400, Emanuel dos Reis Rodrigues
>>>> > wrote:
>>>> > > Thank you man, I did this test but I did not see the id being pushed
>>>> > along
>>>> > > with the Netflow info to Kafka topic. Is there the place the
>>>> > > information
>>>> > > would show up ?
>>>> > >
>>>> > >
>>>> > > On Tue, Apr 14, 2020 at 9:15 AM Paolo Lucente <pa...@pmacct.net> wrote:
>>>> > >
>>>> > > >
>>>> > > > Hi Emanuel,
>>>> > > >
>>>> > > > Apologies i did not get you wanted and ID for the collector. The
>>>> > > > simplest way of achieving that is 'post_tag' as you just have to
>>>> > > > supply
>>>> > > > a number as ID; pre_tag_map expects a map and may be better to be
>>>> > > > reserved for more complex use-cases.
>>>> > > >
>>>> > > > Paolo
>>>> > > >
>>>> > > > On Mon, Apr 13, 2020 at 03:35:52PM -0400, Emanuel dos Reis Rodrigues
>>>> > wrote:
>>>> > > > > Thank you for your help. Appreciate it !
>>>> > > > >
>>>> > > > > See, I did use it for testing after I sent this email. However, the
>>>> > ip
>>>> > > > > showed there was the IP from my nfacctd machine, the collector
>>>> > itself.
>>>> > > > Not
>>>> > > > > the exporter.
>>>> > > > >
>>>> > > > > peer_src_ip : IP address or identificator of
>>>> > > > telemetry
>>>> > > > > exporting device
>>>> > > > >
>>>> > > > > In fact, it may have todo with the fact I currently have an SSH
>>>> > tunnel
>>>> > > > with
>>>> > > > > socat with the remote machine in order to collect the data. This
>>>> > > > > may
>>>> > be
>>>> > > > the
>>>> > > > > reason why which is definitively not a ordinary condition. :)
>>>> > > > >
>>>> > > > > I am wondering if I could use this one to include a different tag
>>>> > > > > on
>>>> > it
>>>> > > > > process/collector, but have not yet figured out how. Any thoughts ?
>>>> > > > >
>>>> > > > > label : String label, ie. as result of
>>>> > > > > pre_tag_map evaluation
>>>> > > > >
>>>> > > > >
>>>> > > > > Thank you again.
>>>> > > > >
>>>> > > > > On Mon, Apr 13, 2020 at 9:07 AM Paolo Lucente <pa...@pmacct.net>
>>>> > wrote:
>>>> > > > >
>>>> > > > > >
>>>> > > > > > Hi Emanuel,
>>>> > > > > >
>>>> > > > > > I think you are looking for (i admit, non-intuitive)
>>>> > > > > > 'peer_src_ip'
>>>> > > > > > primitive:
>>>> > > > > >
>>>> > > > > > $ nfacctd -a | grep peer_src_ip
>>>> > > > > > peer_src_ip : IP address or identificator of
>>>> > > > > > telemetry exporting device
>>>> > > > > >
>>>> > > > > > Without the grep you can see all supported primitives by the
>>>> > nfacctd
>>>> > > > > > release you are using along with a text explanation.
>>>> > > > > >
>>>> > > > > > Paolo
>>>> > > > > >
>>>> > > > > > On Sun, Apr 12, 2020 at 06:55:26PM -0400, Emanuel dos Reis
>>>> > Rodrigues
>>>> > > > wrote:
>>>> > > > > > > Hello guys,
>>>> > > > > > >
>>>> > > > > > > I implemented nfacctd acting as a Netflow collector using
>>>> > pmacct. It
>>>> > > > is
>>>> > > > > > > working perfectly and writing the flows to a Kafka topic which
>>>> > > > > > > I
>>>> > > > have an
>>>> > > > > > > application processing it.
>>>> > > > > > >
>>>> > > > > > > Following is my configuration:
>>>> > > > > > >
>>>> > > > > > > kafka_topic: netflow
>>>> > > > > > > kafka_broker_host: Kafka-host
>>>> > > > > > > kafka_broker_port: 9092
>>>> > > > > > > kafka_refresh_time: 1
>>>> > > > > > > daemonize: true
>>>> > > > > > > plugins: kafka
>>>> > > > > > > pcap_interface: enp0s8
>>>> > > > > > > nfacctd_ip: 192.168.1.100
>>>> > > > > > > nfacctd_port: 9995
>>>> > > > > > > aggregate: src_host, dst_host, timestamp_start, timestamp_end,
>>>> > > > src_port,
>>>> > > > > > > dst_port, proto
>>>> > > > > > >
>>>> > > > > > > Currently, there is only one Netflow exporter sending data to
>>>> > this
>>>> > > > > > > demon and I would like to add another exporter. The problem is
>>>> > that
>>>> > > > I am
>>>> > > > > > > not finding a way to differentiate the flows coming from
>>>> > different
>>>> > > > > > > exporters.
>>>> > > > > > >
>>>> > > > > > > Let's say I have the exporter A currently sending data to
>>>> > > > > > > nfacctd
>>>> > > > running
>>>> > > > > > > at port 9995 and the data is being written to Kafka topic
>>>> > Netflow.
>>>> > > > > > >
>>>> > > > > > > Now I want a new exporter B to start sending data to nfacctd
>>>> > > > > > > port
>>>> > > > 9996
>>>> > > > > > which
>>>> > > > > > > will be running as a separate demon ( just because I though so,
>>>> > not
>>>> > > > sure
>>>> > > > > > > yet if it is a necessary approach) and writing the data to the
>>>> > > > > > > same Netflow topic in Kafka.
>>>> > > > > > >
>>>> > > > > > > When the data comes from Kafka to my application, I cannot tell
>>>> > from
>>>> > > > > > > which exporter the data came from. I would need some sort of
>>>> > > > > > identification
>>>> > > > > > > in order to make this differentiation. It is important for me,
>>>> > > > because my
>>>> > > > > > > application may treat differently Netflow traffic coming from
>>>> > these
>>>> > > > > > > two Netflow exporters.
>>>> > > > > > >
>>>> > > > > > > Thanks in advance.
>>>> > > > > > >
>>>> > > > > > > Emanuel
>>>> > > > > >
>>>> > > > > > > _______________________________________________
>>>> > > > > > > pmacct-discussion mailing list
>>>> > > > > > > http://www.pmacct.net/#mailinglists
>>>> > > > > >
>>>> > > > > >
>>>> > > > > > _______________________________________________
>>>> > > > > > pmacct-discussion mailing list
>>>> > > > > > http://www.pmacct.net/#mailinglists
>>>> > > > > >
>>>> > > >
>>>> >_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
