Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-14 Thread Emanuel dos Reis Rodrigues
I tried, follow my config:

kafka_topic: netflow
kafka_broker_host: 192.168100.105
kafka_broker_port: 9092
kafka_refresh_time: 1
#daemonize: true
plugins: kafka
nfacctd_port: 9995
post_tag: 1
aggregate: tag, peer_src_ip, src_host, dst_host, timestamp_start,
timestamp_end, src_port, dst_port, proto


I kept the peer_src_ip, but the tag one is not being posted to Kafka.

{'event_type': 'purge', 'peer_ip_src': '172.18.0.2', 'ip_src':
'192.168.1.100', 'ip_dst': 'x.46.x.245', 'port_src': 51184, 'port_dst':
443, 'ip_proto': 'tcp', 'timestamp_start': '2020-04-14 14:15:39.00',
'timestamp_end': '2020-04-14 14:15:54.00', 'packets': 5, 'bytes': 260,
'writer_id': 'default_kafka/75091'}

Did I miss anything ?


Thanks !



On Tue, Apr 14, 2020 at 10:26 AM Paolo Lucente  wrote:

>
> I may have skipped the important detail you need to add the 'tag' key to
> your 'aggregate' line in the config, my bad. This is in addition to, say,
> 'post_tag: 1' to identify collector 1. Let me know how it goes.
>
> Paolo
>
> On Tue, Apr 14, 2020 at 10:18:55AM -0400, Emanuel dos Reis Rodrigues wrote:
> > Thank you man, I did this test but I did not see the id being pushed
> along
> > with the Netflow info to Kafka topic. Is there the place the information
> > would show up ?
> >
> >
> > On Tue, Apr 14, 2020 at 9:15 AM Paolo Lucente  wrote:
> >
> > >
> > > Hi Emanuel,
> > >
> > > Apologies i did not get you wanted and ID for the collector. The
> > > simplest way of achieving that is 'post_tag' as you just have to supply
> > > a number as ID; pre_tag_map expects a map and may be better to be
> > > reserved for more complex use-cases.
> > >
> > > Paolo
> > >
> > > On Mon, Apr 13, 2020 at 03:35:52PM -0400, Emanuel dos Reis Rodrigues
> wrote:
> > > > Thank you for your help. Appreciate it !
> > > >
> > > > See, I did use it for testing after I sent this email. However, the
> ip
> > > > showed there was the IP from my nfacctd machine, the collector
> itself.
> > > Not
> > > > the exporter.
> > > >
> > > > peer_src_ip  : IP address or identificator of
> > > telemetry
> > > > exporting device
> > > >
> > > > In fact, it may have todo with the fact I currently have an SSH
> tunnel
> > > with
> > > > socat with the remote machine in order to collect the data. This may
> be
> > > the
> > > > reason why which is definitively not a ordinary condition. :)
> > > >
> > > > I am wondering if I could use this one to include a different tag on
> it
> > > > process/collector, but have not yet figured out how. Any thoughts ?
> > > >
> > > > label: String label, ie. as result of
> > > > pre_tag_map evaluation
> > > >
> > > >
> > > > Thank you again.
> > > >
> > > > On Mon, Apr 13, 2020 at 9:07 AM Paolo Lucente 
> wrote:
> > > >
> > > > >
> > > > > Hi Emanuel,
> > > > >
> > > > > I think you are looking for (i admit, non-intuitive) 'peer_src_ip'
> > > > > primitive:
> > > > >
> > > > > $ nfacctd -a | grep peer_src_ip
> > > > > peer_src_ip  : IP address or identificator of
> > > > > telemetry exporting device
> > > > >
> > > > > Without the grep you can see all supported primitives by the
> nfacctd
> > > > > release you are using along with a text explanation.
> > > > >
> > > > > Paolo
> > > > >
> > > > > On Sun, Apr 12, 2020 at 06:55:26PM -0400, Emanuel dos Reis
> Rodrigues
> > > wrote:
> > > > > > Hello guys,
> > > > > >
> > > > > > I implemented nfacctd acting as a Netflow collector using
> pmacct. It
> > > is
> > > > > > working perfectly and writing the flows to a Kafka topic which I
> > > have an
> > > > > > application processing it.
> > > > > >
> > > > > > Following is my configuration:
> > > > > >
> > > > > > kafka_topic: netflow
> > > > > > kafka_broker_host: Kafka-host
> > > > > > kafka_broker_port: 9092
> > > > > > kafka_refresh_time: 1
> > > > > > daemonize: true
> > > > > > plugins: kafka
> > > > > > pcap_interface: enp0s8
> > > > > > nfacctd_ip: 192.168.1.100
> > > > > > nfacctd_port: 9995
> > > > > > aggregate: src_host, dst_host, timestamp_start, timestamp_end,
> > > src_port,
> > > > > > dst_port, proto
> > > > > >
> > > > > > Currently, there is only one Netflow exporter sending data to
> this
> > > > > > demon and I would like to add another exporter. The problem is
> that
> > > I am
> > > > > > not finding a way to differentiate the flows coming from
> different
> > > > > > exporters.
> > > > > >
> > > > > > Let's say I have the exporter A currently sending data to nfacctd
> > > running
> > > > > > at port 9995 and the data is being written to Kafka topic
> Netflow.
> > > > > >
> > > > > > Now I want a new exporter B to start sending data to nfacctd port
> > > 9996
> > > > > which
> > > > > > will be running as a separate demon ( just because I though so,
> not
> > > sure
> > > > > > yet if it is a necessary approach)  and writing the data to the
> > > > > > same Netflow topic in Kafka.
> > > > > >
> > > > > > When the data comes from Kafka to my application, I 

Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-14 Thread Paolo Lucente


I may have skipped the important detail you need to add the 'tag' key to
your 'aggregate' line in the config, my bad. This is in addition to, say,
'post_tag: 1' to identify collector 1. Let me know how it goes.

Paolo

On Tue, Apr 14, 2020 at 10:18:55AM -0400, Emanuel dos Reis Rodrigues wrote:
> Thank you man, I did this test but I did not see the id being pushed along
> with the Netflow info to Kafka topic. Is there the place the information
> would show up ?
> 
> 
> On Tue, Apr 14, 2020 at 9:15 AM Paolo Lucente  wrote:
> 
> >
> > Hi Emanuel,
> >
> > Apologies i did not get you wanted and ID for the collector. The
> > simplest way of achieving that is 'post_tag' as you just have to supply
> > a number as ID; pre_tag_map expects a map and may be better to be
> > reserved for more complex use-cases.
> >
> > Paolo
> >
> > On Mon, Apr 13, 2020 at 03:35:52PM -0400, Emanuel dos Reis Rodrigues wrote:
> > > Thank you for your help. Appreciate it !
> > >
> > > See, I did use it for testing after I sent this email. However, the ip
> > > showed there was the IP from my nfacctd machine, the collector itself.
> > Not
> > > the exporter.
> > >
> > > peer_src_ip  : IP address or identificator of
> > telemetry
> > > exporting device
> > >
> > > In fact, it may have todo with the fact I currently have an SSH tunnel
> > with
> > > socat with the remote machine in order to collect the data. This may be
> > the
> > > reason why which is definitively not a ordinary condition. :)
> > >
> > > I am wondering if I could use this one to include a different tag on it
> > > process/collector, but have not yet figured out how. Any thoughts ?
> > >
> > > label: String label, ie. as result of
> > > pre_tag_map evaluation
> > >
> > >
> > > Thank you again.
> > >
> > > On Mon, Apr 13, 2020 at 9:07 AM Paolo Lucente  wrote:
> > >
> > > >
> > > > Hi Emanuel,
> > > >
> > > > I think you are looking for (i admit, non-intuitive) 'peer_src_ip'
> > > > primitive:
> > > >
> > > > $ nfacctd -a | grep peer_src_ip
> > > > peer_src_ip  : IP address or identificator of
> > > > telemetry exporting device
> > > >
> > > > Without the grep you can see all supported primitives by the nfacctd
> > > > release you are using along with a text explanation.
> > > >
> > > > Paolo
> > > >
> > > > On Sun, Apr 12, 2020 at 06:55:26PM -0400, Emanuel dos Reis Rodrigues
> > wrote:
> > > > > Hello guys,
> > > > >
> > > > > I implemented nfacctd acting as a Netflow collector using pmacct. It
> > is
> > > > > working perfectly and writing the flows to a Kafka topic which I
> > have an
> > > > > application processing it.
> > > > >
> > > > > Following is my configuration:
> > > > >
> > > > > kafka_topic: netflow
> > > > > kafka_broker_host: Kafka-host
> > > > > kafka_broker_port: 9092
> > > > > kafka_refresh_time: 1
> > > > > daemonize: true
> > > > > plugins: kafka
> > > > > pcap_interface: enp0s8
> > > > > nfacctd_ip: 192.168.1.100
> > > > > nfacctd_port: 9995
> > > > > aggregate: src_host, dst_host, timestamp_start, timestamp_end,
> > src_port,
> > > > > dst_port, proto
> > > > >
> > > > > Currently, there is only one Netflow exporter sending data to this
> > > > > demon and I would like to add another exporter. The problem is that
> > I am
> > > > > not finding a way to differentiate the flows coming from different
> > > > > exporters.
> > > > >
> > > > > Let's say I have the exporter A currently sending data to nfacctd
> > running
> > > > > at port 9995 and the data is being written to Kafka topic Netflow.
> > > > >
> > > > > Now I want a new exporter B to start sending data to nfacctd port
> > 9996
> > > > which
> > > > > will be running as a separate demon ( just because I though so, not
> > sure
> > > > > yet if it is a necessary approach)  and writing the data to the
> > > > > same Netflow topic in Kafka.
> > > > >
> > > > > When the data comes from Kafka to my application, I cannot tell from
> > > > > which exporter the data came from. I would need some sort of
> > > > identification
> > > > > in order to make this differentiation. It is important for me,
> > because my
> > > > > application may treat differently Netflow traffic coming from these
> > > > > two Netflow exporters.
> > > > >
> > > > > Thanks in advance.
> > > > >
> > > > > Emanuel
> > > >
> > > > > ___
> > > > > pmacct-discussion mailing list
> > > > > http://www.pmacct.net/#mailinglists
> > > >
> > > >
> > > > ___
> > > > pmacct-discussion mailing list
> > > > http://www.pmacct.net/#mailinglists
> > > >
> >

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists


Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-14 Thread Emanuel dos Reis Rodrigues
Thank you man, I did this test but I did not see the id being pushed along
with the Netflow info to Kafka topic. Is there the place the information
would show up ?


On Tue, Apr 14, 2020 at 9:15 AM Paolo Lucente  wrote:

>
> Hi Emanuel,
>
> Apologies i did not get you wanted and ID for the collector. The
> simplest way of achieving that is 'post_tag' as you just have to supply
> a number as ID; pre_tag_map expects a map and may be better to be
> reserved for more complex use-cases.
>
> Paolo
>
> On Mon, Apr 13, 2020 at 03:35:52PM -0400, Emanuel dos Reis Rodrigues wrote:
> > Thank you for your help. Appreciate it !
> >
> > See, I did use it for testing after I sent this email. However, the ip
> > showed there was the IP from my nfacctd machine, the collector itself.
> Not
> > the exporter.
> >
> > peer_src_ip  : IP address or identificator of
> telemetry
> > exporting device
> >
> > In fact, it may have todo with the fact I currently have an SSH tunnel
> with
> > socat with the remote machine in order to collect the data. This may be
> the
> > reason why which is definitively not a ordinary condition. :)
> >
> > I am wondering if I could use this one to include a different tag on it
> > process/collector, but have not yet figured out how. Any thoughts ?
> >
> > label: String label, ie. as result of
> > pre_tag_map evaluation
> >
> >
> > Thank you again.
> >
> > On Mon, Apr 13, 2020 at 9:07 AM Paolo Lucente  wrote:
> >
> > >
> > > Hi Emanuel,
> > >
> > > I think you are looking for (i admit, non-intuitive) 'peer_src_ip'
> > > primitive:
> > >
> > > $ nfacctd -a | grep peer_src_ip
> > > peer_src_ip  : IP address or identificator of
> > > telemetry exporting device
> > >
> > > Without the grep you can see all supported primitives by the nfacctd
> > > release you are using along with a text explanation.
> > >
> > > Paolo
> > >
> > > On Sun, Apr 12, 2020 at 06:55:26PM -0400, Emanuel dos Reis Rodrigues
> wrote:
> > > > Hello guys,
> > > >
> > > > I implemented nfacctd acting as a Netflow collector using pmacct. It
> is
> > > > working perfectly and writing the flows to a Kafka topic which I
> have an
> > > > application processing it.
> > > >
> > > > Following is my configuration:
> > > >
> > > > kafka_topic: netflow
> > > > kafka_broker_host: Kafka-host
> > > > kafka_broker_port: 9092
> > > > kafka_refresh_time: 1
> > > > daemonize: true
> > > > plugins: kafka
> > > > pcap_interface: enp0s8
> > > > nfacctd_ip: 192.168.1.100
> > > > nfacctd_port: 9995
> > > > aggregate: src_host, dst_host, timestamp_start, timestamp_end,
> src_port,
> > > > dst_port, proto
> > > >
> > > > Currently, there is only one Netflow exporter sending data to this
> > > > demon and I would like to add another exporter. The problem is that
> I am
> > > > not finding a way to differentiate the flows coming from different
> > > > exporters.
> > > >
> > > > Let's say I have the exporter A currently sending data to nfacctd
> running
> > > > at port 9995 and the data is being written to Kafka topic Netflow.
> > > >
> > > > Now I want a new exporter B to start sending data to nfacctd port
> 9996
> > > which
> > > > will be running as a separate demon ( just because I though so, not
> sure
> > > > yet if it is a necessary approach)  and writing the data to the
> > > > same Netflow topic in Kafka.
> > > >
> > > > When the data comes from Kafka to my application, I cannot tell from
> > > > which exporter the data came from. I would need some sort of
> > > identification
> > > > in order to make this differentiation. It is important for me,
> because my
> > > > application may treat differently Netflow traffic coming from these
> > > > two Netflow exporters.
> > > >
> > > > Thanks in advance.
> > > >
> > > > Emanuel
> > >
> > > > ___
> > > > pmacct-discussion mailing list
> > > > http://www.pmacct.net/#mailinglists
> > >
> > >
> > > ___
> > > pmacct-discussion mailing list
> > > http://www.pmacct.net/#mailinglists
> > >
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] How to record ICMP and ICMP6 types/codes by pmacctd?

2020-04-14 Thread Paolo Lucente


Hi,

I see. I am sorry to confirm that, yes, the feature is not there right
now. It's not a biggie but still it would require a bit of work in order
to converge. I can gladly put on my todo list but it may take a few
weeks to get it out; or if you could perform a small fun C coding work
on you own, please get in touch by unicast email, i'd be happy to
assist.

Paolo
 
On Mon, Apr 13, 2020 at 03:59:48PM -0400, fireballiso wrote:
> 
>   
> 
>   
>   
> Hi Paolo,
> 
> 
> 
> Sorry, I should have said I was
>   replacing the netflow *generators*, not collectors. My mistake!
> 
> 
> 
> Yes, I posted the config that generates
>   the netflow 9 flows, since I hoped to see if it was missing
>   something for including the ICMP and ICMP6 types/codes.
> 
> 
> -Indy
> 
> 
> 
> 
> On 4/13/2020 8:59 AM, Paolo Lucente
>   wrote:
> 
>cite="mid:20200413125955.gb16...@moussaka.pmacct.net">
>   
> Hi,
> 
> Let me confirm that collecting the ICMP type is partially supported; the
> native dst_port primitive is locked to UDP and TCP only - making this
> not suitable for NetFlow v5 kind of scenarios; but if using NetFlow v9
> and/or IPFIX you could define your own custom primitive via the
> aggregate_primitives infrastructure, see also an example here:  
> 
>  href="https://github.com/pmacct/pmacct/blob/1.7.4/examples/primitives.lst.example;>https://github.com/pmacct/pmacct/blob/1.7.4/examples/primitives.lst.example
> 
> By the way: you speak collecting NetFlow but your config example is
> actually about the 'nfprobe' plugin, that is, generating NetFlow out of
> raw traffic. Is that what you are after?
> 
> Paolo 
> 
> On Sun, Apr 12, 2020 at 04:20:08PM -0400, fireballiso wrote:
> 
>   
> Hi! I've started using pmacctd to 
> replace old netflow collectors for my
> main and test networks, which run both IPv6 and IPv4. It works very
> well, except that I haven't yet found a way to record the ICMP and ICMP6
> types and codes.
> 
> In other collectors, these are often stored in the destination port
> (otherwise unused for ICMP/ICMP6), in the format "A.B", where A is the
> type and B is the code. For example, "3.1" would represent ICMP type 3
> (Destination Unreachable), code 1 (Host Unreachable). I see lots of ICMP
> and ICMP6 flows, but unfortunately, the destination port is always set
> to "0.0", as if nothing is being recorded there.
> 
> A simple config:
> 
> daemonize: true
> !
> interface: net1
> aggregate: src_host, dst_host, src_port, dst_port, proto, tos
> plugins: nfprobe
> nfprobe_receiver: 192.168.14.2:9997
> nfprobe_version: 9
> 
> 
> I haven't found documentation or examples that show how to enable
> recording the types and codes, and no relevant primitives to add to the
> aggregate statement. Would someone be able to tell me how to do this?
> 
> Thank you!
> 
> -Indy
> 
> ___
> pmacct-discussion mailing list
>  href="http://www.pmacct.net/#mailinglists;>http://www.pmacct.net/#mailinglists
> 
>   
>   
> ___
> pmacct-discussion mailing list
>  href="http://www.pmacct.net/#mailinglists;>http://www.pmacct.net/#mailinglists
> 
> 
> 
> 
> -- 
> 
> -Indy
>  href="mailto:fireball...@yahoo.com;>fireball...@yahoo.com
>   
> 

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists


Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-14 Thread Paolo Lucente


Hi Emanuel,

Apologies i did not get you wanted and ID for the collector. The
simplest way of achieving that is 'post_tag' as you just have to supply
a number as ID; pre_tag_map expects a map and may be better to be
reserved for more complex use-cases.

Paolo

On Mon, Apr 13, 2020 at 03:35:52PM -0400, Emanuel dos Reis Rodrigues wrote:
> Thank you for your help. Appreciate it !
> 
> See, I did use it for testing after I sent this email. However, the ip
> showed there was the IP from my nfacctd machine, the collector itself. Not
> the exporter.
> 
> peer_src_ip  : IP address or identificator of telemetry
> exporting device
> 
> In fact, it may have todo with the fact I currently have an SSH tunnel with
> socat with the remote machine in order to collect the data. This may be the
> reason why which is definitively not a ordinary condition. :)
> 
> I am wondering if I could use this one to include a different tag on it
> process/collector, but have not yet figured out how. Any thoughts ?
> 
> label: String label, ie. as result of
> pre_tag_map evaluation
> 
> 
> Thank you again.
> 
> On Mon, Apr 13, 2020 at 9:07 AM Paolo Lucente  wrote:
> 
> >
> > Hi Emanuel,
> >
> > I think you are looking for (i admit, non-intuitive) 'peer_src_ip'
> > primitive:
> >
> > $ nfacctd -a | grep peer_src_ip
> > peer_src_ip  : IP address or identificator of
> > telemetry exporting device
> >
> > Without the grep you can see all supported primitives by the nfacctd
> > release you are using along with a text explanation.
> >
> > Paolo
> >
> > On Sun, Apr 12, 2020 at 06:55:26PM -0400, Emanuel dos Reis Rodrigues wrote:
> > > Hello guys,
> > >
> > > I implemented nfacctd acting as a Netflow collector using pmacct. It is
> > > working perfectly and writing the flows to a Kafka topic which I have an
> > > application processing it.
> > >
> > > Following is my configuration:
> > >
> > > kafka_topic: netflow
> > > kafka_broker_host: Kafka-host
> > > kafka_broker_port: 9092
> > > kafka_refresh_time: 1
> > > daemonize: true
> > > plugins: kafka
> > > pcap_interface: enp0s8
> > > nfacctd_ip: 192.168.1.100
> > > nfacctd_port: 9995
> > > aggregate: src_host, dst_host, timestamp_start, timestamp_end, src_port,
> > > dst_port, proto
> > >
> > > Currently, there is only one Netflow exporter sending data to this
> > > demon and I would like to add another exporter. The problem is that I am
> > > not finding a way to differentiate the flows coming from different
> > > exporters.
> > >
> > > Let's say I have the exporter A currently sending data to nfacctd running
> > > at port 9995 and the data is being written to Kafka topic Netflow.
> > >
> > > Now I want a new exporter B to start sending data to nfacctd port 9996
> > which
> > > will be running as a separate demon ( just because I though so, not sure
> > > yet if it is a necessary approach)  and writing the data to the
> > > same Netflow topic in Kafka.
> > >
> > > When the data comes from Kafka to my application, I cannot tell from
> > > which exporter the data came from. I would need some sort of
> > identification
> > > in order to make this differentiation. It is important for me, because my
> > > application may treat differently Netflow traffic coming from these
> > > two Netflow exporters.
> > >
> > > Thanks in advance.
> > >
> > > Emanuel
> >
> > > ___
> > > pmacct-discussion mailing list
> > > http://www.pmacct.net/#mailinglists
> >
> >
> > ___
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
> >

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists