Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-13 Thread Paolo Lucente


Hi Emanuel,

I think you are looking for (i admit, non-intuitive) 'peer_src_ip'
primitive:

$ nfacctd -a | grep peer_src_ip
peer_src_ip  : IP address or identificator of telemetry 
exporting device

Without the grep you can see all supported primitives by the nfacctd
release you are using along with a text explanation.

Paolo

On Sun, Apr 12, 2020 at 06:55:26PM -0400, Emanuel dos Reis Rodrigues wrote:
> Hello guys,
> 
> I implemented nfacctd acting as a Netflow collector using pmacct. It is
> working perfectly and writing the flows to a Kafka topic which I have an
> application processing it.
> 
> Following is my configuration:
> 
> kafka_topic: netflow
> kafka_broker_host: Kafka-host
> kafka_broker_port: 9092
> kafka_refresh_time: 1
> daemonize: true
> plugins: kafka
> pcap_interface: enp0s8
> nfacctd_ip: 192.168.1.100
> nfacctd_port: 9995
> aggregate: src_host, dst_host, timestamp_start, timestamp_end, src_port,
> dst_port, proto
> 
> Currently, there is only one Netflow exporter sending data to this
> demon and I would like to add another exporter. The problem is that I am
> not finding a way to differentiate the flows coming from different
> exporters.
> 
> Let's say I have the exporter A currently sending data to nfacctd running
> at port 9995 and the data is being written to Kafka topic Netflow.
> 
> Now I want a new exporter B to start sending data to nfacctd port 9996 which
> will be running as a separate demon ( just because I though so, not sure
> yet if it is a necessary approach)  and writing the data to the
> same Netflow topic in Kafka.
> 
> When the data comes from Kafka to my application, I cannot tell from
> which exporter the data came from. I would need some sort of identification
> in order to make this differentiation. It is important for me, because my
> application may treat differently Netflow traffic coming from these
> two Netflow exporters.
> 
> Thanks in advance.
> 
> Emanuel

> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists


Re: [pmacct-discussion] How to record ICMP and ICMP6 types/codes by pmacctd?

2020-04-13 Thread Paolo Lucente


Hi,

Let me confirm that collecting the ICMP type is partially supported; the
native dst_port primitive is locked to UDP and TCP only - making this
not suitable for NetFlow v5 kind of scenarios; but if using NetFlow v9
and/or IPFIX you could define your own custom primitive via the
aggregate_primitives infrastructure, see also an example here:  

https://github.com/pmacct/pmacct/blob/1.7.4/examples/primitives.lst.example

By the way: you speak collecting NetFlow but your config example is
actually about the 'nfprobe' plugin, that is, generating NetFlow out of
raw traffic. Is that what you are after?

Paolo 

On Sun, Apr 12, 2020 at 04:20:08PM -0400, fireballiso wrote:
> Hi! I've started using pmacctd to replace old netflow collectors for my
> main and test networks, which run both IPv6 and IPv4. It works very
> well, except that I haven't yet found a way to record the ICMP and ICMP6
> types and codes.
> 
> In other collectors, these are often stored in the destination port
> (otherwise unused for ICMP/ICMP6), in the format "A.B", where A is the
> type and B is the code. For example, "3.1" would represent ICMP type 3
> (Destination Unreachable), code 1 (Host Unreachable). I see lots of ICMP
> and ICMP6 flows, but unfortunately, the destination port is always set
> to "0.0", as if nothing is being recorded there.
> 
> A simple config:
> 
> daemonize: true
> !
> interface: net1
> aggregate: src_host, dst_host, src_port, dst_port, proto, tos
> plugins: nfprobe
> nfprobe_receiver: 192.168.14.2:9997
> nfprobe_version: 9
> 
> 
> I haven't found documentation or examples that show how to enable
> recording the types and codes, and no relevant primitives to add to the
> aggregate statement. Would someone be able to tell me how to do this?
> 
> Thank you!
> 
> -Indy
> 
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists


Re: [pmacct-discussion] How to record ICMP and ICMP6 types/codes by pmacctd?

2020-04-13 Thread fireballiso

  
  
Hi Paolo,



Sorry, I should have said I was
  replacing the netflow *generators*, not collectors. My mistake!



Yes, I posted the config that generates
  the netflow 9 flows, since I hoped to see if it was missing
  something for including the ICMP and ICMP6 types/codes.


-Indy




On 4/13/2020 8:59 AM, Paolo Lucente
  wrote:


  
Hi,

Let me confirm that collecting the ICMP type is partially supported; the
native dst_port primitive is locked to UDP and TCP only - making this
not suitable for NetFlow v5 kind of scenarios; but if using NetFlow v9
and/or IPFIX you could define your own custom primitive via the
aggregate_primitives infrastructure, see also an example here:  

https://github.com/pmacct/pmacct/blob/1.7.4/examples/primitives.lst.example

By the way: you speak collecting NetFlow but your config example is
actually about the 'nfprobe' plugin, that is, generating NetFlow out of
raw traffic. Is that what you are after?

Paolo 

On Sun, Apr 12, 2020 at 04:20:08PM -0400, fireballiso wrote:

  
Hi! I've started using pmacctd to replace old netflow collectors for my
main and test networks, which run both IPv6 and IPv4. It works very
well, except that I haven't yet found a way to record the ICMP and ICMP6
types and codes.

In other collectors, these are often stored in the destination port
(otherwise unused for ICMP/ICMP6), in the format "A.B", where A is the
type and B is the code. For example, "3.1" would represent ICMP type 3
(Destination Unreachable), code 1 (Host Unreachable). I see lots of ICMP
and ICMP6 flows, but unfortunately, the destination port is always set
to "0.0", as if nothing is being recorded there.

A simple config:

daemonize: true
!
interface: net1
aggregate: src_host, dst_host, src_port, dst_port, proto, tos
plugins: nfprobe
nfprobe_receiver: 192.168.14.2:9997
nfprobe_version: 9


I haven't found documentation or examples that show how to enable
recording the types and codes, and no relevant primitives to add to the
aggregate statement. Would someone be able to tell me how to do this?

Thank you!

-Indy

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

  
  
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists




-- 

-Indy
fireball...@yahoo.com
  


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Multiples nfacctd deamons writing to same Kafka topic

2020-04-13 Thread Emanuel dos Reis Rodrigues
Thank you for your help. Appreciate it !

See, I did use it for testing after I sent this email. However, the ip
showed there was the IP from my nfacctd machine, the collector itself. Not
the exporter.

peer_src_ip  : IP address or identificator of telemetry
exporting device

In fact, it may have todo with the fact I currently have an SSH tunnel with
socat with the remote machine in order to collect the data. This may be the
reason why which is definitively not a ordinary condition. :)

I am wondering if I could use this one to include a different tag on it
process/collector, but have not yet figured out how. Any thoughts ?

label: String label, ie. as result of
pre_tag_map evaluation


Thank you again.

On Mon, Apr 13, 2020 at 9:07 AM Paolo Lucente  wrote:

>
> Hi Emanuel,
>
> I think you are looking for (i admit, non-intuitive) 'peer_src_ip'
> primitive:
>
> $ nfacctd -a | grep peer_src_ip
> peer_src_ip  : IP address or identificator of
> telemetry exporting device
>
> Without the grep you can see all supported primitives by the nfacctd
> release you are using along with a text explanation.
>
> Paolo
>
> On Sun, Apr 12, 2020 at 06:55:26PM -0400, Emanuel dos Reis Rodrigues wrote:
> > Hello guys,
> >
> > I implemented nfacctd acting as a Netflow collector using pmacct. It is
> > working perfectly and writing the flows to a Kafka topic which I have an
> > application processing it.
> >
> > Following is my configuration:
> >
> > kafka_topic: netflow
> > kafka_broker_host: Kafka-host
> > kafka_broker_port: 9092
> > kafka_refresh_time: 1
> > daemonize: true
> > plugins: kafka
> > pcap_interface: enp0s8
> > nfacctd_ip: 192.168.1.100
> > nfacctd_port: 9995
> > aggregate: src_host, dst_host, timestamp_start, timestamp_end, src_port,
> > dst_port, proto
> >
> > Currently, there is only one Netflow exporter sending data to this
> > demon and I would like to add another exporter. The problem is that I am
> > not finding a way to differentiate the flows coming from different
> > exporters.
> >
> > Let's say I have the exporter A currently sending data to nfacctd running
> > at port 9995 and the data is being written to Kafka topic Netflow.
> >
> > Now I want a new exporter B to start sending data to nfacctd port 9996
> which
> > will be running as a separate demon ( just because I though so, not sure
> > yet if it is a necessary approach)  and writing the data to the
> > same Netflow topic in Kafka.
> >
> > When the data comes from Kafka to my application, I cannot tell from
> > which exporter the data came from. I would need some sort of
> identification
> > in order to make this differentiation. It is important for me, because my
> > application may treat differently Netflow traffic coming from these
> > two Netflow exporters.
> >
> > Thanks in advance.
> >
> > Emanuel
>
> > ___
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists