Re: [pmacct-discussion] Custom primitives with netflow

[Paolo Lucente]

Hi Rajesh,

Nice labels worked for you.

Clarify me one thing: the output you did show, with zeroed peer_src_ip
(and exporteripv4address, engineid, enginetype), is the one from
pmacctd, right? Not nfacctd. In that case the output is expected. 
In fact in nfacctd it should not be possible to get a null peer_src_ip
(which is nothing else than the address returned by a recv() on the
socket); setting nfprobe_source_ip is needed only in cases in which
multiple interfaces could be selected for output or for settin field
type #130 for, for example, NAT traversal scenarios.

Paolo

On Sat, Jan 05, 2019 at 12:29:35AM +0530, RAJESH KUMAR S.R wrote:
> Hi Paolo,
> 
> I was able to set labels and export as strings for different pmacct
> instances that was listening on different interfaces. Thanks for the
> suggestion.
> 
> I need a help regarding exporting Exporter's IP as part of flow records.
> Based on old pmacct discussions, I'm using "peer_src_ip and
> exporterIPv4Address" as primitives but both they seem to come as 0 and
> 0.0.0.0 in flows.
> I tried setting the "nfprobe_source_ip: 172.30.130.99", but it goes in
> separate flow as ExporterAddress: 172.30.130.99, but I need the
> "exporterIpv4Address" to be set to correct value in all flow messages, will
> pmacct collect the interface ip and populate in "exporterIpv4Address"
> field. Also, I'm not sure of how to get the engineid working, that also
> seem to go as 0 in flows.
> 
> pmacctd Output:
> 
> *SRC_MAC   DST_MAC PEER_SRC_IP*
> 50:01:d9:a3:41:f1  f8:59:71:73:94:4d  ::
> * SRC_IP   DST_IPSRC_PORT  DST_PORT  PROTOCOL   *
> 52.229.174.94  192.168.1.9 443  43238  tcp
> 
> *exporteripv4addressengineid  enginetype  PACKETS BYTES*
>  0.0.0.0 0 0
> 2 629
> 
> 
> I have the following pmacct conf file
> 
> *primitives*
> 
> name=engineType field_type=0:38 len=1 semantics=u_int
> name=engineId field_type=0:39 len=1 semantics=u_int
> name=exporterIPv4Address field_type=130 len=4 semantics=ip
> 
> *pmacct.conf*
> "
> debug: true
>daemonize: false
>pre_tag_map: ipfix_pretag.map
> 
>nfprobe_engine: 100
>nfprobe_version: 10
>nfprobe_source_ip: 172.30.130.99
>aggregate_primitives: ipfix_primitives.lst
>plugins: nfprobe, print
>interface: wlp1s0
>aggregate: src_mac, dst_mac, src_host, dst_host, src_port, dst_port,
> proto, peer_src_ip, exporterIPv4Address, engineId, engineType
>nfprobe_receiver: 10.40.6.6:17058
> "
> 
> 
> 
> 
> On Wed, Dec 26, 2018 at 12:44 PM RAJESH KUMAR S.R 
> wrote:
> 
> > Hi Paolo,
> >
> > Thanks for the fix and suggestion. I'll try tag and label primitives and
> > see if they match my requirements.
> >
> > On Tue, Dec 25, 2018 at 10:49 PM Paolo Lucente  wrote:
> >
> >>
> >> Hi Rajesh,
> >>
> >> You are right, there was a bug in the serialize_bin() func that was
> >> making it work good only for the first byte. This is now resolved:
> >>
> >>
> >> https://github.com/pmacct/pmacct/commit/1076ff3529f439133357176e4c1260cfcdcef56e
> >>
> >> I've read your question about metadata and was wondering: would tags
> >> (tag, tag2 primitive) or labels (label primitive) defined via a
> >> pre_tag_map be a solution for you? You could do a proof-of-concept
> >> locally, ie. like you were doing already with the print plugin, and if
> >> meeting your requirements we can move onto the nfprobe part; i expect
> >> tags to work no problem; labels should work but may require a bit more
> >> testing.
> >>
> >> Paolo
> >>
> >> On Mon, Dec 24, 2018 at 02:21:18PM +0530, RAJESH KUMAR S.R wrote:
> >> > Hi Paolo,
> >> >
> >> > Thanks for the fix. I tested with pmacctd and nfacctd and I see that
> >> when I
> >> > read 1 byte of raw data, it prints correct on both sides
> >> > but by while reading more bytes, the first byte is alone correct on
> >> nfacct
> >> > side. Not sure I'm testing correctly, but thanks for the fix.
> >> >
> >> > pmacctd side
> >> > dummy_byte  PACKETS   BYTES
> >> > *08-00-45*535   124114
> >> > *86-DD-60*10861
> >> >
> >> > On nfacctd side, I'm getting only the first byte correct
> >> > *08-00-00*535 124114
> >> > *86-00-00*10   861
> >> >
> >> >
> >> > I need one more suggestion/help regarding custom primitives.
> >> > Right now I can read a set of bytes from packet and send it as custom
> >> > primitives using the
> >> > recommended PEN id which I can see in netflow message in wireshark
> >> > also(attached the wireshark image)
> >> >
> >> > I have requirement where I need to send some *metadata*/context as part
> >> as
> >> > netflow records.
> >> > The metadata/context won't be available in the packet. Will I be able to
> >> > define and use
> >> > custom primitives for sending those metadata, (say one or more
> >> > context/metadata in pmacct conf file, similar to ifindex)
> >> > I'

Re: [pmacct-discussion] Custom primitives with netflow

[RAJESH KUMAR S.R]

Hi Paolo,

Thanks for the fix and suggestion. I'll try tag and label primitives and
see if they match my requirements.

On Tue, Dec 25, 2018 at 10:49 PM Paolo Lucente  wrote:

>
> Hi Rajesh,
>
> You are right, there was a bug in the serialize_bin() func that was
> making it work good only for the first byte. This is now resolved:
>
>
> https://github.com/pmacct/pmacct/commit/1076ff3529f439133357176e4c1260cfcdcef56e
>
> I've read your question about metadata and was wondering: would tags
> (tag, tag2 primitive) or labels (label primitive) defined via a
> pre_tag_map be a solution for you? You could do a proof-of-concept
> locally, ie. like you were doing already with the print plugin, and if
> meeting your requirements we can move onto the nfprobe part; i expect
> tags to work no problem; labels should work but may require a bit more
> testing.
>
> Paolo
>
> On Mon, Dec 24, 2018 at 02:21:18PM +0530, RAJESH KUMAR S.R wrote:
> > Hi Paolo,
> >
> > Thanks for the fix. I tested with pmacctd and nfacctd and I see that
> when I
> > read 1 byte of raw data, it prints correct on both sides
> > but by while reading more bytes, the first byte is alone correct on
> nfacct
> > side. Not sure I'm testing correctly, but thanks for the fix.
> >
> > pmacctd side
> > dummy_byte  PACKETS   BYTES
> > *08-00-45*535   124114
> > *86-DD-60*10861
> >
> > On nfacctd side, I'm getting only the first byte correct
> > *08-00-00*535 124114
> > *86-00-00*10   861
> >
> >
> > I need one more suggestion/help regarding custom primitives.
> > Right now I can read a set of bytes from packet and send it as custom
> > primitives using the
> > recommended PEN id which I can see in netflow message in wireshark
> > also(attached the wireshark image)
> >
> > I have requirement where I need to send some *metadata*/context as part
> as
> > netflow records.
> > The metadata/context won't be available in the packet. Will I be able to
> > define and use
> > custom primitives for sending those metadata, (say one or more
> > context/metadata in pmacct conf file, similar to ifindex)
> > I'm reading packet from different interfaces, where I know the context of
> > packets that are arriving on each interface
> > I need to send that context as one of the netflow field. The only way I
> > know to do now is to write that context as part of the packet and read as
> > custom template.
> > So, basically I'm trying to figure out other possible ways of defining
> > custom primitives
> > & fieldtypes and using it without reading from packet.
> > It would be very helpful it any alternate way is feasible.
> >
> > On Thu, Dec 20, 2018 at 6:23 AM Paolo Lucente  wrote:
> >
> > >
> > > Hi Rajesh,
> > >
> > > Thanks for bringing this further point up. While all was working with
> > > hex and u_int semantics, there was an issue with the raw one - which
> was
> > > important to fix since it's a very generic transport for any kind of
> > > info. Here is the commit:
> > >
> > >
> > >
> https://github.com/pmacct/pmacct/commit/e1206a530af64c46748b5f891a9034dec89e9b7a
> > >
> > > Let me know if this works for you.
> > >
> > > Paolo
> > >
> > > On Mon, Dec 17, 2018 at 10:25:27PM +0530, RAJESH KUMAR S.R wrote:
> > > > Hi Paolo,
> > > >
> > > > Thanks for the help and suggestion ,
> > > > Custom primitives works for me with the latest code:) . I'm able to
> read
> > > a
> > > > field from packet and export as custom template and I can see those
> > > > templates in netflow messages also.
> > > > I'll continue with viewing the flows in a collector, I'm using
> Elastiflow
> > > > to view reports.
> > > >
> > > >
> > > > I have a small doubt. I was reading a field of each packet and map
> it to
> > > a
> > > > custom template.(just trying to read ethertype for testing the custom
> > > > templates eventhough etype is available)
> > > > The value shown in pmacct seems to be right.  I tried verifying this
> data
> > > > with nfacctd, but the value shown in nfacctd seems to be different. I
> > > have
> > > > pasted the conf files and output.
> > > >
> > > > *Conf files:*
> > > > *primitives.lst file*
> > > > *name=dummy_byte packet_ptr=packet:+12 len=2 semantics=raw
> > > > field_type=41234:100*
> > > >
> > > > *pmacct.conf*
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *daemonize: false   interface: wlp1s0   !pcap_interfaces_map:
> > > > pcap_interfaces.map   aggregate_primitives: primitives.lst
>  aggregate:
> > > > src_host, dst_host, src_port, dst_port, proto, tos, dummy_byte
>  plugins:
> > > > nfprobe, print   nfprobe_version: 10   nfprobe_engine: 100
> > > > nfprobe_receiver: 192.168.1.6:2100 
> > > > !nfprobe_receiver: 10.40.6.6:16367 *
> > > >
> > > >
> > > > *nfacct.conf*
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *daemonize: false   nfacctd_ip: 192.168.1.6   nfacctd_port: 21

Re: [pmacct-discussion] Custom primitives with netflow

[Paolo Lucente]

Hi Rajesh,

You are right, there was a bug in the serialize_bin() func that was
making it work good only for the first byte. This is now resolved:

https://github.com/pmacct/pmacct/commit/1076ff3529f439133357176e4c1260cfcdcef56e

I've read your question about metadata and was wondering: would tags
(tag, tag2 primitive) or labels (label primitive) defined via a
pre_tag_map be a solution for you? You could do a proof-of-concept
locally, ie. like you were doing already with the print plugin, and if
meeting your requirements we can move onto the nfprobe part; i expect
tags to work no problem; labels should work but may require a bit more
testing. 

Paolo

On Mon, Dec 24, 2018 at 02:21:18PM +0530, RAJESH KUMAR S.R wrote:
> Hi Paolo,
> 
> Thanks for the fix. I tested with pmacctd and nfacctd and I see that when I
> read 1 byte of raw data, it prints correct on both sides
> but by while reading more bytes, the first byte is alone correct on nfacct
> side. Not sure I'm testing correctly, but thanks for the fix.
> 
> pmacctd side
> dummy_byte  PACKETS   BYTES
> *08-00-45*535   124114
> *86-DD-60*10861
> 
> On nfacctd side, I'm getting only the first byte correct
> *08-00-00*535 124114
> *86-00-00*10   861
> 
> 
> I need one more suggestion/help regarding custom primitives.
> Right now I can read a set of bytes from packet and send it as custom
> primitives using the
> recommended PEN id which I can see in netflow message in wireshark
> also(attached the wireshark image)
> 
> I have requirement where I need to send some *metadata*/context as part as
> netflow records.
> The metadata/context won't be available in the packet. Will I be able to
> define and use
> custom primitives for sending those metadata, (say one or more
> context/metadata in pmacct conf file, similar to ifindex)
> I'm reading packet from different interfaces, where I know the context of
> packets that are arriving on each interface
> I need to send that context as one of the netflow field. The only way I
> know to do now is to write that context as part of the packet and read as
> custom template.
> So, basically I'm trying to figure out other possible ways of defining
> custom primitives
> & fieldtypes and using it without reading from packet.
> It would be very helpful it any alternate way is feasible.
> 
> On Thu, Dec 20, 2018 at 6:23 AM Paolo Lucente  wrote:
> 
> >
> > Hi Rajesh,
> >
> > Thanks for bringing this further point up. While all was working with
> > hex and u_int semantics, there was an issue with the raw one - which was
> > important to fix since it's a very generic transport for any kind of
> > info. Here is the commit:
> >
> >
> > https://github.com/pmacct/pmacct/commit/e1206a530af64c46748b5f891a9034dec89e9b7a
> >
> > Let me know if this works for you.
> >
> > Paolo
> >
> > On Mon, Dec 17, 2018 at 10:25:27PM +0530, RAJESH KUMAR S.R wrote:
> > > Hi Paolo,
> > >
> > > Thanks for the help and suggestion ,
> > > Custom primitives works for me with the latest code:) . I'm able to read
> > a
> > > field from packet and export as custom template and I can see those
> > > templates in netflow messages also.
> > > I'll continue with viewing the flows in a collector, I'm using Elastiflow
> > > to view reports.
> > >
> > >
> > > I have a small doubt. I was reading a field of each packet and map it to
> > a
> > > custom template.(just trying to read ethertype for testing the custom
> > > templates eventhough etype is available)
> > > The value shown in pmacct seems to be right.  I tried verifying this data
> > > with nfacctd, but the value shown in nfacctd seems to be different. I
> > have
> > > pasted the conf files and output.
> > >
> > > *Conf files:*
> > > *primitives.lst file*
> > > *name=dummy_byte packet_ptr=packet:+12 len=2 semantics=raw
> > > field_type=41234:100*
> > >
> > > *pmacct.conf*
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *daemonize: false   interface: wlp1s0   !pcap_interfaces_map:
> > > pcap_interfaces.map   aggregate_primitives: primitives.lst   aggregate:
> > > src_host, dst_host, src_port, dst_port, proto, tos, dummy_byte   plugins:
> > > nfprobe, print   nfprobe_version: 10   nfprobe_engine: 100
> > > nfprobe_receiver: 192.168.1.6:2100 
> > > !nfprobe_receiver: 10.40.6.6:16367 *
> > >
> > >
> > > *nfacct.conf*
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *daemonize: false   nfacctd_ip: 192.168.1.6   nfacctd_port: 2100
> > > aggregate_primitives: primitives.lst   aggregate: src_host, dst_host,
> > > src_port, dst_port, proto, tos, dummy_byte   plugins: print
> > > nfacctd_disable_checks: true*
> > >
> > > $   sudo /usr/local/sbin/pmacctd -f pmacct.conf
> > > INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd
> > > 1.7.3-git (20181217-00)
> > > INFO ( default/core ):  '--enable-l2' '--enable-64bit'
> > > '--enable-traffic-bins' '--en

Re: [pmacct-discussion] Custom primitives with netflow

[Paolo Lucente]

Hi Rajesh,

Thanks for bringing this further point up. While all was working with
hex and u_int semantics, there was an issue with the raw one - which was
important to fix since it's a very generic transport for any kind of
info. Here is the commit:

https://github.com/pmacct/pmacct/commit/e1206a530af64c46748b5f891a9034dec89e9b7a

Let me know if this works for you.

Paolo

On Mon, Dec 17, 2018 at 10:25:27PM +0530, RAJESH KUMAR S.R wrote:
> Hi Paolo,
> 
> Thanks for the help and suggestion ,
> Custom primitives works for me with the latest code:) . I'm able to read a
> field from packet and export as custom template and I can see those
> templates in netflow messages also.
> I'll continue with viewing the flows in a collector, I'm using Elastiflow
> to view reports.
> 
> 
> I have a small doubt. I was reading a field of each packet and map it to a
> custom template.(just trying to read ethertype for testing the custom
> templates eventhough etype is available)
> The value shown in pmacct seems to be right.  I tried verifying this data
> with nfacctd, but the value shown in nfacctd seems to be different. I have
> pasted the conf files and output.
> 
> *Conf files:*
> *primitives.lst file*
> *name=dummy_byte packet_ptr=packet:+12 len=2 semantics=raw
> field_type=41234:100*
> 
> *pmacct.conf*
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> *daemonize: false   interface: wlp1s0   !pcap_interfaces_map:
> pcap_interfaces.map   aggregate_primitives: primitives.lst   aggregate:
> src_host, dst_host, src_port, dst_port, proto, tos, dummy_byte   plugins:
> nfprobe, print   nfprobe_version: 10   nfprobe_engine: 100
> nfprobe_receiver: 192.168.1.6:2100 
> !nfprobe_receiver: 10.40.6.6:16367 *
> 
> 
> *nfacct.conf*
> 
> 
> 
> 
> 
> 
> 
> *daemonize: false   nfacctd_ip: 192.168.1.6   nfacctd_port: 2100
> aggregate_primitives: primitives.lst   aggregate: src_host, dst_host,
> src_port, dst_port, proto, tos, dummy_byte   plugins: print
> nfacctd_disable_checks: true*
> 
> $   sudo /usr/local/sbin/pmacctd -f pmacct.conf
> INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd
> 1.7.3-git (20181217-00)
> INFO ( default/core ):  '--enable-l2' '--enable-64bit'
> '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
> '--enable-st-bins'
> INFO ( default/core ): Reading configuration file
> '/home/certes-rajesh/pmacct/pmacct/pmacct.conf'.
> INFO ( default/core ): [primitives.lst] (re)loading map.
> INFO ( default/core ): [primitives.lst] map successfully (re)loaded.
> INFO ( default_nfprobe/nfprobe ): NetFlow probe plugin is originally based
> on softflowd 0.9.7 software, Copyright 2002 Damien Miller 
> All rights reserved.
> INFO ( default_nfprobe/nfprobe ):   TCP timeout: 3600s
> INFO ( default_nfprobe/nfprobe ):  TCP post-RST timeout: 120s
> INFO ( default_nfprobe/nfprobe ):  TCP post-FIN timeout: 300s
> INFO ( default_nfprobe/nfprobe ):   UDP timeout: 300s
> INFO ( default_nfprobe/nfprobe ):  ICMP timeout: 300s
> INFO ( default_nfprobe/nfprobe ):   General timeout: 3600s
> INFO ( default_nfprobe/nfprobe ):  Maximum lifetime: 604800s
> INFO ( default_nfprobe/nfprobe ):   Expiry interval: 60s
> INFO ( default_nfprobe/nfprobe ): Exporting flows to [172.24.1.219]:2100
> INFO ( default_print/print ): cache entries=16411 base cache
> memory=54878384 bytes
> WARN ( default_print/print ): no print_output_file and no
> print_output_lock_file defined.
> INFO ( default/core ): [enp0s31f6,0] link type is: 1
> INFO ( default_print/print ): *** Purging cache - START (PID: 4301) ***
> SRC_IP DST_IP   SRC_PORT
> DST_PORT  PROTOCOLTOS*dummy_byte*  udp_len  PACKETS
> BYTES
> 172.24.1.197239.255.255.250 56940
> 1900   udp 0   *08-00*   180
> 4 800
> fe80::3436:5d8f:abd7:6f0c  f2::fb  5353
> 5353 udp 0*86-DD*
> 48   3 264
> 
> 
> $   sudo nfacctd  -f nfacct.conf
> INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.7.3-git
> (20181217-00)
> INFO ( default/core ):  '--enable-l2' '--enable-64bit'
> '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
> '--enable-st-bins'
> INFO ( default/core ): Reading configuration file
> '/home/certes-rajesh/pmacct/pmacct/nfacct.conf'.
> INFO ( default/core ): [primitives.lst] (re)loading map.
> INFO ( default/core ): [primitives.lst] map successfully (re)loaded.
> INFO ( default/core ): waiting for NetFlow/IPFIX data on 172.24.1.219:2100
> INFO ( default_print/print ): cache entries=16411 base cache
> memory=54878384 bytes
> WARN ( default_print/print ): no print_output_file and no
> print_output_lock_file defined.
> INFO ( default_print/print ): *** Purging cache - START (PID: 4356) ***
> INFO ( default_print/print ): *** Purging cache - END (PID: 4356, QN: 0/0,
> ET: X) ***
> INFO ( def

Re: [pmacct-discussion] Custom primitives with netflow

[RAJESH KUMAR S.R]

Hi Paolo,

Thanks for the help and suggestion ,
Custom primitives works for me with the latest code:) . I'm able to read a
field from packet and export as custom template and I can see those
templates in netflow messages also.
I'll continue with viewing the flows in a collector, I'm using Elastiflow
to view reports.


I have a small doubt. I was reading a field of each packet and map it to a
custom template.(just trying to read ethertype for testing the custom
templates eventhough etype is available)
The value shown in pmacct seems to be right.  I tried verifying this data
with nfacctd, but the value shown in nfacctd seems to be different. I have
pasted the conf files and output.

*Conf files:*
*primitives.lst file*
*name=dummy_byte packet_ptr=packet:+12 len=2 semantics=raw
field_type=41234:100*

*pmacct.conf*










*daemonize: false   interface: wlp1s0   !pcap_interfaces_map:
pcap_interfaces.map   aggregate_primitives: primitives.lst   aggregate:
src_host, dst_host, src_port, dst_port, proto, tos, dummy_byte   plugins:
nfprobe, print   nfprobe_version: 10   nfprobe_engine: 100
nfprobe_receiver: 192.168.1.6:2100 
!nfprobe_receiver: 10.40.6.6:16367 *


*nfacct.conf*







*daemonize: false   nfacctd_ip: 192.168.1.6   nfacctd_port: 2100
aggregate_primitives: primitives.lst   aggregate: src_host, dst_host,
src_port, dst_port, proto, tos, dummy_byte   plugins: print
nfacctd_disable_checks: true*

$   sudo /usr/local/sbin/pmacctd -f pmacct.conf
INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd
1.7.3-git (20181217-00)
INFO ( default/core ):  '--enable-l2' '--enable-64bit'
'--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
'--enable-st-bins'
INFO ( default/core ): Reading configuration file
'/home/certes-rajesh/pmacct/pmacct/pmacct.conf'.
INFO ( default/core ): [primitives.lst] (re)loading map.
INFO ( default/core ): [primitives.lst] map successfully (re)loaded.
INFO ( default_nfprobe/nfprobe ): NetFlow probe plugin is originally based
on softflowd 0.9.7 software, Copyright 2002 Damien Miller 
All rights reserved.
INFO ( default_nfprobe/nfprobe ):   TCP timeout: 3600s
INFO ( default_nfprobe/nfprobe ):  TCP post-RST timeout: 120s
INFO ( default_nfprobe/nfprobe ):  TCP post-FIN timeout: 300s
INFO ( default_nfprobe/nfprobe ):   UDP timeout: 300s
INFO ( default_nfprobe/nfprobe ):  ICMP timeout: 300s
INFO ( default_nfprobe/nfprobe ):   General timeout: 3600s
INFO ( default_nfprobe/nfprobe ):  Maximum lifetime: 604800s
INFO ( default_nfprobe/nfprobe ):   Expiry interval: 60s
INFO ( default_nfprobe/nfprobe ): Exporting flows to [172.24.1.219]:2100
INFO ( default_print/print ): cache entries=16411 base cache
memory=54878384 bytes
WARN ( default_print/print ): no print_output_file and no
print_output_lock_file defined.
INFO ( default/core ): [enp0s31f6,0] link type is: 1
INFO ( default_print/print ): *** Purging cache - START (PID: 4301) ***
SRC_IP DST_IP   SRC_PORT
DST_PORT  PROTOCOLTOS*dummy_byte*  udp_len  PACKETS
BYTES
172.24.1.197239.255.255.250 56940
1900   udp 0   *08-00*   180
4 800
fe80::3436:5d8f:abd7:6f0c  f2::fb  5353
5353 udp 0*86-DD*
48   3 264


$   sudo nfacctd  -f nfacct.conf
INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.7.3-git
(20181217-00)
INFO ( default/core ):  '--enable-l2' '--enable-64bit'
'--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
'--enable-st-bins'
INFO ( default/core ): Reading configuration file
'/home/certes-rajesh/pmacct/pmacct/nfacct.conf'.
INFO ( default/core ): [primitives.lst] (re)loading map.
INFO ( default/core ): [primitives.lst] map successfully (re)loaded.
INFO ( default/core ): waiting for NetFlow/IPFIX data on 172.24.1.219:2100
INFO ( default_print/print ): cache entries=16411 base cache
memory=54878384 bytes
WARN ( default_print/print ): no print_output_file and no
print_output_lock_file defined.
INFO ( default_print/print ): *** Purging cache - START (PID: 4356) ***
INFO ( default_print/print ): *** Purging cache - END (PID: 4356, QN: 0/0,
ET: X) ***
INFO ( default_print/print ): *** Purging cache - START (PID: 4379) ***
INFO ( default_print/print ): *** Purging cache - END (PID: 4379, QN: 0/0,
ET: X) ***
INFO ( default_print/print ): *** Purging cache - START (PID: 4410) ***
INFO ( default_print/print ): *** Purging cache - END (PID: 4410, QN: 0/0,
ET: X) ***
INFO ( default_print/print ): *** Purging cache - START (PID: 4443) ***
SRC_IP  DST_IP SRC_PORT  DST_PORT
PROTOCOLTOS*dummy_byte*  PACKETS   BYTES
172.24.1.197  239.255.255.25056940
1900  udp 0*30-38*
4 800





On Mon, Dec 17

Re: [pmacct-discussion] Custom primitives with netflow

[Paolo Lucente]

Hi Rajesh,

Thanks for pointing this out. I've committed some code to unlock
field_type also for uacctd/pmacctd daemons precisely for the use case
you mentioned. Here the details:

https://github.com/pmacct/pmacct/commit/87ebf3a9f907c331f752c96a76ea247e77f99107

You can back port this patch to latest stable release or use master
code. Keep me posted if it works for you - it did work for me in lab
using your config as a base.

One recommendation: use IPFIX instead of NetFlow v9 if possible. IPFIX
allows to define the field type as :, where pmacct PEN
is documented here:

https://github.com/pmacct/pmacct/blob/master/docs/IPFIX

So you could use, say, 43874:100 as field type instead of squatting the
public code points.

Paolo 

On Sat, Dec 15, 2018 at 12:04:54AM +0530, RAJESH KUMAR S.R wrote:
> Hi,
> 
> I need some understanding in exporting the custom defined primitives in
> netflow v9 messages, if that is possible, as I want to define custom fields
> and send out to netflow collector and visualize using graphs (if the
> collector supports custom templates)
> 
> As a first step, I am trying to use the custom aggregate primitive  used in
> examples/primitives.lst.example.
> 
> " Defines a primitive called 'udp_len': base pointer is set to the UDP
> header
>  (l4:17) plus 4 bytes offset, reads for 2 byte and will present it as
> unsigned
>  int.
> 
> name=udp_lenpacket_ptr=l4:17+4  len=2   semantics=u_int
> "
> 
> I used to classify flows after defining "udp_len" as mentioned above.
> My conf file for pmacctd is
> 
> 
> 
> 
> 
> 
> 
> 
> *"   daemonize:false   interface: wlp1s0   aggregate_primitives:
> primitives.lst   aggregate: etype, proto, src_host, dst_host, src_port,
> dst_port, udp_len   plugins: nfprobe, print   nfprobe_receiver:
> 172.24.1.123:9996    nfprobe_version: 9*
> *"*
> My primitives.lst file defines custom primitive as follows
> 
> *"name=udp_lenpacket_ptr=l4:17+4  len=2   semantics=u_int"*
> 
> When I run the pmacct "sudo pmacctd -f pmacct.conf", I'm able to see the
> flows that has udp_len column displayed in the console using print plugin.
> 
> Output of
> "sudo pmacctd -f pmacct.conf"
> 
> INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd
> 1.7.2-git (20180701-01)
> INFO ( default/core ):  '--enable-l2' '--enable-ipv6' '--enable-64bit'
> '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
> '--enable-st-bins'
> INFO ( default/core ): Reading configuration file
> '/home/certes-rajesh/pmacct/pmacct/pmacct.conf'.
> INFO ( default/core ): [primitives.lst] (re)loading map.
> INFO ( default/core ): [primitives.lst] map successfully (re)loaded.
> INFO ( default_nfprobe/nfprobe ): NetFlow probe plugin is originally based
> on softflowd 0.9.7 software, Copyright 2002 Damien Miller 
> All rights reserved.
> INFO ( default_nfprobe/nfprobe ):   TCP timeout: 3600s
> INFO ( default_nfprobe/nfprobe ):  TCP post-RST timeout: 120s
> INFO ( default_nfprobe/nfprobe ):  TCP post-FIN timeout: 300s
> INFO ( default_nfprobe/nfprobe ):   UDP timeout: 300s
> INFO ( default_nfprobe/nfprobe ):  ICMP timeout: 300s
> INFO ( default_nfprobe/nfprobe ):   General timeout: 3600s
> INFO ( default_nfprobe/nfprobe ):  Maximum lifetime: 604800s
> INFO ( default_nfprobe/nfprobe ):   Expiry interval: 60s
> INFO ( default_nfprobe/nfprobe ): Exporting flows to [192.168.122.1]:9996
> *ERROR ( default_nfprobe/nfprobe ): custom primitive 'udp_len' has null
> field_type*
> INFO ( default_print/print ): cache entries=16411 base cache
> memory=54878384 bytes
> WARN ( default_print/print ): no print_output_file and no
> print_output_lock_file defined.
> INFO ( default/core ): [wlp1s0,0] link type is: 1
> *WARN ( default/core ): connection lost to 'default_nfprobe-nfprobe';
> closing connection.*
> INFO ( default_print/print ): *** Purging cache - START (PID: 2837) ***
> ETYPE  SRC_IP
> DST_IP SRC_PORT  DST_PORT
> PROTOCOLudp_len  PACKETS   BYTES
> 86dd   fd50:1d9:a341:f100:8ae:86f3:123d:3654
> ff02::fb   5353  5353
> udp 41   3 243
> ...
> 
> When I try to give a dummy field type, it throws
> "WARN ( default/core ): [primitives.lst] field_type is only supported in
> nfacctd.".
> 
> I need help in figuring out whether I'm doing the right thing for exporting
> custom fields as part netflow messages as I will need to send out more
> custom fields that are read from the packet.

> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] Custom primitives with netflow

[RAJESH KUMAR S.R]

Hi,

I need some understanding in exporting the custom defined primitives in
netflow v9 messages, if that is possible, as I want to define custom fields
and send out to netflow collector and visualize using graphs (if the
collector supports custom templates)

As a first step, I am trying to use the custom aggregate primitive  used in
examples/primitives.lst.example.

" Defines a primitive called 'udp_len': base pointer is set to the UDP
header
 (l4:17) plus 4 bytes offset, reads for 2 byte and will present it as
unsigned
 int.

name=udp_lenpacket_ptr=l4:17+4  len=2   semantics=u_int
"

I used to classify flows after defining "udp_len" as mentioned above.
My conf file for pmacctd is








*"   daemonize:false   interface: wlp1s0   aggregate_primitives:
primitives.lst   aggregate: etype, proto, src_host, dst_host, src_port,
dst_port, udp_len   plugins: nfprobe, print   nfprobe_receiver:
172.24.1.123:9996    nfprobe_version: 9*
*"*
My primitives.lst file defines custom primitive as follows

*"name=udp_lenpacket_ptr=l4:17+4  len=2   semantics=u_int"*

When I run the pmacct "sudo pmacctd -f pmacct.conf", I'm able to see the
flows that has udp_len column displayed in the console using print plugin.

Output of
"sudo pmacctd -f pmacct.conf"

INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd
1.7.2-git (20180701-01)
INFO ( default/core ):  '--enable-l2' '--enable-ipv6' '--enable-64bit'
'--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
'--enable-st-bins'
INFO ( default/core ): Reading configuration file
'/home/certes-rajesh/pmacct/pmacct/pmacct.conf'.
INFO ( default/core ): [primitives.lst] (re)loading map.
INFO ( default/core ): [primitives.lst] map successfully (re)loaded.
INFO ( default_nfprobe/nfprobe ): NetFlow probe plugin is originally based
on softflowd 0.9.7 software, Copyright 2002 Damien Miller 
All rights reserved.
INFO ( default_nfprobe/nfprobe ):   TCP timeout: 3600s
INFO ( default_nfprobe/nfprobe ):  TCP post-RST timeout: 120s
INFO ( default_nfprobe/nfprobe ):  TCP post-FIN timeout: 300s
INFO ( default_nfprobe/nfprobe ):   UDP timeout: 300s
INFO ( default_nfprobe/nfprobe ):  ICMP timeout: 300s
INFO ( default_nfprobe/nfprobe ):   General timeout: 3600s
INFO ( default_nfprobe/nfprobe ):  Maximum lifetime: 604800s
INFO ( default_nfprobe/nfprobe ):   Expiry interval: 60s
INFO ( default_nfprobe/nfprobe ): Exporting flows to [192.168.122.1]:9996
*ERROR ( default_nfprobe/nfprobe ): custom primitive 'udp_len' has null
field_type*
INFO ( default_print/print ): cache entries=16411 base cache
memory=54878384 bytes
WARN ( default_print/print ): no print_output_file and no
print_output_lock_file defined.
INFO ( default/core ): [wlp1s0,0] link type is: 1
*WARN ( default/core ): connection lost to 'default_nfprobe-nfprobe';
closing connection.*
INFO ( default_print/print ): *** Purging cache - START (PID: 2837) ***
ETYPE  SRC_IP
DST_IP SRC_PORT  DST_PORT
PROTOCOLudp_len  PACKETS   BYTES
86dd   fd50:1d9:a341:f100:8ae:86f3:123d:3654
ff02::fb   5353  5353
udp 41   3 243
...

When I try to give a dummy field type, it throws
"WARN ( default/core ): [primitives.lst] field_type is only supported in
nfacctd.".

I need help in figuring out whether I'm doing the right thing for exporting
custom fields as part netflow messages as I will need to send out more
custom fields that are read from the packet.
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists