Hi Rajesh,

You are right, there was a bug in the serialize_bin() func that was
making it work good only for the first byte. This is now resolved:

https://github.com/pmacct/pmacct/commit/1076ff3529f439133357176e4c1260cfcdcef56e

I've read your question about metadata and was wondering: would tags
(tag, tag2 primitive) or labels (label primitive) defined via a
pre_tag_map be a solution for you? You could do a proof-of-concept
locally, ie. like you were doing already with the print plugin, and if
meeting your requirements we can move onto the nfprobe part; i expect
tags to work no problem; labels should work but may require a bit more
testing. 

Paolo

On Mon, Dec 24, 2018 at 02:21:18PM +0530, RAJESH KUMAR S.R wrote:
> Hi Paolo,
> 
> Thanks for the fix. I tested with pmacctd and nfacctd and I see that when I
> read 1 byte of raw data, it prints correct on both sides
> but by while reading more bytes, the first byte is alone correct on nfacct
> side. Not sure I'm testing correctly, but thanks for the fix.
> 
> pmacctd side
> dummy_byte  PACKETS               BYTES
> *08-00-45*    535                   124114
> *86-DD-60*    10                    861
> 
> On nfacctd side, I'm getting only the first byte correct
> *08-00-00*    535                 124114
> *86-00-00*    10                   861
> 
> 
> I need one more suggestion/help regarding custom primitives.
> Right now I can read a set of bytes from packet and send it as custom
> primitives using the
> recommended PEN id which I can see in netflow message in wireshark
> also(attached the wireshark image)
> 
> I have requirement where I need to send some *metadata*/context as part as
> netflow records.
> The metadata/context won't be available in the packet. Will I be able to
> define and use
> custom primitives for sending those metadata, (say one or more
> context/metadata in pmacct conf file, similar to ifindex)
> I'm reading packet from different interfaces, where I know the context of
> packets that are arriving on each interface
> I need to send that context as one of the netflow field. The only way I
> know to do now is to write that context as part of the packet and read as
> custom template.
> So, basically I'm trying to figure out other possible ways of defining
> custom primitives
> & fieldtypes and using it without reading from packet.
> It would be very helpful it any alternate way is feasible.
> 
> On Thu, Dec 20, 2018 at 6:23 AM Paolo Lucente <pa...@pmacct.net> wrote:
> 
> >
> > Hi Rajesh,
> >
> > Thanks for bringing this further point up. While all was working with
> > hex and u_int semantics, there was an issue with the raw one - which was
> > important to fix since it's a very generic transport for any kind of
> > info. Here is the commit:
> >
> >
> > https://github.com/pmacct/pmacct/commit/e1206a530af64c46748b5f891a9034dec89e9b7a
> >
> > Let me know if this works for you.
> >
> > Paolo
> >
> > On Mon, Dec 17, 2018 at 10:25:27PM +0530, RAJESH KUMAR S.R wrote:
> > > Hi Paolo,
> > >
> > > Thanks for the help and suggestion ,
> > > Custom primitives works for me with the latest code:) . I'm able to read
> > a
> > > field from packet and export as custom template and I can see those
> > > templates in netflow messages also.
> > > I'll continue with viewing the flows in a collector, I'm using Elastiflow
> > > to view reports.
> > >
> > >
> > > I have a small doubt. I was reading a field of each packet and map it to
> > a
> > > custom template.(just trying to read ethertype for testing the custom
> > > templates eventhough etype is available)
> > > The value shown in pmacct seems to be right.  I tried verifying this data
> > > with nfacctd, but the value shown in nfacctd seems to be different. I
> > have
> > > pasted the conf files and output.
> > >
> > > *Conf files:*
> > > *primitives.lst file*
> > > *name=dummy_byte packet_ptr=packet:+12 len=2 semantics=raw
> > > field_type=41234:100*
> > >
> > > *pmacct.conf*
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *daemonize: false   interface: wlp1s0   !pcap_interfaces_map:
> > > pcap_interfaces.map   aggregate_primitives: primitives.lst   aggregate:
> > > src_host, dst_host, src_port, dst_port, proto, tos, dummy_byte   plugins:
> > > nfprobe, print   nfprobe_version: 10   nfprobe_engine: 100
> > > nfprobe_receiver: 192.168.1.6:2100 <http://192.168.1.6:2100>
> > > !nfprobe_receiver: 10.40.6.6:16367 <http://10.40.6.6:16367>*
> > >
> > >
> > > *nfacct.conf*
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *daemonize: false   nfacctd_ip: 192.168.1.6   nfacctd_port: 2100
> > > aggregate_primitives: primitives.lst   aggregate: src_host, dst_host,
> > > src_port, dst_port, proto, tos, dummy_byte   plugins: print
> > > nfacctd_disable_checks: true*
> > >
> > > $   sudo /usr/local/sbin/pmacctd -f pmacct.conf
> > > INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd
> > > 1.7.3-git (20181217-00)
> > > INFO ( default/core ):  '--enable-l2' '--enable-64bit'
> > > '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
> > > '--enable-st-bins'
> > > INFO ( default/core ): Reading configuration file
> > > '/home/certes-rajesh/pmacct/pmacct/pmacct.conf'.
> > > INFO ( default/core ): [primitives.lst] (re)loading map.
> > > INFO ( default/core ): [primitives.lst] map successfully (re)loaded.
> > > INFO ( default_nfprobe/nfprobe ): NetFlow probe plugin is originally
> > based
> > > on softflowd 0.9.7 software, Copyright 2002 Damien Miller <
> > d...@mindrot.org>
> > > All rights reserved.
> > > INFO ( default_nfprobe/nfprobe ):           TCP timeout: 3600s
> > > INFO ( default_nfprobe/nfprobe ):  TCP post-RST timeout: 120s
> > > INFO ( default_nfprobe/nfprobe ):  TCP post-FIN timeout: 300s
> > > INFO ( default_nfprobe/nfprobe ):           UDP timeout: 300s
> > > INFO ( default_nfprobe/nfprobe ):          ICMP timeout: 300s
> > > INFO ( default_nfprobe/nfprobe ):       General timeout: 3600s
> > > INFO ( default_nfprobe/nfprobe ):      Maximum lifetime: 604800s
> > > INFO ( default_nfprobe/nfprobe ):       Expiry interval: 60s
> > > INFO ( default_nfprobe/nfprobe ): Exporting flows to [172.24.1.219]:2100
> > > INFO ( default_print/print ): cache entries=16411 base cache
> > > memory=54878384 bytes
> > > WARN ( default_print/print ): no print_output_file and no
> > > print_output_lock_file defined.
> > > INFO ( default/core ): [enp0s31f6,0] link type is: 1
> > > INFO ( default_print/print ): *** Purging cache - START (PID: 4301) ***
> > > SRC_IP                         DST_IP                       SRC_PORT
> > > DST_PORT  PROTOCOL    TOS    *dummy_byte*  udp_len  PACKETS
> > > BYTES
> > > 172.24.1.197            239.255.255.250                 56940
> > > 1900               udp                 0       *08-00*           180
> > > 4                     800
> > > fe80::3436:5d8f:abd7:6f0c      f2::fb              5353
> > > 5353         udp                 0            *86-DD*
> > > 48       3                     264
> > >
> > >
> > > $   sudo nfacctd  -f nfacct.conf
> > > INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.7.3-git
> > > (20181217-00)
> > > INFO ( default/core ):  '--enable-l2' '--enable-64bit'
> > > '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
> > > '--enable-st-bins'
> > > INFO ( default/core ): Reading configuration file
> > > '/home/certes-rajesh/pmacct/pmacct/nfacct.conf'.
> > > INFO ( default/core ): [primitives.lst] (re)loading map.
> > > INFO ( default/core ): [primitives.lst] map successfully (re)loaded.
> > > INFO ( default/core ): waiting for NetFlow/IPFIX data on
> > 172.24.1.219:2100
> > > INFO ( default_print/print ): cache entries=16411 base cache
> > > memory=54878384 bytes
> > > WARN ( default_print/print ): no print_output_file and no
> > > print_output_lock_file defined.
> > > INFO ( default_print/print ): *** Purging cache - START (PID: 4356) ***
> > > INFO ( default_print/print ): *** Purging cache - END (PID: 4356, QN:
> > 0/0,
> > > ET: X) ***
> > > INFO ( default_print/print ): *** Purging cache - START (PID: 4379) ***
> > > INFO ( default_print/print ): *** Purging cache - END (PID: 4379, QN:
> > 0/0,
> > > ET: X) ***
> > > INFO ( default_print/print ): *** Purging cache - START (PID: 4410) ***
> > > INFO ( default_print/print ): *** Purging cache - END (PID: 4410, QN:
> > 0/0,
> > > ET: X) ***
> > > INFO ( default_print/print ): *** Purging cache - START (PID: 4443) ***
> > > SRC_IP                          DST_IP                 SRC_PORT  DST_PORT
> > > PROTOCOL    TOS    *dummy_byte*  PACKETS               BYTES
> > > 172.24.1.197          239.255.255.250            56940
> > > 1900          udp                 0            *30-38*
> > > 4                     800
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Dec 17, 2018 at 6:47 AM Paolo Lucente <pa...@pmacct.net> wrote:
> > >
> > > >
> > > > Hi Rajesh,
> > > >
> > > > Thanks for pointing this out. I've committed some code to unlock
> > > > field_type also for uacctd/pmacctd daemons precisely for the use case
> > > > you mentioned. Here the details:
> > > >
> > > >
> > > >
> > https://github.com/pmacct/pmacct/commit/87ebf3a9f907c331f752c96a76ea247e77f99107
> > > >
> > > > You can back port this patch to latest stable release or use master
> > > > code. Keep me posted if it works for you - it did work for me in lab
> > > > using your config as a base.
> > > >
> > > > One recommendation: use IPFIX instead of NetFlow v9 if possible. IPFIX
> > > > allows to define the field type as <PEN>:<field_type>, where pmacct PEN
> > > > is documented here:
> > > >
> > > > https://github.com/pmacct/pmacct/blob/master/docs/IPFIX
> > > >
> > > > So you could use, say, 43874:100 as field type instead of squatting the
> > > > public code points.
> > > >
> > > > Paolo
> > > >
> > > > On Sat, Dec 15, 2018 at 12:04:54AM +0530, RAJESH KUMAR S.R wrote:
> > > > > Hi,
> > > > >
> > > > > I need some understanding in exporting the custom defined primitives
> > in
> > > > > netflow v9 messages, if that is possible, as I want to define custom
> > > > fields
> > > > > and send out to netflow collector and visualize using graphs (if the
> > > > > collector supports custom templates)
> > > > >
> > > > > As a first step, I am trying to use the custom aggregate primitive
> > used
> > > > in
> > > > > examples/primitives.lst.example.
> > > > >
> > > > > " Defines a primitive called 'udp_len': base pointer is set to the
> > UDP
> > > > > header
> > > > >  (l4:17) plus 4 bytes offset, reads for 2 byte and will present it as
> > > > > unsigned
> > > > >  int.
> > > > >
> > > > > name=udp_len    packet_ptr=l4:17+4      len=2   semantics=u_int
> > > > > "
> > > > >
> > > > > I used to classify flows after defining "udp_len" as mentioned above.
> > > > > My conf file for pmacctd is
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *"   daemonize:false   interface: wlp1s0   aggregate_primitives:
> > > > > primitives.lst   aggregate: etype, proto, src_host, dst_host,
> > src_port,
> > > > > dst_port, udp_len   plugins: nfprobe, print   nfprobe_receiver:
> > > > > 172.24.1.123:9996 <http://172.24.1.123:9996>   nfprobe_version: 9*
> > > > > *"*
> > > > > My primitives.lst file defines custom primitive as follows
> > > > >
> > > > > *"name=udp_len    packet_ptr=l4:17+4      len=2   semantics=u_int"*
> > > > >
> > > > > When I run the pmacct "sudo pmacctd -f pmacct.conf", I'm able to see
> > the
> > > > > flows that has udp_len column displayed in the console using print
> > > > plugin.
> > > > >
> > > > > Output of
> > > > > "sudo pmacctd -f pmacct.conf"
> > > > >
> > > > > INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd
> > > > > 1.7.2-git (20180701-01)
> > > > > INFO ( default/core ):  '--enable-l2' '--enable-ipv6'
> > '--enable-64bit'
> > > > > '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
> > > > > '--enable-st-bins'
> > > > > INFO ( default/core ): Reading configuration file
> > > > > '/home/certes-rajesh/pmacct/pmacct/pmacct.conf'.
> > > > > INFO ( default/core ): [primitives.lst] (re)loading map.
> > > > > INFO ( default/core ): [primitives.lst] map successfully (re)loaded.
> > > > > INFO ( default_nfprobe/nfprobe ): NetFlow probe plugin is originally
> > > > based
> > > > > on softflowd 0.9.7 software, Copyright 2002 Damien Miller <
> > > > d...@mindrot.org>
> > > > > All rights reserved.
> > > > > INFO ( default_nfprobe/nfprobe ):           TCP timeout: 3600s
> > > > > INFO ( default_nfprobe/nfprobe ):  TCP post-RST timeout: 120s
> > > > > INFO ( default_nfprobe/nfprobe ):  TCP post-FIN timeout: 300s
> > > > > INFO ( default_nfprobe/nfprobe ):           UDP timeout: 300s
> > > > > INFO ( default_nfprobe/nfprobe ):          ICMP timeout: 300s
> > > > > INFO ( default_nfprobe/nfprobe ):       General timeout: 3600s
> > > > > INFO ( default_nfprobe/nfprobe ):      Maximum lifetime: 604800s
> > > > > INFO ( default_nfprobe/nfprobe ):       Expiry interval: 60s
> > > > > INFO ( default_nfprobe/nfprobe ): Exporting flows to
> > [192.168.122.1]:9996
> > > > > *ERROR ( default_nfprobe/nfprobe ): custom primitive 'udp_len' has
> > null
> > > > > field_type*
> > > > > INFO ( default_print/print ): cache entries=16411 base cache
> > > > > memory=54878384 bytes
> > > > > WARN ( default_print/print ): no print_output_file and no
> > > > > print_output_lock_file defined.
> > > > > INFO ( default/core ): [wlp1s0,0] link type is: 1
> > > > > *WARN ( default/core ): connection lost to 'default_nfprobe-nfprobe';
> > > > > closing connection.*
> > > > > INFO ( default_print/print ): *** Purging cache - START (PID: 2837)
> > ***
> > > > > ETYPE  SRC_IP
> > > > > DST_IP                                         SRC_PORT  DST_PORT
> > > > > PROTOCOL    udp_len  PACKETS               BYTES
> > > > > 86dd   fd50:1d9:a341:f100:8ae:86f3:123d:3654
> > > > > ff02::fb                                       5353      5353
> > > > > udp         41       3                     243
> > > > > .......
> > > > >
> > > > > When I try to give a dummy field type, it throws
> > > > > "WARN ( default/core ): [primitives.lst] field_type is only
> > supported in
> > > > > nfacctd.".
> > > > >
> > > > > I need help in figuring out whether I'm doing the right thing for
> > > > exporting
> > > > > custom fields as part netflow messages as I will need to send out
> > more
> > > > > custom fields that are read from the packet.
> > > >
> > > > > _______________________________________________
> > > > > pmacct-discussion mailing list
> > > > > http://www.pmacct.net/#mailinglists
> > > >
> > > >
> > > > _______________________________________________
> > > > pmacct-discussion mailing list
> > > > http://www.pmacct.net/#mailinglists
> > > >
> >




_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Reply via email to