Hi Rajesh,
You are right, there was a bug in the serialize_bin() func that was making it work good only for the first byte. This is now resolved: https://github.com/pmacct/pmacct/commit/1076ff3529f439133357176e4c1260cfcdcef56e I've read your question about metadata and was wondering: would tags (tag, tag2 primitive) or labels (label primitive) defined via a pre_tag_map be a solution for you? You could do a proof-of-concept locally, ie. like you were doing already with the print plugin, and if meeting your requirements we can move onto the nfprobe part; i expect tags to work no problem; labels should work but may require a bit more testing. Paolo On Mon, Dec 24, 2018 at 02:21:18PM +0530, RAJESH KUMAR S.R wrote: > Hi Paolo, > > Thanks for the fix. I tested with pmacctd and nfacctd and I see that when I > read 1 byte of raw data, it prints correct on both sides > but by while reading more bytes, the first byte is alone correct on nfacct > side. Not sure I'm testing correctly, but thanks for the fix. > > pmacctd side > dummy_byte PACKETS BYTES > *08-00-45* 535 124114 > *86-DD-60* 10 861 > > On nfacctd side, I'm getting only the first byte correct > *08-00-00* 535 124114 > *86-00-00* 10 861 > > > I need one more suggestion/help regarding custom primitives. > Right now I can read a set of bytes from packet and send it as custom > primitives using the > recommended PEN id which I can see in netflow message in wireshark > also(attached the wireshark image) > > I have requirement where I need to send some *metadata*/context as part as > netflow records. > The metadata/context won't be available in the packet. Will I be able to > define and use > custom primitives for sending those metadata, (say one or more > context/metadata in pmacct conf file, similar to ifindex) > I'm reading packet from different interfaces, where I know the context of > packets that are arriving on each interface > I need to send that context as one of the netflow field. The only way I > know to do now is to write that context as part of the packet and read as > custom template. > So, basically I'm trying to figure out other possible ways of defining > custom primitives > & fieldtypes and using it without reading from packet. > It would be very helpful it any alternate way is feasible. > > On Thu, Dec 20, 2018 at 6:23 AM Paolo Lucente <[email protected]> wrote: > > > > > Hi Rajesh, > > > > Thanks for bringing this further point up. While all was working with > > hex and u_int semantics, there was an issue with the raw one - which was > > important to fix since it's a very generic transport for any kind of > > info. Here is the commit: > > > > > > https://github.com/pmacct/pmacct/commit/e1206a530af64c46748b5f891a9034dec89e9b7a > > > > Let me know if this works for you. > > > > Paolo > > > > On Mon, Dec 17, 2018 at 10:25:27PM +0530, RAJESH KUMAR S.R wrote: > > > Hi Paolo, > > > > > > Thanks for the help and suggestion , > > > Custom primitives works for me with the latest code:) . I'm able to read > > a > > > field from packet and export as custom template and I can see those > > > templates in netflow messages also. > > > I'll continue with viewing the flows in a collector, I'm using Elastiflow > > > to view reports. > > > > > > > > > I have a small doubt. I was reading a field of each packet and map it to > > a > > > custom template.(just trying to read ethertype for testing the custom > > > templates eventhough etype is available) > > > The value shown in pmacct seems to be right. I tried verifying this data > > > with nfacctd, but the value shown in nfacctd seems to be different. I > > have > > > pasted the conf files and output. > > > > > > *Conf files:* > > > *primitives.lst file* > > > *name=dummy_byte packet_ptr=packet:+12 len=2 semantics=raw > > > field_type=41234:100* > > > > > > *pmacct.conf* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *daemonize: false interface: wlp1s0 !pcap_interfaces_map: > > > pcap_interfaces.map aggregate_primitives: primitives.lst aggregate: > > > src_host, dst_host, src_port, dst_port, proto, tos, dummy_byte plugins: > > > nfprobe, print nfprobe_version: 10 nfprobe_engine: 100 > > > nfprobe_receiver: 192.168.1.6:2100 <http://192.168.1.6:2100> > > > !nfprobe_receiver: 10.40.6.6:16367 <http://10.40.6.6:16367>* > > > > > > > > > *nfacct.conf* > > > > > > > > > > > > > > > > > > > > > > > > *daemonize: false nfacctd_ip: 192.168.1.6 nfacctd_port: 2100 > > > aggregate_primitives: primitives.lst aggregate: src_host, dst_host, > > > src_port, dst_port, proto, tos, dummy_byte plugins: print > > > nfacctd_disable_checks: true* > > > > > > $ sudo /usr/local/sbin/pmacctd -f pmacct.conf > > > INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd > > > 1.7.3-git (20181217-00) > > > INFO ( default/core ): '--enable-l2' '--enable-64bit' > > > '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' > > > '--enable-st-bins' > > > INFO ( default/core ): Reading configuration file > > > '/home/certes-rajesh/pmacct/pmacct/pmacct.conf'. > > > INFO ( default/core ): [primitives.lst] (re)loading map. > > > INFO ( default/core ): [primitives.lst] map successfully (re)loaded. > > > INFO ( default_nfprobe/nfprobe ): NetFlow probe plugin is originally > > based > > > on softflowd 0.9.7 software, Copyright 2002 Damien Miller < > > [email protected]> > > > All rights reserved. > > > INFO ( default_nfprobe/nfprobe ): TCP timeout: 3600s > > > INFO ( default_nfprobe/nfprobe ): TCP post-RST timeout: 120s > > > INFO ( default_nfprobe/nfprobe ): TCP post-FIN timeout: 300s > > > INFO ( default_nfprobe/nfprobe ): UDP timeout: 300s > > > INFO ( default_nfprobe/nfprobe ): ICMP timeout: 300s > > > INFO ( default_nfprobe/nfprobe ): General timeout: 3600s > > > INFO ( default_nfprobe/nfprobe ): Maximum lifetime: 604800s > > > INFO ( default_nfprobe/nfprobe ): Expiry interval: 60s > > > INFO ( default_nfprobe/nfprobe ): Exporting flows to [172.24.1.219]:2100 > > > INFO ( default_print/print ): cache entries=16411 base cache > > > memory=54878384 bytes > > > WARN ( default_print/print ): no print_output_file and no > > > print_output_lock_file defined. > > > INFO ( default/core ): [enp0s31f6,0] link type is: 1 > > > INFO ( default_print/print ): *** Purging cache - START (PID: 4301) *** > > > SRC_IP DST_IP SRC_PORT > > > DST_PORT PROTOCOL TOS *dummy_byte* udp_len PACKETS > > > BYTES > > > 172.24.1.197 239.255.255.250 56940 > > > 1900 udp 0 *08-00* 180 > > > 4 800 > > > fe80::3436:5d8f:abd7:6f0c f2::fb 5353 > > > 5353 udp 0 *86-DD* > > > 48 3 264 > > > > > > > > > $ sudo nfacctd -f nfacct.conf > > > INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.7.3-git > > > (20181217-00) > > > INFO ( default/core ): '--enable-l2' '--enable-64bit' > > > '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' > > > '--enable-st-bins' > > > INFO ( default/core ): Reading configuration file > > > '/home/certes-rajesh/pmacct/pmacct/nfacct.conf'. > > > INFO ( default/core ): [primitives.lst] (re)loading map. > > > INFO ( default/core ): [primitives.lst] map successfully (re)loaded. > > > INFO ( default/core ): waiting for NetFlow/IPFIX data on > > 172.24.1.219:2100 > > > INFO ( default_print/print ): cache entries=16411 base cache > > > memory=54878384 bytes > > > WARN ( default_print/print ): no print_output_file and no > > > print_output_lock_file defined. > > > INFO ( default_print/print ): *** Purging cache - START (PID: 4356) *** > > > INFO ( default_print/print ): *** Purging cache - END (PID: 4356, QN: > > 0/0, > > > ET: X) *** > > > INFO ( default_print/print ): *** Purging cache - START (PID: 4379) *** > > > INFO ( default_print/print ): *** Purging cache - END (PID: 4379, QN: > > 0/0, > > > ET: X) *** > > > INFO ( default_print/print ): *** Purging cache - START (PID: 4410) *** > > > INFO ( default_print/print ): *** Purging cache - END (PID: 4410, QN: > > 0/0, > > > ET: X) *** > > > INFO ( default_print/print ): *** Purging cache - START (PID: 4443) *** > > > SRC_IP DST_IP SRC_PORT DST_PORT > > > PROTOCOL TOS *dummy_byte* PACKETS BYTES > > > 172.24.1.197 239.255.255.250 56940 > > > 1900 udp 0 *30-38* > > > 4 800 > > > > > > > > > > > > > > > > > > On Mon, Dec 17, 2018 at 6:47 AM Paolo Lucente <[email protected]> wrote: > > > > > > > > > > > Hi Rajesh, > > > > > > > > Thanks for pointing this out. I've committed some code to unlock > > > > field_type also for uacctd/pmacctd daemons precisely for the use case > > > > you mentioned. Here the details: > > > > > > > > > > > > > > https://github.com/pmacct/pmacct/commit/87ebf3a9f907c331f752c96a76ea247e77f99107 > > > > > > > > You can back port this patch to latest stable release or use master > > > > code. Keep me posted if it works for you - it did work for me in lab > > > > using your config as a base. > > > > > > > > One recommendation: use IPFIX instead of NetFlow v9 if possible. IPFIX > > > > allows to define the field type as <PEN>:<field_type>, where pmacct PEN > > > > is documented here: > > > > > > > > https://github.com/pmacct/pmacct/blob/master/docs/IPFIX > > > > > > > > So you could use, say, 43874:100 as field type instead of squatting the > > > > public code points. > > > > > > > > Paolo > > > > > > > > On Sat, Dec 15, 2018 at 12:04:54AM +0530, RAJESH KUMAR S.R wrote: > > > > > Hi, > > > > > > > > > > I need some understanding in exporting the custom defined primitives > > in > > > > > netflow v9 messages, if that is possible, as I want to define custom > > > > fields > > > > > and send out to netflow collector and visualize using graphs (if the > > > > > collector supports custom templates) > > > > > > > > > > As a first step, I am trying to use the custom aggregate primitive > > used > > > > in > > > > > examples/primitives.lst.example. > > > > > > > > > > " Defines a primitive called 'udp_len': base pointer is set to the > > UDP > > > > > header > > > > > (l4:17) plus 4 bytes offset, reads for 2 byte and will present it as > > > > > unsigned > > > > > int. > > > > > > > > > > name=udp_len packet_ptr=l4:17+4 len=2 semantics=u_int > > > > > " > > > > > > > > > > I used to classify flows after defining "udp_len" as mentioned above. > > > > > My conf file for pmacctd is > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *" daemonize:false interface: wlp1s0 aggregate_primitives: > > > > > primitives.lst aggregate: etype, proto, src_host, dst_host, > > src_port, > > > > > dst_port, udp_len plugins: nfprobe, print nfprobe_receiver: > > > > > 172.24.1.123:9996 <http://172.24.1.123:9996> nfprobe_version: 9* > > > > > *"* > > > > > My primitives.lst file defines custom primitive as follows > > > > > > > > > > *"name=udp_len packet_ptr=l4:17+4 len=2 semantics=u_int"* > > > > > > > > > > When I run the pmacct "sudo pmacctd -f pmacct.conf", I'm able to see > > the > > > > > flows that has udp_len column displayed in the console using print > > > > plugin. > > > > > > > > > > Output of > > > > > "sudo pmacctd -f pmacct.conf" > > > > > > > > > > INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd > > > > > 1.7.2-git (20180701-01) > > > > > INFO ( default/core ): '--enable-l2' '--enable-ipv6' > > '--enable-64bit' > > > > > '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' > > > > > '--enable-st-bins' > > > > > INFO ( default/core ): Reading configuration file > > > > > '/home/certes-rajesh/pmacct/pmacct/pmacct.conf'. > > > > > INFO ( default/core ): [primitives.lst] (re)loading map. > > > > > INFO ( default/core ): [primitives.lst] map successfully (re)loaded. > > > > > INFO ( default_nfprobe/nfprobe ): NetFlow probe plugin is originally > > > > based > > > > > on softflowd 0.9.7 software, Copyright 2002 Damien Miller < > > > > [email protected]> > > > > > All rights reserved. > > > > > INFO ( default_nfprobe/nfprobe ): TCP timeout: 3600s > > > > > INFO ( default_nfprobe/nfprobe ): TCP post-RST timeout: 120s > > > > > INFO ( default_nfprobe/nfprobe ): TCP post-FIN timeout: 300s > > > > > INFO ( default_nfprobe/nfprobe ): UDP timeout: 300s > > > > > INFO ( default_nfprobe/nfprobe ): ICMP timeout: 300s > > > > > INFO ( default_nfprobe/nfprobe ): General timeout: 3600s > > > > > INFO ( default_nfprobe/nfprobe ): Maximum lifetime: 604800s > > > > > INFO ( default_nfprobe/nfprobe ): Expiry interval: 60s > > > > > INFO ( default_nfprobe/nfprobe ): Exporting flows to > > [192.168.122.1]:9996 > > > > > *ERROR ( default_nfprobe/nfprobe ): custom primitive 'udp_len' has > > null > > > > > field_type* > > > > > INFO ( default_print/print ): cache entries=16411 base cache > > > > > memory=54878384 bytes > > > > > WARN ( default_print/print ): no print_output_file and no > > > > > print_output_lock_file defined. > > > > > INFO ( default/core ): [wlp1s0,0] link type is: 1 > > > > > *WARN ( default/core ): connection lost to 'default_nfprobe-nfprobe'; > > > > > closing connection.* > > > > > INFO ( default_print/print ): *** Purging cache - START (PID: 2837) > > *** > > > > > ETYPE SRC_IP > > > > > DST_IP SRC_PORT DST_PORT > > > > > PROTOCOL udp_len PACKETS BYTES > > > > > 86dd fd50:1d9:a341:f100:8ae:86f3:123d:3654 > > > > > ff02::fb 5353 5353 > > > > > udp 41 3 243 > > > > > ....... > > > > > > > > > > When I try to give a dummy field type, it throws > > > > > "WARN ( default/core ): [primitives.lst] field_type is only > > supported in > > > > > nfacctd.". > > > > > > > > > > I need help in figuring out whether I'm doing the right thing for > > > > exporting > > > > > custom fields as part netflow messages as I will need to send out > > more > > > > > custom fields that are read from the packet. > > > > > > > > > _______________________________________________ > > > > > pmacct-discussion mailing list > > > > > http://www.pmacct.net/#mailinglists > > > > > > > > > > > > _______________________________________________ > > > > pmacct-discussion mailing list > > > > http://www.pmacct.net/#mailinglists > > > > > > _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
