Hi Rajesh,
Nice labels worked for you. Clarify me one thing: the output you did show, with zeroed peer_src_ip (and exporteripv4address, engineid, enginetype), is the one from pmacctd, right? Not nfacctd. In that case the output is expected. In fact in nfacctd it should not be possible to get a null peer_src_ip (which is nothing else than the address returned by a recv() on the socket); setting nfprobe_source_ip is needed only in cases in which multiple interfaces could be selected for output or for settin field type #130 for, for example, NAT traversal scenarios. Paolo On Sat, Jan 05, 2019 at 12:29:35AM +0530, RAJESH KUMAR S.R wrote: > Hi Paolo, > > I was able to set labels and export as strings for different pmacct > instances that was listening on different interfaces. Thanks for the > suggestion. > > I need a help regarding exporting Exporter's IP as part of flow records. > Based on old pmacct discussions, I'm using "peer_src_ip and > exporterIPv4Address" as primitives but both they seem to come as 0 and > 0.0.0.0 in flows. > I tried setting the "nfprobe_source_ip: 172.30.130.99", but it goes in > separate flow as ExporterAddress: 172.30.130.99, but I need the > "exporterIpv4Address" to be set to correct value in all flow messages, will > pmacct collect the interface ip and populate in "exporterIpv4Address" > field. Also, I'm not sure of how to get the engineid working, that also > seem to go as 0 in flows. > > pmacctd Output: > > *SRC_MAC DST_MAC PEER_SRC_IP* > 50:01:d9:a3:41:f1 f8:59:71:73:94:4d :: > * SRC_IP DST_IP SRC_PORT DST_PORT PROTOCOL * > 52.229.174.94 192.168.1.9 443 43238 tcp > > *exporteripv4address engineid enginetype PACKETS BYTES* > 0.0.0.0 0 0 > 2 629 > > > I have the following pmacct conf file > > *primitives* > > name=engineType field_type=0:38 len=1 semantics=u_int > name=engineId field_type=0:39 len=1 semantics=u_int > name=exporterIPv4Address field_type=130 len=4 semantics=ip > > *pmacct.conf* > " > debug: true > daemonize: false > pre_tag_map: ipfix_pretag.map > > nfprobe_engine: 100 > nfprobe_version: 10 > nfprobe_source_ip: 172.30.130.99 > aggregate_primitives: ipfix_primitives.lst > plugins: nfprobe, print > interface: wlp1s0 > aggregate: src_mac, dst_mac, src_host, dst_host, src_port, dst_port, > proto, peer_src_ip, exporterIPv4Address, engineId, engineType > nfprobe_receiver: 10.40.6.6:17058 > " > > > > > On Wed, Dec 26, 2018 at 12:44 PM RAJESH KUMAR S.R <[email protected]> > wrote: > > > Hi Paolo, > > > > Thanks for the fix and suggestion. I'll try tag and label primitives and > > see if they match my requirements. > > > > On Tue, Dec 25, 2018 at 10:49 PM Paolo Lucente <[email protected]> wrote: > > > >> > >> Hi Rajesh, > >> > >> You are right, there was a bug in the serialize_bin() func that was > >> making it work good only for the first byte. This is now resolved: > >> > >> > >> https://github.com/pmacct/pmacct/commit/1076ff3529f439133357176e4c1260cfcdcef56e > >> > >> I've read your question about metadata and was wondering: would tags > >> (tag, tag2 primitive) or labels (label primitive) defined via a > >> pre_tag_map be a solution for you? You could do a proof-of-concept > >> locally, ie. like you were doing already with the print plugin, and if > >> meeting your requirements we can move onto the nfprobe part; i expect > >> tags to work no problem; labels should work but may require a bit more > >> testing. > >> > >> Paolo > >> > >> On Mon, Dec 24, 2018 at 02:21:18PM +0530, RAJESH KUMAR S.R wrote: > >> > Hi Paolo, > >> > > >> > Thanks for the fix. I tested with pmacctd and nfacctd and I see that > >> when I > >> > read 1 byte of raw data, it prints correct on both sides > >> > but by while reading more bytes, the first byte is alone correct on > >> nfacct > >> > side. Not sure I'm testing correctly, but thanks for the fix. > >> > > >> > pmacctd side > >> > dummy_byte PACKETS BYTES > >> > *08-00-45* 535 124114 > >> > *86-DD-60* 10 861 > >> > > >> > On nfacctd side, I'm getting only the first byte correct > >> > *08-00-00* 535 124114 > >> > *86-00-00* 10 861 > >> > > >> > > >> > I need one more suggestion/help regarding custom primitives. > >> > Right now I can read a set of bytes from packet and send it as custom > >> > primitives using the > >> > recommended PEN id which I can see in netflow message in wireshark > >> > also(attached the wireshark image) > >> > > >> > I have requirement where I need to send some *metadata*/context as part > >> as > >> > netflow records. > >> > The metadata/context won't be available in the packet. Will I be able to > >> > define and use > >> > custom primitives for sending those metadata, (say one or more > >> > context/metadata in pmacct conf file, similar to ifindex) > >> > I'm reading packet from different interfaces, where I know the context > >> of > >> > packets that are arriving on each interface > >> > I need to send that context as one of the netflow field. The only way I > >> > know to do now is to write that context as part of the packet and read > >> as > >> > custom template. > >> > So, basically I'm trying to figure out other possible ways of defining > >> > custom primitives > >> > & fieldtypes and using it without reading from packet. > >> > It would be very helpful it any alternate way is feasible. > >> > > >> > On Thu, Dec 20, 2018 at 6:23 AM Paolo Lucente <[email protected]> wrote: > >> > > >> > > > >> > > Hi Rajesh, > >> > > > >> > > Thanks for bringing this further point up. While all was working with > >> > > hex and u_int semantics, there was an issue with the raw one - which > >> was > >> > > important to fix since it's a very generic transport for any kind of > >> > > info. Here is the commit: > >> > > > >> > > > >> > > > >> https://github.com/pmacct/pmacct/commit/e1206a530af64c46748b5f891a9034dec89e9b7a > >> > > > >> > > Let me know if this works for you. > >> > > > >> > > Paolo > >> > > > >> > > On Mon, Dec 17, 2018 at 10:25:27PM +0530, RAJESH KUMAR S.R wrote: > >> > > > Hi Paolo, > >> > > > > >> > > > Thanks for the help and suggestion , > >> > > > Custom primitives works for me with the latest code:) . I'm able to > >> read > >> > > a > >> > > > field from packet and export as custom template and I can see those > >> > > > templates in netflow messages also. > >> > > > I'll continue with viewing the flows in a collector, I'm using > >> Elastiflow > >> > > > to view reports. > >> > > > > >> > > > > >> > > > I have a small doubt. I was reading a field of each packet and map > >> it to > >> > > a > >> > > > custom template.(just trying to read ethertype for testing the > >> custom > >> > > > templates eventhough etype is available) > >> > > > The value shown in pmacct seems to be right. I tried verifying > >> this data > >> > > > with nfacctd, but the value shown in nfacctd seems to be different. > >> I > >> > > have > >> > > > pasted the conf files and output. > >> > > > > >> > > > *Conf files:* > >> > > > *primitives.lst file* > >> > > > *name=dummy_byte packet_ptr=packet:+12 len=2 semantics=raw > >> > > > field_type=41234:100* > >> > > > > >> > > > *pmacct.conf* > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > *daemonize: false interface: wlp1s0 !pcap_interfaces_map: > >> > > > pcap_interfaces.map aggregate_primitives: primitives.lst > >> aggregate: > >> > > > src_host, dst_host, src_port, dst_port, proto, tos, dummy_byte > >> plugins: > >> > > > nfprobe, print nfprobe_version: 10 nfprobe_engine: 100 > >> > > > nfprobe_receiver: 192.168.1.6:2100 <http://192.168.1.6:2100> > >> > > > !nfprobe_receiver: 10.40.6.6:16367 <http://10.40.6.6:16367>* > >> > > > > >> > > > > >> > > > *nfacct.conf* > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > *daemonize: false nfacctd_ip: 192.168.1.6 nfacctd_port: 2100 > >> > > > aggregate_primitives: primitives.lst aggregate: src_host, > >> dst_host, > >> > > > src_port, dst_port, proto, tos, dummy_byte plugins: print > >> > > > nfacctd_disable_checks: true* > >> > > > > >> > > > $ sudo /usr/local/sbin/pmacctd -f pmacct.conf > >> > > > INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd > >> > > > 1.7.3-git (20181217-00) > >> > > > INFO ( default/core ): '--enable-l2' '--enable-64bit' > >> > > > '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' > >> > > > '--enable-st-bins' > >> > > > INFO ( default/core ): Reading configuration file > >> > > > '/home/certes-rajesh/pmacct/pmacct/pmacct.conf'. > >> > > > INFO ( default/core ): [primitives.lst] (re)loading map. > >> > > > INFO ( default/core ): [primitives.lst] map successfully (re)loaded. > >> > > > INFO ( default_nfprobe/nfprobe ): NetFlow probe plugin is originally > >> > > based > >> > > > on softflowd 0.9.7 software, Copyright 2002 Damien Miller < > >> > > [email protected]> > >> > > > All rights reserved. > >> > > > INFO ( default_nfprobe/nfprobe ): TCP timeout: 3600s > >> > > > INFO ( default_nfprobe/nfprobe ): TCP post-RST timeout: 120s > >> > > > INFO ( default_nfprobe/nfprobe ): TCP post-FIN timeout: 300s > >> > > > INFO ( default_nfprobe/nfprobe ): UDP timeout: 300s > >> > > > INFO ( default_nfprobe/nfprobe ): ICMP timeout: 300s > >> > > > INFO ( default_nfprobe/nfprobe ): General timeout: 3600s > >> > > > INFO ( default_nfprobe/nfprobe ): Maximum lifetime: 604800s > >> > > > INFO ( default_nfprobe/nfprobe ): Expiry interval: 60s > >> > > > INFO ( default_nfprobe/nfprobe ): Exporting flows to > >> [172.24.1.219]:2100 > >> > > > INFO ( default_print/print ): cache entries=16411 base cache > >> > > > memory=54878384 bytes > >> > > > WARN ( default_print/print ): no print_output_file and no > >> > > > print_output_lock_file defined. > >> > > > INFO ( default/core ): [enp0s31f6,0] link type is: 1 > >> > > > INFO ( default_print/print ): *** Purging cache - START (PID: 4301) > >> *** > >> > > > SRC_IP DST_IP SRC_PORT > >> > > > DST_PORT PROTOCOL TOS *dummy_byte* udp_len PACKETS > >> > > > BYTES > >> > > > 172.24.1.197 239.255.255.250 56940 > >> > > > 1900 udp 0 *08-00* 180 > >> > > > 4 800 > >> > > > fe80::3436:5d8f:abd7:6f0c f2::fb 5353 > >> > > > 5353 udp 0 *86-DD* > >> > > > 48 3 264 > >> > > > > >> > > > > >> > > > $ sudo nfacctd -f nfacct.conf > >> > > > INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.7.3-git > >> > > > (20181217-00) > >> > > > INFO ( default/core ): '--enable-l2' '--enable-64bit' > >> > > > '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' > >> > > > '--enable-st-bins' > >> > > > INFO ( default/core ): Reading configuration file > >> > > > '/home/certes-rajesh/pmacct/pmacct/nfacct.conf'. > >> > > > INFO ( default/core ): [primitives.lst] (re)loading map. > >> > > > INFO ( default/core ): [primitives.lst] map successfully (re)loaded. > >> > > > INFO ( default/core ): waiting for NetFlow/IPFIX data on > >> > > 172.24.1.219:2100 > >> > > > INFO ( default_print/print ): cache entries=16411 base cache > >> > > > memory=54878384 bytes > >> > > > WARN ( default_print/print ): no print_output_file and no > >> > > > print_output_lock_file defined. > >> > > > INFO ( default_print/print ): *** Purging cache - START (PID: 4356) > >> *** > >> > > > INFO ( default_print/print ): *** Purging cache - END (PID: 4356, > >> QN: > >> > > 0/0, > >> > > > ET: X) *** > >> > > > INFO ( default_print/print ): *** Purging cache - START (PID: 4379) > >> *** > >> > > > INFO ( default_print/print ): *** Purging cache - END (PID: 4379, > >> QN: > >> > > 0/0, > >> > > > ET: X) *** > >> > > > INFO ( default_print/print ): *** Purging cache - START (PID: 4410) > >> *** > >> > > > INFO ( default_print/print ): *** Purging cache - END (PID: 4410, > >> QN: > >> > > 0/0, > >> > > > ET: X) *** > >> > > > INFO ( default_print/print ): *** Purging cache - START (PID: 4443) > >> *** > >> > > > SRC_IP DST_IP SRC_PORT > >> DST_PORT > >> > > > PROTOCOL TOS *dummy_byte* PACKETS BYTES > >> > > > 172.24.1.197 239.255.255.250 56940 > >> > > > 1900 udp 0 *30-38* > >> > > > 4 800 > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > On Mon, Dec 17, 2018 at 6:47 AM Paolo Lucente <[email protected]> > >> wrote: > >> > > > > >> > > > > > >> > > > > Hi Rajesh, > >> > > > > > >> > > > > Thanks for pointing this out. I've committed some code to unlock > >> > > > > field_type also for uacctd/pmacctd daemons precisely for the use > >> case > >> > > > > you mentioned. Here the details: > >> > > > > > >> > > > > > >> > > > > > >> > > > >> https://github.com/pmacct/pmacct/commit/87ebf3a9f907c331f752c96a76ea247e77f99107 > >> > > > > > >> > > > > You can back port this patch to latest stable release or use > >> master > >> > > > > code. Keep me posted if it works for you - it did work for me in > >> lab > >> > > > > using your config as a base. > >> > > > > > >> > > > > One recommendation: use IPFIX instead of NetFlow v9 if possible. > >> IPFIX > >> > > > > allows to define the field type as <PEN>:<field_type>, where > >> pmacct PEN > >> > > > > is documented here: > >> > > > > > >> > > > > https://github.com/pmacct/pmacct/blob/master/docs/IPFIX > >> > > > > > >> > > > > So you could use, say, 43874:100 as field type instead of > >> squatting the > >> > > > > public code points. > >> > > > > > >> > > > > Paolo > >> > > > > > >> > > > > On Sat, Dec 15, 2018 at 12:04:54AM +0530, RAJESH KUMAR S.R wrote: > >> > > > > > Hi, > >> > > > > > > >> > > > > > I need some understanding in exporting the custom defined > >> primitives > >> > > in > >> > > > > > netflow v9 messages, if that is possible, as I want to define > >> custom > >> > > > > fields > >> > > > > > and send out to netflow collector and visualize using graphs > >> (if the > >> > > > > > collector supports custom templates) > >> > > > > > > >> > > > > > As a first step, I am trying to use the custom aggregate > >> primitive > >> > > used > >> > > > > in > >> > > > > > examples/primitives.lst.example. > >> > > > > > > >> > > > > > " Defines a primitive called 'udp_len': base pointer is set to > >> the > >> > > UDP > >> > > > > > header > >> > > > > > (l4:17) plus 4 bytes offset, reads for 2 byte and will present > >> it as > >> > > > > > unsigned > >> > > > > > int. > >> > > > > > > >> > > > > > name=udp_len packet_ptr=l4:17+4 len=2 semantics=u_int > >> > > > > > " > >> > > > > > > >> > > > > > I used to classify flows after defining "udp_len" as mentioned > >> above. > >> > > > > > My conf file for pmacctd is > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > *" daemonize:false interface: wlp1s0 aggregate_primitives: > >> > > > > > primitives.lst aggregate: etype, proto, src_host, dst_host, > >> > > src_port, > >> > > > > > dst_port, udp_len plugins: nfprobe, print nfprobe_receiver: > >> > > > > > 172.24.1.123:9996 <http://172.24.1.123:9996> > >> nfprobe_version: 9* > >> > > > > > *"* > >> > > > > > My primitives.lst file defines custom primitive as follows > >> > > > > > > >> > > > > > *"name=udp_len packet_ptr=l4:17+4 len=2 > >> semantics=u_int"* > >> > > > > > > >> > > > > > When I run the pmacct "sudo pmacctd -f pmacct.conf", I'm able > >> to see > >> > > the > >> > > > > > flows that has udp_len column displayed in the console using > >> print > >> > > > > plugin. > >> > > > > > > >> > > > > > Output of > >> > > > > > "sudo pmacctd -f pmacct.conf" > >> > > > > > > >> > > > > > INFO ( default/core ): Promiscuous Mode Accounting Daemon, > >> pmacctd > >> > > > > > 1.7.2-git (20180701-01) > >> > > > > > INFO ( default/core ): '--enable-l2' '--enable-ipv6' > >> > > '--enable-64bit' > >> > > > > > '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' > >> > > > > > '--enable-st-bins' > >> > > > > > INFO ( default/core ): Reading configuration file > >> > > > > > '/home/certes-rajesh/pmacct/pmacct/pmacct.conf'. > >> > > > > > INFO ( default/core ): [primitives.lst] (re)loading map. > >> > > > > > INFO ( default/core ): [primitives.lst] map successfully > >> (re)loaded. > >> > > > > > INFO ( default_nfprobe/nfprobe ): NetFlow probe plugin is > >> originally > >> > > > > based > >> > > > > > on softflowd 0.9.7 software, Copyright 2002 Damien Miller < > >> > > > > [email protected]> > >> > > > > > All rights reserved. > >> > > > > > INFO ( default_nfprobe/nfprobe ): TCP timeout: 3600s > >> > > > > > INFO ( default_nfprobe/nfprobe ): TCP post-RST timeout: 120s > >> > > > > > INFO ( default_nfprobe/nfprobe ): TCP post-FIN timeout: 300s > >> > > > > > INFO ( default_nfprobe/nfprobe ): UDP timeout: 300s > >> > > > > > INFO ( default_nfprobe/nfprobe ): ICMP timeout: 300s > >> > > > > > INFO ( default_nfprobe/nfprobe ): General timeout: 3600s > >> > > > > > INFO ( default_nfprobe/nfprobe ): Maximum lifetime: 604800s > >> > > > > > INFO ( default_nfprobe/nfprobe ): Expiry interval: 60s > >> > > > > > INFO ( default_nfprobe/nfprobe ): Exporting flows to > >> > > [192.168.122.1]:9996 > >> > > > > > *ERROR ( default_nfprobe/nfprobe ): custom primitive 'udp_len' > >> has > >> > > null > >> > > > > > field_type* > >> > > > > > INFO ( default_print/print ): cache entries=16411 base cache > >> > > > > > memory=54878384 bytes > >> > > > > > WARN ( default_print/print ): no print_output_file and no > >> > > > > > print_output_lock_file defined. > >> > > > > > INFO ( default/core ): [wlp1s0,0] link type is: 1 > >> > > > > > *WARN ( default/core ): connection lost to > >> 'default_nfprobe-nfprobe'; > >> > > > > > closing connection.* > >> > > > > > INFO ( default_print/print ): *** Purging cache - START (PID: > >> 2837) > >> > > *** > >> > > > > > ETYPE SRC_IP > >> > > > > > DST_IP SRC_PORT > >> DST_PORT > >> > > > > > PROTOCOL udp_len PACKETS BYTES > >> > > > > > 86dd fd50:1d9:a341:f100:8ae:86f3:123d:3654 > >> > > > > > ff02::fb 5353 5353 > >> > > > > > udp 41 3 243 > >> > > > > > ....... > >> > > > > > > >> > > > > > When I try to give a dummy field type, it throws > >> > > > > > "WARN ( default/core ): [primitives.lst] field_type is only > >> > > supported in > >> > > > > > nfacctd.". > >> > > > > > > >> > > > > > I need help in figuring out whether I'm doing the right thing > >> for > >> > > > > exporting > >> > > > > > custom fields as part netflow messages as I will need to send > >> out > >> > > more > >> > > > > > custom fields that are read from the packet. > >> > > > > > >> > > > > > _______________________________________________ > >> > > > > > pmacct-discussion mailing list > >> > > > > > http://www.pmacct.net/#mailinglists > >> > > > > > >> > > > > > >> > > > > _______________________________________________ > >> > > > > pmacct-discussion mailing list > >> > > > > http://www.pmacct.net/#mailinglists > >> > > > > > >> > > > >> > >> > >> > >> _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
