Hi Rajesh,

Nice labels worked for you.

Clarify me one thing: the output you did show, with zeroed peer_src_ip
(and exporteripv4address, engineid, enginetype), is the one from
pmacctd, right? Not nfacctd. In that case the output is expected. 
In fact in nfacctd it should not be possible to get a null peer_src_ip
(which is nothing else than the address returned by a recv() on the
socket); setting nfprobe_source_ip is needed only in cases in which
multiple interfaces could be selected for output or for settin field
type #130 for, for example, NAT traversal scenarios.

Paolo

On Sat, Jan 05, 2019 at 12:29:35AM +0530, RAJESH KUMAR S.R wrote:
> Hi Paolo,
> 
> I was able to set labels and export as strings for different pmacct
> instances that was listening on different interfaces. Thanks for the
> suggestion.
> 
> I need a help regarding exporting Exporter's IP as part of flow records.
> Based on old pmacct discussions, I'm using "peer_src_ip and
> exporterIPv4Address" as primitives but both they seem to come as 0 and
> 0.0.0.0 in flows.
> I tried setting the "nfprobe_source_ip: 172.30.130.99", but it goes in
> separate flow as ExporterAddress: 172.30.130.99, but I need the
> "exporterIpv4Address" to be set to correct value in all flow messages, will
> pmacct collect the interface ip and populate in "exporterIpv4Address"
> field. Also, I'm not sure of how to get the engineid working, that also
> seem to go as 0 in flows.
> 
> pmacctd Output:
> 
> *SRC_MAC   DST_MAC             PEER_SRC_IP*
> 50:01:d9:a3:41:f1  f8:59:71:73:94:4d      ::
> * SRC_IP               DST_IP        SRC_PORT  DST_PORT  PROTOCOL   *
> 52.229.174.94  192.168.1.9     443              43238          tcp
> 
> *exporteripv4address    engineid  enginetype  PACKETS     BYTES*
>  0.0.0.0                                         0         0
> 2                     629
> 
> 
> I have the following pmacct conf file
> 
> *primitives*
> 
> name=engineType field_type=0:38 len=1 semantics=u_int
> name=engineId field_type=0:39 len=1 semantics=u_int
> name=exporterIPv4Address field_type=130 len=4 semantics=ip
> 
> *pmacct.conf*
> "
> debug: true
>    daemonize: false
>    pre_tag_map: ipfix_pretag.map
> 
>    nfprobe_engine: 100
>    nfprobe_version: 10
>    nfprobe_source_ip: 172.30.130.99
>    aggregate_primitives: ipfix_primitives.lst
>    plugins: nfprobe, print
>    interface: wlp1s0
>    aggregate: src_mac, dst_mac, src_host, dst_host, src_port, dst_port,
> proto, peer_src_ip, exporterIPv4Address, engineId, engineType
>    nfprobe_receiver: 10.40.6.6:17058
> "
> 
> 
> 
> 
> On Wed, Dec 26, 2018 at 12:44 PM RAJESH KUMAR S.R <rajuuu1...@gmail.com>
> wrote:
> 
> > Hi Paolo,
> >
> > Thanks for the fix and suggestion. I'll try tag and label primitives and
> > see if they match my requirements.
> >
> > On Tue, Dec 25, 2018 at 10:49 PM Paolo Lucente <pa...@pmacct.net> wrote:
> >
> >>
> >> Hi Rajesh,
> >>
> >> You are right, there was a bug in the serialize_bin() func that was
> >> making it work good only for the first byte. This is now resolved:
> >>
> >>
> >> https://github.com/pmacct/pmacct/commit/1076ff3529f439133357176e4c1260cfcdcef56e
> >>
> >> I've read your question about metadata and was wondering: would tags
> >> (tag, tag2 primitive) or labels (label primitive) defined via a
> >> pre_tag_map be a solution for you? You could do a proof-of-concept
> >> locally, ie. like you were doing already with the print plugin, and if
> >> meeting your requirements we can move onto the nfprobe part; i expect
> >> tags to work no problem; labels should work but may require a bit more
> >> testing.
> >>
> >> Paolo
> >>
> >> On Mon, Dec 24, 2018 at 02:21:18PM +0530, RAJESH KUMAR S.R wrote:
> >> > Hi Paolo,
> >> >
> >> > Thanks for the fix. I tested with pmacctd and nfacctd and I see that
> >> when I
> >> > read 1 byte of raw data, it prints correct on both sides
> >> > but by while reading more bytes, the first byte is alone correct on
> >> nfacct
> >> > side. Not sure I'm testing correctly, but thanks for the fix.
> >> >
> >> > pmacctd side
> >> > dummy_byte  PACKETS               BYTES
> >> > *08-00-45*    535                   124114
> >> > *86-DD-60*    10                    861
> >> >
> >> > On nfacctd side, I'm getting only the first byte correct
> >> > *08-00-00*    535                 124114
> >> > *86-00-00*    10                   861
> >> >
> >> >
> >> > I need one more suggestion/help regarding custom primitives.
> >> > Right now I can read a set of bytes from packet and send it as custom
> >> > primitives using the
> >> > recommended PEN id which I can see in netflow message in wireshark
> >> > also(attached the wireshark image)
> >> >
> >> > I have requirement where I need to send some *metadata*/context as part
> >> as
> >> > netflow records.
> >> > The metadata/context won't be available in the packet. Will I be able to
> >> > define and use
> >> > custom primitives for sending those metadata, (say one or more
> >> > context/metadata in pmacct conf file, similar to ifindex)
> >> > I'm reading packet from different interfaces, where I know the context
> >> of
> >> > packets that are arriving on each interface
> >> > I need to send that context as one of the netflow field. The only way I
> >> > know to do now is to write that context as part of the packet and read
> >> as
> >> > custom template.
> >> > So, basically I'm trying to figure out other possible ways of defining
> >> > custom primitives
> >> > & fieldtypes and using it without reading from packet.
> >> > It would be very helpful it any alternate way is feasible.
> >> >
> >> > On Thu, Dec 20, 2018 at 6:23 AM Paolo Lucente <pa...@pmacct.net> wrote:
> >> >
> >> > >
> >> > > Hi Rajesh,
> >> > >
> >> > > Thanks for bringing this further point up. While all was working with
> >> > > hex and u_int semantics, there was an issue with the raw one - which
> >> was
> >> > > important to fix since it's a very generic transport for any kind of
> >> > > info. Here is the commit:
> >> > >
> >> > >
> >> > >
> >> https://github.com/pmacct/pmacct/commit/e1206a530af64c46748b5f891a9034dec89e9b7a
> >> > >
> >> > > Let me know if this works for you.
> >> > >
> >> > > Paolo
> >> > >
> >> > > On Mon, Dec 17, 2018 at 10:25:27PM +0530, RAJESH KUMAR S.R wrote:
> >> > > > Hi Paolo,
> >> > > >
> >> > > > Thanks for the help and suggestion ,
> >> > > > Custom primitives works for me with the latest code:) . I'm able to
> >> read
> >> > > a
> >> > > > field from packet and export as custom template and I can see those
> >> > > > templates in netflow messages also.
> >> > > > I'll continue with viewing the flows in a collector, I'm using
> >> Elastiflow
> >> > > > to view reports.
> >> > > >
> >> > > >
> >> > > > I have a small doubt. I was reading a field of each packet and map
> >> it to
> >> > > a
> >> > > > custom template.(just trying to read ethertype for testing the
> >> custom
> >> > > > templates eventhough etype is available)
> >> > > > The value shown in pmacct seems to be right.  I tried verifying
> >> this data
> >> > > > with nfacctd, but the value shown in nfacctd seems to be different.
> >> I
> >> > > have
> >> > > > pasted the conf files and output.
> >> > > >
> >> > > > *Conf files:*
> >> > > > *primitives.lst file*
> >> > > > *name=dummy_byte packet_ptr=packet:+12 len=2 semantics=raw
> >> > > > field_type=41234:100*
> >> > > >
> >> > > > *pmacct.conf*
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > *daemonize: false   interface: wlp1s0   !pcap_interfaces_map:
> >> > > > pcap_interfaces.map   aggregate_primitives: primitives.lst
> >>  aggregate:
> >> > > > src_host, dst_host, src_port, dst_port, proto, tos, dummy_byte
> >>  plugins:
> >> > > > nfprobe, print   nfprobe_version: 10   nfprobe_engine: 100
> >> > > > nfprobe_receiver: 192.168.1.6:2100 <http://192.168.1.6:2100>
> >> > > > !nfprobe_receiver: 10.40.6.6:16367 <http://10.40.6.6:16367>*
> >> > > >
> >> > > >
> >> > > > *nfacct.conf*
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > *daemonize: false   nfacctd_ip: 192.168.1.6   nfacctd_port: 2100
> >> > > > aggregate_primitives: primitives.lst   aggregate: src_host,
> >> dst_host,
> >> > > > src_port, dst_port, proto, tos, dummy_byte   plugins: print
> >> > > > nfacctd_disable_checks: true*
> >> > > >
> >> > > > $   sudo /usr/local/sbin/pmacctd -f pmacct.conf
> >> > > > INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd
> >> > > > 1.7.3-git (20181217-00)
> >> > > > INFO ( default/core ):  '--enable-l2' '--enable-64bit'
> >> > > > '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
> >> > > > '--enable-st-bins'
> >> > > > INFO ( default/core ): Reading configuration file
> >> > > > '/home/certes-rajesh/pmacct/pmacct/pmacct.conf'.
> >> > > > INFO ( default/core ): [primitives.lst] (re)loading map.
> >> > > > INFO ( default/core ): [primitives.lst] map successfully (re)loaded.
> >> > > > INFO ( default_nfprobe/nfprobe ): NetFlow probe plugin is originally
> >> > > based
> >> > > > on softflowd 0.9.7 software, Copyright 2002 Damien Miller <
> >> > > d...@mindrot.org>
> >> > > > All rights reserved.
> >> > > > INFO ( default_nfprobe/nfprobe ):           TCP timeout: 3600s
> >> > > > INFO ( default_nfprobe/nfprobe ):  TCP post-RST timeout: 120s
> >> > > > INFO ( default_nfprobe/nfprobe ):  TCP post-FIN timeout: 300s
> >> > > > INFO ( default_nfprobe/nfprobe ):           UDP timeout: 300s
> >> > > > INFO ( default_nfprobe/nfprobe ):          ICMP timeout: 300s
> >> > > > INFO ( default_nfprobe/nfprobe ):       General timeout: 3600s
> >> > > > INFO ( default_nfprobe/nfprobe ):      Maximum lifetime: 604800s
> >> > > > INFO ( default_nfprobe/nfprobe ):       Expiry interval: 60s
> >> > > > INFO ( default_nfprobe/nfprobe ): Exporting flows to
> >> [172.24.1.219]:2100
> >> > > > INFO ( default_print/print ): cache entries=16411 base cache
> >> > > > memory=54878384 bytes
> >> > > > WARN ( default_print/print ): no print_output_file and no
> >> > > > print_output_lock_file defined.
> >> > > > INFO ( default/core ): [enp0s31f6,0] link type is: 1
> >> > > > INFO ( default_print/print ): *** Purging cache - START (PID: 4301)
> >> ***
> >> > > > SRC_IP                         DST_IP                       SRC_PORT
> >> > > > DST_PORT  PROTOCOL    TOS    *dummy_byte*  udp_len  PACKETS
> >> > > > BYTES
> >> > > > 172.24.1.197            239.255.255.250                 56940
> >> > > > 1900               udp                 0       *08-00*           180
> >> > > > 4                     800
> >> > > > fe80::3436:5d8f:abd7:6f0c      f2::fb              5353
> >> > > > 5353         udp                 0            *86-DD*
> >> > > > 48       3                     264
> >> > > >
> >> > > >
> >> > > > $   sudo nfacctd  -f nfacct.conf
> >> > > > INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.7.3-git
> >> > > > (20181217-00)
> >> > > > INFO ( default/core ):  '--enable-l2' '--enable-64bit'
> >> > > > '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
> >> > > > '--enable-st-bins'
> >> > > > INFO ( default/core ): Reading configuration file
> >> > > > '/home/certes-rajesh/pmacct/pmacct/nfacct.conf'.
> >> > > > INFO ( default/core ): [primitives.lst] (re)loading map.
> >> > > > INFO ( default/core ): [primitives.lst] map successfully (re)loaded.
> >> > > > INFO ( default/core ): waiting for NetFlow/IPFIX data on
> >> > > 172.24.1.219:2100
> >> > > > INFO ( default_print/print ): cache entries=16411 base cache
> >> > > > memory=54878384 bytes
> >> > > > WARN ( default_print/print ): no print_output_file and no
> >> > > > print_output_lock_file defined.
> >> > > > INFO ( default_print/print ): *** Purging cache - START (PID: 4356)
> >> ***
> >> > > > INFO ( default_print/print ): *** Purging cache - END (PID: 4356,
> >> QN:
> >> > > 0/0,
> >> > > > ET: X) ***
> >> > > > INFO ( default_print/print ): *** Purging cache - START (PID: 4379)
> >> ***
> >> > > > INFO ( default_print/print ): *** Purging cache - END (PID: 4379,
> >> QN:
> >> > > 0/0,
> >> > > > ET: X) ***
> >> > > > INFO ( default_print/print ): *** Purging cache - START (PID: 4410)
> >> ***
> >> > > > INFO ( default_print/print ): *** Purging cache - END (PID: 4410,
> >> QN:
> >> > > 0/0,
> >> > > > ET: X) ***
> >> > > > INFO ( default_print/print ): *** Purging cache - START (PID: 4443)
> >> ***
> >> > > > SRC_IP                          DST_IP                 SRC_PORT
> >> DST_PORT
> >> > > > PROTOCOL    TOS    *dummy_byte*  PACKETS               BYTES
> >> > > > 172.24.1.197          239.255.255.250            56940
> >> > > > 1900          udp                 0            *30-38*
> >> > > > 4                     800
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Mon, Dec 17, 2018 at 6:47 AM Paolo Lucente <pa...@pmacct.net>
> >> wrote:
> >> > > >
> >> > > > >
> >> > > > > Hi Rajesh,
> >> > > > >
> >> > > > > Thanks for pointing this out. I've committed some code to unlock
> >> > > > > field_type also for uacctd/pmacctd daemons precisely for the use
> >> case
> >> > > > > you mentioned. Here the details:
> >> > > > >
> >> > > > >
> >> > > > >
> >> > >
> >> https://github.com/pmacct/pmacct/commit/87ebf3a9f907c331f752c96a76ea247e77f99107
> >> > > > >
> >> > > > > You can back port this patch to latest stable release or use
> >> master
> >> > > > > code. Keep me posted if it works for you - it did work for me in
> >> lab
> >> > > > > using your config as a base.
> >> > > > >
> >> > > > > One recommendation: use IPFIX instead of NetFlow v9 if possible.
> >> IPFIX
> >> > > > > allows to define the field type as <PEN>:<field_type>, where
> >> pmacct PEN
> >> > > > > is documented here:
> >> > > > >
> >> > > > > https://github.com/pmacct/pmacct/blob/master/docs/IPFIX
> >> > > > >
> >> > > > > So you could use, say, 43874:100 as field type instead of
> >> squatting the
> >> > > > > public code points.
> >> > > > >
> >> > > > > Paolo
> >> > > > >
> >> > > > > On Sat, Dec 15, 2018 at 12:04:54AM +0530, RAJESH KUMAR S.R wrote:
> >> > > > > > Hi,
> >> > > > > >
> >> > > > > > I need some understanding in exporting the custom defined
> >> primitives
> >> > > in
> >> > > > > > netflow v9 messages, if that is possible, as I want to define
> >> custom
> >> > > > > fields
> >> > > > > > and send out to netflow collector and visualize using graphs
> >> (if the
> >> > > > > > collector supports custom templates)
> >> > > > > >
> >> > > > > > As a first step, I am trying to use the custom aggregate
> >> primitive
> >> > > used
> >> > > > > in
> >> > > > > > examples/primitives.lst.example.
> >> > > > > >
> >> > > > > > " Defines a primitive called 'udp_len': base pointer is set to
> >> the
> >> > > UDP
> >> > > > > > header
> >> > > > > >  (l4:17) plus 4 bytes offset, reads for 2 byte and will present
> >> it as
> >> > > > > > unsigned
> >> > > > > >  int.
> >> > > > > >
> >> > > > > > name=udp_len    packet_ptr=l4:17+4      len=2   semantics=u_int
> >> > > > > > "
> >> > > > > >
> >> > > > > > I used to classify flows after defining "udp_len" as mentioned
> >> above.
> >> > > > > > My conf file for pmacctd is
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > *"   daemonize:false   interface: wlp1s0   aggregate_primitives:
> >> > > > > > primitives.lst   aggregate: etype, proto, src_host, dst_host,
> >> > > src_port,
> >> > > > > > dst_port, udp_len   plugins: nfprobe, print   nfprobe_receiver:
> >> > > > > > 172.24.1.123:9996 <http://172.24.1.123:9996>
> >>  nfprobe_version: 9*
> >> > > > > > *"*
> >> > > > > > My primitives.lst file defines custom primitive as follows
> >> > > > > >
> >> > > > > > *"name=udp_len    packet_ptr=l4:17+4      len=2
> >>  semantics=u_int"*
> >> > > > > >
> >> > > > > > When I run the pmacct "sudo pmacctd -f pmacct.conf", I'm able
> >> to see
> >> > > the
> >> > > > > > flows that has udp_len column displayed in the console using
> >> print
> >> > > > > plugin.
> >> > > > > >
> >> > > > > > Output of
> >> > > > > > "sudo pmacctd -f pmacct.conf"
> >> > > > > >
> >> > > > > > INFO ( default/core ): Promiscuous Mode Accounting Daemon,
> >> pmacctd
> >> > > > > > 1.7.2-git (20180701-01)
> >> > > > > > INFO ( default/core ):  '--enable-l2' '--enable-ipv6'
> >> > > '--enable-64bit'
> >> > > > > > '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
> >> > > > > > '--enable-st-bins'
> >> > > > > > INFO ( default/core ): Reading configuration file
> >> > > > > > '/home/certes-rajesh/pmacct/pmacct/pmacct.conf'.
> >> > > > > > INFO ( default/core ): [primitives.lst] (re)loading map.
> >> > > > > > INFO ( default/core ): [primitives.lst] map successfully
> >> (re)loaded.
> >> > > > > > INFO ( default_nfprobe/nfprobe ): NetFlow probe plugin is
> >> originally
> >> > > > > based
> >> > > > > > on softflowd 0.9.7 software, Copyright 2002 Damien Miller <
> >> > > > > d...@mindrot.org>
> >> > > > > > All rights reserved.
> >> > > > > > INFO ( default_nfprobe/nfprobe ):           TCP timeout: 3600s
> >> > > > > > INFO ( default_nfprobe/nfprobe ):  TCP post-RST timeout: 120s
> >> > > > > > INFO ( default_nfprobe/nfprobe ):  TCP post-FIN timeout: 300s
> >> > > > > > INFO ( default_nfprobe/nfprobe ):           UDP timeout: 300s
> >> > > > > > INFO ( default_nfprobe/nfprobe ):          ICMP timeout: 300s
> >> > > > > > INFO ( default_nfprobe/nfprobe ):       General timeout: 3600s
> >> > > > > > INFO ( default_nfprobe/nfprobe ):      Maximum lifetime: 604800s
> >> > > > > > INFO ( default_nfprobe/nfprobe ):       Expiry interval: 60s
> >> > > > > > INFO ( default_nfprobe/nfprobe ): Exporting flows to
> >> > > [192.168.122.1]:9996
> >> > > > > > *ERROR ( default_nfprobe/nfprobe ): custom primitive 'udp_len'
> >> has
> >> > > null
> >> > > > > > field_type*
> >> > > > > > INFO ( default_print/print ): cache entries=16411 base cache
> >> > > > > > memory=54878384 bytes
> >> > > > > > WARN ( default_print/print ): no print_output_file and no
> >> > > > > > print_output_lock_file defined.
> >> > > > > > INFO ( default/core ): [wlp1s0,0] link type is: 1
> >> > > > > > *WARN ( default/core ): connection lost to
> >> 'default_nfprobe-nfprobe';
> >> > > > > > closing connection.*
> >> > > > > > INFO ( default_print/print ): *** Purging cache - START (PID:
> >> 2837)
> >> > > ***
> >> > > > > > ETYPE  SRC_IP
> >> > > > > > DST_IP                                         SRC_PORT
> >> DST_PORT
> >> > > > > > PROTOCOL    udp_len  PACKETS               BYTES
> >> > > > > > 86dd   fd50:1d9:a341:f100:8ae:86f3:123d:3654
> >> > > > > > ff02::fb                                       5353      5353
> >> > > > > > udp         41       3                     243
> >> > > > > > .......
> >> > > > > >
> >> > > > > > When I try to give a dummy field type, it throws
> >> > > > > > "WARN ( default/core ): [primitives.lst] field_type is only
> >> > > supported in
> >> > > > > > nfacctd.".
> >> > > > > >
> >> > > > > > I need help in figuring out whether I'm doing the right thing
> >> for
> >> > > > > exporting
> >> > > > > > custom fields as part netflow messages as I will need to send
> >> out
> >> > > more
> >> > > > > > custom fields that are read from the packet.
> >> > > > >
> >> > > > > > _______________________________________________
> >> > > > > > pmacct-discussion mailing list
> >> > > > > > http://www.pmacct.net/#mailinglists
> >> > > > >
> >> > > > >
> >> > > > > _______________________________________________
> >> > > > > pmacct-discussion mailing list
> >> > > > > http://www.pmacct.net/#mailinglists
> >> > > > >
> >> > >
> >>
> >>
> >>
> >>

_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Reply via email to