Hi Eddi,
Thanks for the kind words and the interesting email. Any chance you can
send me a brief sample of such data via unicast email so to be able to
reproduce things at my end?
Thanks,
Paolo
On Wed, Nov 01, 2017 at 06:24:26PM +0200, edd! wrote:
> Hi,
>
> After wasting a couple of months on testing several IPFIX collectors that
> can absorb the load I have, I finally found pmacct!
> nfacctd is the only software I could find that is holding rock solid. No
> rocket science here, compiled, fine-tuned the buffers and it is sucking it
> all and asking for more.
>
> My requirements: Log each and every visited web site (http/https) on their
> standard ports.
> Equipment: Procera boxes sending netflow v10
> Problem: The defined custom fields rarely include data on the output
>
> Setup details
> --------------------
>
> # conf file:
> nfacctd_port: 9996
> nfacctd_allow_file: /usr/local/etc/nfacctd.allow
> daemonize: true
> pidfile: /var/run/nfacctd
> plugins: print[web]
> plugin_pipe_size: 81920000
> plugin_buffer_size: 8192
> logfile: /var/log/nfacctd.log
> print_output_file_append: true
> print_output_file[web]: /data/live/procera1.log
> timestamps_secs: true
> timestamps_since_epoch: false
> print_output[web]: csv
> print_output_separator[web]: ,
> print_num_protos: true
> nfacctd_time_secs: false
> nfacctd_time_new: false
> nfacctd_templates_file: /tmp/procera1.tmpl
> nfacctd_disable_checks: true
> pre_tag_map: /usr/local/etc/nfacctd-pretag.map
> pre_tag_filter[web]: 80443
> aggregate_primitives: /usr/local/etc/nfacctd-primitives.lst
> aggregate[web]: timestamp_start, timestamp_end, proto, src_host, src_port,
> dst_host, dst_port, proc_svr_host, proc_http_url
>
> # nfacctd-primitives.lst:
> name=proc_svr_host field_type=15397:18 len=655535
> semantics=string
> name=proc_http_url field_type=15397:22 len=655535
> semantics=string
>
> # nfacctd-pretag.map:
> set_tag=80443 filter='dst port 80'
> set_tag=80443 filter='dst port 443'
>
> # template fields as per the nfacctd_templates_file:
> {"type": 0, "otpl": {"off": 0, "len": 4, "tpl_len": 4, "tpl_index": 12}}
> {"type": 0, "otpl": {"off": 4, "len": 2, "tpl_len": 2, "tpl_index": 11}}
> {"type": 0, "otpl": {"off": 6, "len": 4, "tpl_len": 4, "tpl_index": 151}}
> {"type": 0, "otpl": {"off": 10, "len": 4, "tpl_len": 4, "tpl_index": 150}}
> {"type": 1, "utpl": {"pen": 15397, "type": 22, "off": 14, "len": 0,
> "tpl_len": 65535, "repeat_id": 0, "ie_idx": 0}}
> {"type": 1, "utpl": {"pen": 15397, "type": 18, "off": 0, "len": 0,
> "tpl_len": 65535, "repeat_id": 0, "ie_idx": 0}}
> {"type": 0, "otpl": {"off": 0, "len": 1, "tpl_len": 1, "tpl_index": 4}}
> {"type": 0, "otpl": {"off": 0, "len": 4, "tpl_len": 4, "tpl_index": 8}}
> {"type": 0, "otpl": {"off": 0, "len": 2, "tpl_len": 2, "tpl_index": 7}}
>
> # log file showing the startup:
> Nov 01 16:55:44 INFO ( default/core ): NetFlow Accounting Daemon, nfacctd
> 1.7.0 (20170924-00+c1)
> Nov 01 16:55:44 INFO ( default/core ): '--enable-jansson' '--enable-l2'
> '--enable-ipv6' '--enable-64bit' '--enable-threads' '--enable-traffic-bins'
> '--enable-bgp-bins' '--enable-bmp-bins' '--enable-st-bins'
> Nov 01 16:55:44 INFO ( default/core ): Reading configuration file
> '/usr/local/etc/nfacctd-procera1.conf'.
> Nov 01 16:55:44 INFO ( default/core ):
> [/usr/local/etc/nfacctd-primitives.lst] (re)loading map.
> Nov 01 16:55:44 INFO ( default/core ):
> [/usr/local/etc/nfacctd-primitives.lst] map successfully (re)loaded.
> Nov 01 16:55:44 INFO ( web/print ): plugin_pipe_size=81920000 bytes
> plugin_buffer_size=8192 bytes
> Nov 01 16:55:44 INFO ( web/print ): ctrl channel: obtained=124928 bytes
> target=80000 bytes
> Nov 01 16:55:44 INFO ( default/core ): [/usr/local/etc/nfacctd-pretag.map]
> (re)loading map.
> Nov 01 16:55:44 INFO ( web/print ): cache entries=16411 base cache
> memory=54878384 bytes
> Nov 01 16:55:44 INFO ( default/core ): [/usr/local/etc/nfacctd-pretag.map]
> map successfully (re)loaded.
> Nov 01 16:55:44 INFO ( default/core ): [/usr/local/etc/nfacctd-pretag.map]
> (re)loading map.
> Nov 01 16:55:44 INFO ( default/core ): [/usr/local/etc/nfacctd-pretag.map]
> map successfully (re)loaded.
> Nov 01 16:55:44 INFO ( default/core ): waiting for NetFlow/IPFIX data on
> x.x.x.x:9996
>
> # netflow template captured and decoded by tshark:
> Cisco NetFlow/IPFIX
> Version: 10
> Length: 68
> Timestamp: Nov 1, 2017 18:08:00.000000000 Middle East Standard Time
> ExportTime: 1509552480
> FlowSequence: 3775142542
> Observation Domain Id: 2879742714
> Set 1 [id=2] (Data Template): 12098
> FlowSet Id: Data Template (V10 [IPFIX]) (2)
> FlowSet Length: 52
> Template (Id = 12098, Count = 9)
> Template Id: 12098
> Field Count: 9
> Field (1/9): IP_DST_ADDR
> 0... .... .... .... = Pen provided: No
> .000 0000 0000 1100 = Type: IP_DST_ADDR (12)
> Length: 4
> Field (2/9): L4_DST_PORT
> 0... .... .... .... = Pen provided: No
> .000 0000 0000 1011 = Type: L4_DST_PORT (11)
> Length: 2
> Field (3/9): flowEndSeconds
> 0... .... .... .... = Pen provided: No
> .000 0000 1001 0111 = Type: flowEndSeconds (151)
> Length: 4
> Field (4/9): flowStartSeconds
> 0... .... .... .... = Pen provided: No
> .000 0000 1001 0110 = Type: flowStartSeconds (150)
> Length: 4
> Field (5/9): 22 [pen: Netintact AB]
> 1... .... .... .... = Pen provided: Yes
> .000 0000 0001 0110 = Type: 22 [pen: Netintact AB]
> Length: 65535 [i.e.: "Variable Length"]
> PEN: Netintact AB (15397)
> Field (6/9): 18 [pen: Netintact AB]
> 1... .... .... .... = Pen provided: Yes
> .000 0000 0001 0010 = Type: 18 [pen: Netintact AB]
> Length: 65535 [i.e.: "Variable Length"]
> PEN: Netintact AB (15397)
> Field (7/9): PROTOCOL
> 0... .... .... .... = Pen provided: No
> .000 0000 0000 0100 = Type: PROTOCOL (4)
> Length: 1
> Field (8/9): IP_SRC_ADDR
> 0... .... .... .... = Pen provided: No
> .000 0000 0000 1000 = Type: IP_SRC_ADDR (8)
> Length: 4
> Field (9/9): L4_SRC_PORT
> 0... .... .... .... = Pen provided: No
> .000 0000 0000 0111 = Type: L4_SRC_PORT (7)
> Length: 2
>
> # Sample output:
> 100.66.99.63,31.13.86.34,59295,443,6,2017-11-01 18:10:00.0,2017-11-01
> 18:15:00.0,,,0,0
> 100.74.114.163,13.112.60.248,10458,443,6,2017-11-01 18:10:00.0,2017-11-01
> 18:15:00.0,,,0,0
> 100.66.86.242,216.58.205.97,64124,443,6,2017-11-01 18:12:36.0,2017-11-01
> 18:15:00.0,,,0,0
> 100.102.182.15,31.13.86.51,36051,443,6,2017-11-01 18:10:00.0,2017-11-01
> 18:15:00.0,,,0,0
> 100.102.182.15,31.13.86.51,36051,443,6,2017-11-01 18:10:21.0,2017-11-01
> 18:12:39.0,,,0,0
> 100.66.150.251,169.45.214.236,54964,443,6,2017-11-01 18:10:00.0,2017-11-01
> 18:15:00.0,,,0,0
> 100.101.179.68,40.101.48.88,63261,443,6,2017-11-01 18:12:39.0,2017-11-01
> 18:12:39.0,,,0,0
> 100.74.121.140,31.13.86.8,63145,443,6,2017-11-01 18:12:39.0,2017-11-01
> 18:12:39.0,,,0,0
> 100.74.117.42,216.58.205.78,54123,80,6,2017-11-01 18:11:54.0,2017-11-01
> 18:12:39.0,,,0,0
> 100.101.233.28,31.13.86.51,59636,443,6,2017-11-01 18:10:00.0,2017-11-01
> 18:15:00.0,,,0,0
> 100.74.103.34,185.176.144.17,24577,443,6,2017-11-01 18:10:14.0,2017-11-01
> 18:12:39.0,,,0,0
> 100.74.107.25,157.240.1.23,49299,443,6,2017-11-01 18:12:08.0,2017-11-01
> 18:15:00.0,,,0,0
> 100.101.233.28,31.13.86.51,59636,443,6,2017-11-01 18:11:15.0,2017-11-01
> 18:12:39.0,,,0,0
> 100.101.106.136,185.54.60.160,49850,443,6,2017-11-01 18:12:31.0,2017-11-01
> 18:12:39.0,,,0,0
> 100.74.118.76,34.235.42.103,14386,443,6,2017-11-01 18:12:38.0,2017-11-01
> 18:12:39.0,,,0,0
> 100.74.121.140,31.13.86.8,56464,443,6,2017-11-01 18:12:37.0,2017-11-01
> 18:15:00.0,,,0,0
> 100.100.87.90,101.167.166.38,41029,80,6,2017-11-01 18:12:14.0,2017-11-01
> 18:12:39.0,,,0,0
> 100.74.114.244,31.13.86.52,64799,443,6,2017-11-01 18:11:30.0,2017-11-01
> 18:15:00.0,,,0,0
>
> sometimes, but rarely, I get data in proc_http_url field only:
> 100.66.143.216,17.253.49.204,50113,80,6,2017-11-01 18:10:00.0,2017-11-01
> 18:15:00.0,,http://appldnld.apple.com/ios11.1seed/*** (rest is removed for
> privacy)
>
> I verified that the data is being sent from Procera using wireshark.
>
> Please help
>
> Thank you,
> Eddi
> _______________________________________________
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
