Hi,
After wasting a couple of months on testing several IPFIX collectors that
can absorb the load I have, I finally found pmacct!
nfacctd is the only software I could find that is holding rock solid. No
rocket science here, compiled, fine-tuned the buffers and it is sucking it
all and asking for more.
My requirements: Log each and every visited web site (http/https) on their
standard ports.
Equipment: Procera boxes sending netflow v10
Problem: The defined custom fields rarely include data on the output
Setup details
--------------------
# conf file:
nfacctd_port: 9996
nfacctd_allow_file: /usr/local/etc/nfacctd.allow
daemonize: true
pidfile: /var/run/nfacctd
plugins: print[web]
plugin_pipe_size: 81920000
plugin_buffer_size: 8192
logfile: /var/log/nfacctd.log
print_output_file_append: true
print_output_file[web]: /data/live/procera1.log
timestamps_secs: true
timestamps_since_epoch: false
print_output[web]: csv
print_output_separator[web]: ,
print_num_protos: true
nfacctd_time_secs: false
nfacctd_time_new: false
nfacctd_templates_file: /tmp/procera1.tmpl
nfacctd_disable_checks: true
pre_tag_map: /usr/local/etc/nfacctd-pretag.map
pre_tag_filter[web]: 80443
aggregate_primitives: /usr/local/etc/nfacctd-primitives.lst
aggregate[web]: timestamp_start, timestamp_end, proto, src_host, src_port,
dst_host, dst_port, proc_svr_host, proc_http_url
# nfacctd-primitives.lst:
name=proc_svr_host field_type=15397:18 len=655535
semantics=string
name=proc_http_url field_type=15397:22 len=655535
semantics=string
# nfacctd-pretag.map:
set_tag=80443 filter='dst port 80'
set_tag=80443 filter='dst port 443'
# template fields as per the nfacctd_templates_file:
{"type": 0, "otpl": {"off": 0, "len": 4, "tpl_len": 4, "tpl_index": 12}}
{"type": 0, "otpl": {"off": 4, "len": 2, "tpl_len": 2, "tpl_index": 11}}
{"type": 0, "otpl": {"off": 6, "len": 4, "tpl_len": 4, "tpl_index": 151}}
{"type": 0, "otpl": {"off": 10, "len": 4, "tpl_len": 4, "tpl_index": 150}}
{"type": 1, "utpl": {"pen": 15397, "type": 22, "off": 14, "len": 0,
"tpl_len": 65535, "repeat_id": 0, "ie_idx": 0}}
{"type": 1, "utpl": {"pen": 15397, "type": 18, "off": 0, "len": 0,
"tpl_len": 65535, "repeat_id": 0, "ie_idx": 0}}
{"type": 0, "otpl": {"off": 0, "len": 1, "tpl_len": 1, "tpl_index": 4}}
{"type": 0, "otpl": {"off": 0, "len": 4, "tpl_len": 4, "tpl_index": 8}}
{"type": 0, "otpl": {"off": 0, "len": 2, "tpl_len": 2, "tpl_index": 7}}
# log file showing the startup:
Nov 01 16:55:44 INFO ( default/core ): NetFlow Accounting Daemon, nfacctd
1.7.0 (20170924-00+c1)
Nov 01 16:55:44 INFO ( default/core ): '--enable-jansson' '--enable-l2'
'--enable-ipv6' '--enable-64bit' '--enable-threads' '--enable-traffic-bins'
'--enable-bgp-bins' '--enable-bmp-bins' '--enable-st-bins'
Nov 01 16:55:44 INFO ( default/core ): Reading configuration file
'/usr/local/etc/nfacctd-procera1.conf'.
Nov 01 16:55:44 INFO ( default/core ):
[/usr/local/etc/nfacctd-primitives.lst] (re)loading map.
Nov 01 16:55:44 INFO ( default/core ):
[/usr/local/etc/nfacctd-primitives.lst] map successfully (re)loaded.
Nov 01 16:55:44 INFO ( web/print ): plugin_pipe_size=81920000 bytes
plugin_buffer_size=8192 bytes
Nov 01 16:55:44 INFO ( web/print ): ctrl channel: obtained=124928 bytes
target=80000 bytes
Nov 01 16:55:44 INFO ( default/core ): [/usr/local/etc/nfacctd-pretag.map]
(re)loading map.
Nov 01 16:55:44 INFO ( web/print ): cache entries=16411 base cache
memory=54878384 bytes
Nov 01 16:55:44 INFO ( default/core ): [/usr/local/etc/nfacctd-pretag.map]
map successfully (re)loaded.
Nov 01 16:55:44 INFO ( default/core ): [/usr/local/etc/nfacctd-pretag.map]
(re)loading map.
Nov 01 16:55:44 INFO ( default/core ): [/usr/local/etc/nfacctd-pretag.map]
map successfully (re)loaded.
Nov 01 16:55:44 INFO ( default/core ): waiting for NetFlow/IPFIX data on
x.x.x.x:9996
# netflow template captured and decoded by tshark:
Cisco NetFlow/IPFIX
Version: 10
Length: 68
Timestamp: Nov 1, 2017 18:08:00.000000000 Middle East Standard Time
ExportTime: 1509552480
FlowSequence: 3775142542
Observation Domain Id: 2879742714
Set 1 [id=2] (Data Template): 12098
FlowSet Id: Data Template (V10 [IPFIX]) (2)
FlowSet Length: 52
Template (Id = 12098, Count = 9)
Template Id: 12098
Field Count: 9
Field (1/9): IP_DST_ADDR
0... .... .... .... = Pen provided: No
.000 0000 0000 1100 = Type: IP_DST_ADDR (12)
Length: 4
Field (2/9): L4_DST_PORT
0... .... .... .... = Pen provided: No
.000 0000 0000 1011 = Type: L4_DST_PORT (11)
Length: 2
Field (3/9): flowEndSeconds
0... .... .... .... = Pen provided: No
.000 0000 1001 0111 = Type: flowEndSeconds (151)
Length: 4
Field (4/9): flowStartSeconds
0... .... .... .... = Pen provided: No
.000 0000 1001 0110 = Type: flowStartSeconds (150)
Length: 4
Field (5/9): 22 [pen: Netintact AB]
1... .... .... .... = Pen provided: Yes
.000 0000 0001 0110 = Type: 22 [pen: Netintact AB]
Length: 65535 [i.e.: "Variable Length"]
PEN: Netintact AB (15397)
Field (6/9): 18 [pen: Netintact AB]
1... .... .... .... = Pen provided: Yes
.000 0000 0001 0010 = Type: 18 [pen: Netintact AB]
Length: 65535 [i.e.: "Variable Length"]
PEN: Netintact AB (15397)
Field (7/9): PROTOCOL
0... .... .... .... = Pen provided: No
.000 0000 0000 0100 = Type: PROTOCOL (4)
Length: 1
Field (8/9): IP_SRC_ADDR
0... .... .... .... = Pen provided: No
.000 0000 0000 1000 = Type: IP_SRC_ADDR (8)
Length: 4
Field (9/9): L4_SRC_PORT
0... .... .... .... = Pen provided: No
.000 0000 0000 0111 = Type: L4_SRC_PORT (7)
Length: 2
# Sample output:
100.66.99.63,31.13.86.34,59295,443,6,2017-11-01 18:10:00.0,2017-11-01
18:15:00.0,,,0,0
100.74.114.163,13.112.60.248,10458,443,6,2017-11-01 18:10:00.0,2017-11-01
18:15:00.0,,,0,0
100.66.86.242,216.58.205.97,64124,443,6,2017-11-01 18:12:36.0,2017-11-01
18:15:00.0,,,0,0
100.102.182.15,31.13.86.51,36051,443,6,2017-11-01 18:10:00.0,2017-11-01
18:15:00.0,,,0,0
100.102.182.15,31.13.86.51,36051,443,6,2017-11-01 18:10:21.0,2017-11-01
18:12:39.0,,,0,0
100.66.150.251,169.45.214.236,54964,443,6,2017-11-01 18:10:00.0,2017-11-01
18:15:00.0,,,0,0
100.101.179.68,40.101.48.88,63261,443,6,2017-11-01 18:12:39.0,2017-11-01
18:12:39.0,,,0,0
100.74.121.140,31.13.86.8,63145,443,6,2017-11-01 18:12:39.0,2017-11-01
18:12:39.0,,,0,0
100.74.117.42,216.58.205.78,54123,80,6,2017-11-01 18:11:54.0,2017-11-01
18:12:39.0,,,0,0
100.101.233.28,31.13.86.51,59636,443,6,2017-11-01 18:10:00.0,2017-11-01
18:15:00.0,,,0,0
100.74.103.34,185.176.144.17,24577,443,6,2017-11-01 18:10:14.0,2017-11-01
18:12:39.0,,,0,0
100.74.107.25,157.240.1.23,49299,443,6,2017-11-01 18:12:08.0,2017-11-01
18:15:00.0,,,0,0
100.101.233.28,31.13.86.51,59636,443,6,2017-11-01 18:11:15.0,2017-11-01
18:12:39.0,,,0,0
100.101.106.136,185.54.60.160,49850,443,6,2017-11-01 18:12:31.0,2017-11-01
18:12:39.0,,,0,0
100.74.118.76,34.235.42.103,14386,443,6,2017-11-01 18:12:38.0,2017-11-01
18:12:39.0,,,0,0
100.74.121.140,31.13.86.8,56464,443,6,2017-11-01 18:12:37.0,2017-11-01
18:15:00.0,,,0,0
100.100.87.90,101.167.166.38,41029,80,6,2017-11-01 18:12:14.0,2017-11-01
18:12:39.0,,,0,0
100.74.114.244,31.13.86.52,64799,443,6,2017-11-01 18:11:30.0,2017-11-01
18:15:00.0,,,0,0
sometimes, but rarely, I get data in proc_http_url field only:
100.66.143.216,17.253.49.204,50113,80,6,2017-11-01 18:10:00.0,2017-11-01
18:15:00.0,,http://appldnld.apple.com/ios11.1seed/*** (rest is removed for
privacy)
I verified that the data is being sent from Procera using wireshark.
Please help
Thank you,
Eddi
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
