Hi,
After wasting a couple of months on testing several IPFIX collectors that
can absorb the load I have, I finally found pmacct!
nfacctd is the only software I could find that is holding rock solid. No
rocket science here, compiled, fine-tuned the buffers and it is sucking it
all and asking for more.
My requirements: Log each and every visited web site (http/https) on their
standard ports.
Equipment: Procera boxes sending netflow v10
Problem: The defined custom fields rarely include data on the output
Setup details
# conf file:
nfacctd_port: 9996
nfacctd_allow_file: /usr/local/etc/nfacctd.allow
daemonize: true
pidfile: /var/run/nfacctd
plugins: print[web]
plugin_pipe_size: 8192
plugin_buffer_size: 8192
logfile: /var/log/nfacctd.log
print_output_file_append: true
print_output_file[web]: /data/live/procera1.log
timestamps_secs: true
timestamps_since_epoch: false
print_output[web]: csv
print_output_separator[web]: ,
print_num_protos: true
nfacctd_time_secs: false
nfacctd_time_new: false
nfacctd_templates_file: /tmp/procera1.tmpl
nfacctd_disable_checks: true
pre_tag_map: /usr/local/etc/nfacctd-pretag.map
pre_tag_filter[web]: 80443
aggregate_primitives: /usr/local/etc/nfacctd-primitives.lst
aggregate[web]: timestamp_start, timestamp_end, proto, src_host, src_port,
dst_host, dst_port, proc_svr_host, proc_http_url
# nfacctd-primitives.lst:
name=proc_svr_host field_type=15397:18 len=655535
semantics=string
name=proc_http_url field_type=15397:22 len=655535
semantics=string
# nfacctd-pretag.map:
set_tag=80443 filter='dst port 80'
set_tag=80443 filter='dst port 443'
# template fields as per the nfacctd_templates_file:
{"type": 0, "otpl": {"off": 0, "len": 4, "tpl_len": 4, "tpl_index": 12}}
{"type": 0, "otpl": {"off": 4, "len": 2, "tpl_len": 2, "tpl_index": 11}}
{"type": 0, "otpl": {"off": 6, "len": 4, "tpl_len": 4, "tpl_index": 151}}
{"type": 0, "otpl": {"off": 10, "len": 4, "tpl_len": 4, "tpl_index": 150}}
{"type": 1, "utpl": {"pen": 15397, "type": 22, "off": 14, "len": 0,
"tpl_len": 65535, "repeat_id": 0, "ie_idx": 0}}
{"type": 1, "utpl": {"pen": 15397, "type": 18, "off": 0, "len": 0,
"tpl_len": 65535, "repeat_id": 0, "ie_idx": 0}}
{"type": 0, "otpl": {"off": 0, "len": 1, "tpl_len": 1, "tpl_index": 4}}
{"type": 0, "otpl": {"off": 0, "len": 4, "tpl_len": 4, "tpl_index": 8}}
{"type": 0, "otpl": {"off": 0, "len": 2, "tpl_len": 2, "tpl_index": 7}}
# log file showing the startup:
Nov 01 16:55:44 INFO ( default/core ): NetFlow Accounting Daemon, nfacctd
1.7.0 (20170924-00+c1)
Nov 01 16:55:44 INFO ( default/core ): '--enable-jansson' '--enable-l2'
'--enable-ipv6' '--enable-64bit' '--enable-threads' '--enable-traffic-bins'
'--enable-bgp-bins' '--enable-bmp-bins' '--enable-st-bins'
Nov 01 16:55:44 INFO ( default/core ): Reading configuration file
'/usr/local/etc/nfacctd-procera1.conf'.
Nov 01 16:55:44 INFO ( default/core ):
[/usr/local/etc/nfacctd-primitives.lst] (re)loading map.
Nov 01 16:55:44 INFO ( default/core ):
[/usr/local/etc/nfacctd-primitives.lst] map successfully (re)loaded.
Nov 01 16:55:44 INFO ( web/print ): plugin_pipe_size=8192 bytes
plugin_buffer_size=8192 bytes
Nov 01 16:55:44 INFO ( web/print ): ctrl channel: obtained=124928 bytes
target=8 bytes
Nov 01 16:55:44 INFO ( default/core ): [/usr/local/etc/nfacctd-pretag.map]
(re)loading map.
Nov 01 16:55:44 INFO ( web/print ): cache entries=16411 base cache
memory=54878384 bytes
Nov 01 16:55:44 INFO ( default/core ): [/usr/local/etc/nfacctd-pretag.map]
map successfully (re)loaded.
Nov 01 16:55:44 INFO ( default/core ): [/usr/local/etc/nfacctd-pretag.map]
(re)loading map.
Nov 01 16:55:44 INFO ( default/core ): [/usr/local/etc/nfacctd-pretag.map]
map successfully (re)loaded.
Nov 01 16:55:44 INFO ( default/core ): waiting for NetFlow/IPFIX data on
x.x.x.x:9996
# netflow template captured and decoded by tshark:
Cisco NetFlow/IPFIX
Version: 10
Length: 68
Timestamp: Nov 1, 2017 18:08:00.0 Middle East Standard Time
ExportTime: 1509552480
FlowSequence: 3775142542
Observation Domain Id: 2879742714
Set 1 [id=2] (Data Template): 12098
FlowSet Id: Data Template (V10 [IPFIX]) (2)
FlowSet Length: 52
Template (Id = 12098, Count = 9)
Template Id: 12098
Field Count: 9
Field (1/9): IP_DST_ADDR
0... = Pen provided: No
.000 1100 = Type: IP_DST_ADDR (12)
Length: 4
Field (2/9): L4_DST_PORT
0... = Pen provided: No
.000 1011 = Type: L4_DST_PORT (11)
Length: 2
Field (3/9): flowEndSeconds
0... = Pen provided: No
.000 1001 0111 = Type: flowEndSeconds (151)
Length: 4
Field (4/9): flowStartSeconds
0... = Pen provided: No
.000
