Hi Matthew, What you suggest makes sense. Let me review and get back to you - for now thanks very much for contributing your patch.
Cheers, Paolo On Tue, May 07, 2013 at 09:40:34AM +1200, Matthew Grant wrote: > Hi! > > Adding some more details > > On 06/05/13 16:01, Matthew Grant wrote: > > Hi! > > > > Running pmacct 0.14.2 for traffic accounting here in NZ. We are a small ISP > > in > > the antipodes, and run all servers on NZ local time, as the country only > > has one > > time zone effectively. > > > > Found a bug in pmacct 0.14.2 with runing through a DST roll over. > We were running nfacctd through April using 1 month sql_refresh_time. > DST here in NZ ended 2am 7th April. nfacctd was left running over this > period, and stamp_inserted for the 1st May 2013-05-01 was 23:00am!! > > nfacctd.conf included after patch > > Looks like an > > oversight. The tm_isdst flag is set to zero in the calls to mktime() in > > src/util.c. It should be initialized to -1 per mktime(3) man page. This > > should > > work for Linux, OpenBSD, FreeBSD, NetBSD and MacOSX/Darwin. > Checked use of flag on mktime() call via using python interactively and > cutting and pasting between virtual machines with clock set to before > and after NZ DST. > > > > If you want to add a config handler flag for this, it might be a good idea. > > Best Regards, > > Matthew Grant > > Patch is: > > ------- > This patch sets the struct_time tm_isdst flag to -1, to tell libc to > determine > DST from system time zone database. This should prevent pmacct getting > stamp_inserted wrong over DST roll over. > > This patch should work for Linux, FreeBSD, NetBSD, OpenBSD and > MacOSX/Darwin. > > Matthew Grant <[email protected]> Mon, 06 May 2013 14:46:38 +1200 > Index: pmacct-0.14.2/src/util.c > =================================================================== > --- pmacct-0.14.2.orig/src/util.c 2013-05-06 14:50:36.811097741 +1200 > +++ pmacct-0.14.2/src/util.c 2013-05-06 14:53:46.281424257 +1200 > @@ -330,6 +330,8 @@ > } > } > > + /* Tell libc to sort DST rollover when calculating over start/end of > DST */ > + rounded->tm_isdst = -1; > t = mktime(rounded); > return t; > } > @@ -349,6 +351,8 @@ > howmany--; > } > > + /* Tell libc to sort DST rollover when calculating over start/end of > DST */ > + tmt->tm_isdst = -1; > final = mktime(tmt); > > return (final-base); > ---- > ! nfacctd configuration > ! > ! > ! > daemonize: false > syslog: daemon > nfacctd_port: 9996 > pidfile: /var/run/pmacct/nfacctd.pid > ! Adjust nfacctd core process receiver queue > plugin_pipe_size: 40960000 > plugin_buffer_size: 40960 > plugin_pipe_backlog: 25 > pre_tag_map: /var/lib/ipac/pretag.map > refresh_maps: true > ! Need to use AS file to deconvolute pretag.map - Sheesh, pmacct can > have a config brain fuck > networks_file: /var/lib/ipac/pretag-networks.conf > nfacctd_as_new: file > ! > ! storage methods > plugins: pgsql[1h_ip_in], pgsql[1h_ip_out], pgsql[1d_ip_in], > pgsql[1d_ip_out], pgsql[1month_ip_in], pgsql[1month_ip_out], > pgsql[1h_ip_in_country], pgsql[1h_ip_out_country], > pgsql[1d_ip_in_country], pgsql[1d_ip_out_country], > pgsql[1month_ip_in_country],pgsql[1month_ip_out_country], > pgsql[1h_ipip], pgsql[1h_detail], pgsql[1h_type], > pgsql[5m_ip_in_country], pgsql[5m_ip_out_country], pgsql[5m_ip_in], > pgsql[5m_ip_out] > > ! DB details > ! use uinix socket as it is most effecient. > sql_host: /var/run/postgresql > sql_passwd: 1RAeLODtp01Z508K > > ! BGP setup > !bgp_daemon: true > !bgp_daemon_ip: 111.65.238.13 > !bgp_daemon_max_peers: 25 > !bgp_daemon_msglog: true > !nfacctd_as_new: false > !nfacctd_as_new: bgp > > ! Default table values > sql_locking_style: table > sql_optimize_clauses: true > aggregate: src_host, src_port, dst_host, dst_port, proto, tag > nfacctd_time_new: true > nfacctd_renormalize: true > ! Only need this for wierd netflow sources > sampling_map: /etc/ipac/sampling-rate.map > ! Could be good for deconvolving IPv6 if I knew how it worked. > !networks_mask: 64 > geoip_ipv4_file: /usr/share/GeoIP/GeoIP.dat > geoip_ipv6_file: /usr/share/GeoIP/GeoIPv6.dat > sql_max_writers: 25 > sql_delimiter: | > ! Prevent running out of buckets, and dropped traffic accounting entries > sql_cache_entries: 193939 > > ! BGP hourly table > !sql_table[1h_bgp]: acct_1h_bgp > !aggregate[1h_bgp]: src_as, dst_as, peer_src_as, peer_dst_as, > peer_src_ip, peer_dst_ip, as_path, local_pref, med, tag, tag2 > !sql_history[1h_bgp]: 1h > !sql_history_roundoff[1h_bgp]: h > !sql_refresh_time[1h_bgp]: 300 > !sql_dont_try_update[1h_bgp]: true > !sql_use_copy[1h_bgp]: true > > ! Per host summary, in, hourly > sql_table[1h_ip_in]: acct_1h_ip_in > aggregate[1h_ip_in]: dst_host, tag > networks_file[1h_ip_in]: /var/lib/ipac/prefix-networks.conf > sql_history[1h_ip_in]: 1h > sql_history_roundoff[1h_ip_in]: h > sql_refresh_time[1h_ip_in]: 300 > pre_tag2_filter[1h_ip_in]: 1 > > ! Per host summary, out, hourly > sql_table[1h_ip_out]: acct_1h_ip_out > aggregate[1h_ip_out]: src_host, tag > networks_file[1h_ip_out]: /var/lib/ipac/prefix-networks.conf > sql_history[1h_ip_out]: 1h > sql_history_roundoff[1h_ip_out]: h > sql_refresh_time[1h_ip_out]: 300 > pre_tag2_filter[1h_ip_out]: 2 > > ! Per host summary, in > sql_table[1d_ip_in]: acct_1d_ip_in > aggregate[1d_ip_in]: dst_host, tag > networks_file[1d_ip_in]: /var/lib/ipac/prefix-networks.conf > sql_history[1d_ip_in]: 1d > sql_history_roundoff[1d_ip_in]: d > sql_refresh_time[1d_ip_in]: 300 > pre_tag2_filter[1d_ip_in]: 1 > > ! Per host summary, out > sql_table[1d_ip_out]: acct_1d_ip_out > aggregate[1d_ip_out]: src_host, tag > networks_file[1d_ip_out]: /var/lib/ipac/prefix-networks.conf > sql_history[1d_ip_out]: 1d > sql_history_roundoff[1d_ip_out]: d > sql_refresh_time[1d_ip_out]: 300 > pre_tag2_filter[1d_ip_out]: 2 > > ! Per host summary, in, monthly > sql_table[1month_ip_in]: acct_1month_ip_in > aggregate[1month_ip_in]: dst_host, tag > networks_file[1month_ip_in]: /var/lib/ipac/prefix-networks.conf > sql_history[1month_ip_in]: 1M > sql_history_roundoff[1month_ip_in]: M > sql_refresh_time[1month_ip_in]: 300 > pre_tag2_filter[1month_ip_in]: 1 > > ! Per host summary, out, monthly > sql_table[1month_ip_out]: acct_1month_ip_out > aggregate[1month_ip_out]: src_host, tag > networks_file[1month_ip_out]: /var/lib/ipac/prefix-networks.conf > sql_history[1month_ip_out]: 1M > sql_history_roundoff[1month_ip_out]: M > sql_refresh_time[1month_ip_out]: 300 > pre_tag2_filter[1month_ip_out]: 2 > > ! Per host summary, in, percentile 5 minutes > sql_table[5m_ip_in_country]: acct_5m_ip_in_country > aggregate[5m_ip_in_country]: dst_host, src_host_country, tag > networks_file[5m_ip_in_country]: /var/lib/ipac/prefix-networks.conf > sql_history[5m_ip_in_country]: 5m > sql_history_roundoff[5m_ip_in_country]: m > sql_refresh_time[5m_ip_in_country]: 300 > sql_dont_try_update[5m_ip_in_country]: true > sql_use_copy[5m_ip_in_country]: true > pre_tag2_filter[5m_ip_in_country]: 1 > > ! Per host summary, out, percentile 5 minutes > sql_table[5m_ip_out_country]: acct_5m_ip_out_country > aggregate[5m_ip_out_country]: src_host, dst_host_country, tag > networks_file[5m_ip_out_country]: /var/lib/ipac/prefix-networks.conf > sql_history[5m_ip_out_country]: 5m > sql_history_roundoff[5m_ip_out_country]: m > sql_refresh_time[5m_ip_out_country]: 300 > sql_dont_try_update[5m_ip_out_country]: true > sql_use_copy[5m_ip_out_country]: true > pre_tag2_filter[5m_ip_out_country]: 2 > > ! Per host summary, in, domestic hourly > sql_table[1h_ip_in_country]: acct_1h_ip_in_country > aggregate[1h_ip_in_country]: dst_host, src_host_country, tag > networks_file[1h_ip_in_country]: /var/lib/ipac/prefix-networks.conf > sql_history[1h_ip_in_country]: 1h > sql_history_roundoff[1h_ip_in_country]: h > sql_refresh_time[1h_ip_in_country]: 300 > pre_tag2_filter[1h_ip_in_country]: 1 > > ! Per host summary, out, domestic hourly > sql_table[1h_ip_out_country]: acct_1h_ip_out_country > aggregate[1h_ip_out_country]: src_host, dst_host_country, tag > networks_file[1h_ip_out_country]: /var/lib/ipac/prefix-networks.conf > sql_history[1h_ip_out_country]: 1h > sql_history_roundoff[1h_ip_out_country]: h > sql_refresh_time[1h_ip_out_country]: 300 > pre_tag2_filter[1h_ip_out_country]: 2 > > ! Per host summary, in, domestic daily > sql_table[1d_ip_in_country]: acct_1d_ip_in_country > aggregate[1d_ip_in_country]: dst_host, src_host_country, tag > networks_file[1d_ip_in_country]: /var/lib/ipac/prefix-networks.conf > sql_history[1d_ip_in_country]: 1d > sql_history_roundoff[1d_ip_in_country]: d > sql_refresh_time[1d_ip_in_country]: 300 > pre_tag2_filter[1d_ip_in_country]: 1 > > ! Per host summary, out, domestic daily > sql_table[1d_ip_out_country]: acct_1d_ip_out_country > aggregate[1d_ip_out_country]: src_host, dst_host_country, tag > networks_file[1d_ip_out_country]: /var/lib/ipac/prefix-networks.conf > sql_history[1d_ip_out_country]: 1d > sql_history_roundoff[1d_ip_out_country]: d > sql_refresh_time[1d_ip_out_country]: 300 > pre_tag2_filter[1d_ip_out_country]: 2 > > ! Per host summary, in, domestic monthly > sql_table[1month_ip_in_country]: acct_1month_ip_in_country > aggregate[1month_ip_in_country]: dst_host, src_host_country, tag > networks_file[1month_ip_in_country]: /var/lib/ipac/prefix-networks.conf > sql_history[1month_ip_in_country]: 1M > sql_history_roundoff[1month_ip_in_country]: M > sql_refresh_time[1month_ip_in_country]: 300 > pre_tag2_filter[1month_ip_in_country]: 1 > > ! Per host summary, out, domestic monthly > sql_table[1month_ip_out_country]: acct_1month_ip_out_country > aggregate[1month_ip_out_country]: src_host, dst_host_country, tag > networks_file[1month_ip_out_country]: /var/lib/ipac/prefix-networks.conf > sql_history[1month_ip_out_country]: 1M > sql_history_roundoff[1month_ip_out_country]: M > sql_refresh_time[1month_ip_out_country]: 300 > pre_tag2_filter[1month_ip_out_country]: 2 > > ! Hourly IP to IP file > sql_table[1h_ipip]: acct_1h_ipip > aggregate[1h_ipip]: src_host, dst_host, tag, tag2 > ports_file[1h_ipip]: /var/lib/ipac/ports.conf > sql_history[1h_ipip]: 1h > sql_history_roundoff[1h_ipip]: h > sql_refresh_time[1h_ipip]: 3600 > networks_file[1h_ipip]: /etc/ipac/default-route-networks.conf > sql_dont_try_update[1h_ipip]: true > sql_use_copy[1h_ipip]: true > pre_tag2_filter[1h_ipip]: 1,2,5 > > ! Hourly detail table > sql_table[1h_detail]: acct_1h_detail > aggregate[1h_detail]: src_host, src_port, dst_host, dst_port, proto, > tag, tag2 > ports_file[1h_detail]: /var/lib/ipac/ports.conf > sql_history[1h_detail]: 1h > sql_history_roundoff[1h_detail]: h > sql_refresh_time[1h_detail]: 3600 > networks_file[1h_detail]: /etc/ipac/default-route-networks.conf > sql_dont_try_update[1h_detail]: true > sql_use_copy[1h_detail]: true > > sql_table[1h_type]: acct_1h_type > aggregate[1h_type]: src_port, dst_port, proto, flows, tag2 > sql_history[1h_type]: 1h > sql_history_roundoff[1h_type]: h > sql_refresh_time[1h_type]: 3600 > networks_file[1h_type]: /etc/ipac/default-route-networks.conf > sql_dont_try_update[1h_type]: true > sql_use_copy[1h_type]: true > pre_tag2_filter[1h_type]: 1,2,5 > > ! Per host summary, in, percentile 5 minutes > sql_table[5m_ip_in_country]: acct_5m_ip_in_country > aggregate[5m_ip_in_country]: dst_host, src_host_country, tag > networks_file[5m_ip_in_country]: /var/lib/ipac/prefix-networks.conf > sql_history[5m_ip_in_country]: 5m > sql_history_roundoff[5m_ip_in_country]: m > sql_refresh_time[5m_ip_in_country]: 300 > sql_dont_try_update[5m_ip_in_country]: true > sql_use_copy[5m_ip_in_country]: true > pre_tag2_filter[5m_ip_in_country]: 1 > > ! Per host summary, out, percentile 5 minutes > sql_table[5m_ip_out_country]: acct_5m_ip_out_country > aggregate[5m_ip_out_country]: src_host, dst_host_country, tag > networks_file[5m_ip_out_country]: /var/lib/ipac/prefix-networks.conf > sql_history[5m_ip_out_country]: 5m > sql_history_roundoff[5m_ip_out_country]: m > sql_refresh_time[5m_ip_out_country]: 300 > sql_dont_try_update[5m_ip_out_country]: true > sql_use_copy[5m_ip_out_country]: true > pre_tag2_filter[5m_ip_out_country]: 2 > > ! Per host summary, in, percentile 5 minutes > sql_table[5m_ip_in]: acct_5m_ip_in > aggregate[5m_ip_in]: dst_host, tag > networks_file[5m_ip_in]: /var/lib/ipac/prefix-networks.conf > sql_history[5m_ip_in]: 5m > sql_history_roundoff[5m_ip_in]: m > sql_refresh_time[5m_ip_in]: 300 > sql_dont_try_update[5m_ip_in]: true > sql_use_copy[5m_ip_in]: true > pre_tag2_filter[5m_ip_in]: 1 > > ! Per host summary, out, percentile 5 minutes > sql_table[5m_ip_out]: acct_5m_ip_out > aggregate[5m_ip_out]: src_host, tag > networks_file[5m_ip_out]: /var/lib/ipac/prefix-networks.conf > sql_history[5m_ip_out]: 5m > sql_history_roundoff[5m_ip_out]: m > sql_refresh_time[5m_ip_out]: 300 > sql_dont_try_update[5m_ip_out]: true > sql_use_copy[5m_ip_out]: true > pre_tag2_filter[5m_ip_out]: 2 > > > > > > _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
