Hi Paolo! Just like to add that most C compilers effectively memset() structs to zero on 'allocation'. (Long knowledge of C programming embedded systems like routers and looking at binary stack dumps etc. That is where I get tm_isdst being initialized to zero in src/util.c mktime(3) calls.
Cheers, Matthew On 09/05/13 11:04, Paolo Lucente wrote: > Hi Matthew, > > What you suggest makes sense. Let me review and get back to you - for > now thanks very much for contributing your patch. > > Cheers, > Paolo > > On Tue, May 07, 2013 at 09:40:34AM +1200, Matthew Grant wrote: >> Hi! >> >> Adding some more details >> >> On 06/05/13 16:01, Matthew Grant wrote: >>> Hi! >>> >>> Running pmacct 0.14.2 for traffic accounting here in NZ. We are a small ISP >>> in >>> the antipodes, and run all servers on NZ local time, as the country only >>> has one >>> time zone effectively. >>> >>> Found a bug in pmacct 0.14.2 with runing through a DST roll over. >> We were running nfacctd through April using 1 month sql_refresh_time. >> DST here in NZ ended 2am 7th April. nfacctd was left running over this >> period, and stamp_inserted for the 1st May 2013-05-01 was 23:00am!! >> >> nfacctd.conf included after patch >>> Looks like an >>> oversight. The tm_isdst flag is set to zero in the calls to mktime() in >>> src/util.c. It should be initialized to -1 per mktime(3) man page. This >>> should >>> work for Linux, OpenBSD, FreeBSD, NetBSD and MacOSX/Darwin. >> Checked use of flag on mktime() call via using python interactively and >> cutting and pasting between virtual machines with clock set to before >> and after NZ DST. >>> If you want to add a config handler flag for this, it might be a good idea. >> Best Regards, >> >> Matthew Grant >> >> Patch is: >> >> ------- >> This patch sets the struct_time tm_isdst flag to -1, to tell libc to >> determine >> DST from system time zone database. This should prevent pmacct getting >> stamp_inserted wrong over DST roll over. >> >> This patch should work for Linux, FreeBSD, NetBSD, OpenBSD and >> MacOSX/Darwin. >> >> Matthew Grant <[email protected]> Mon, 06 May 2013 14:46:38 +1200 >> Index: pmacct-0.14.2/src/util.c >> =================================================================== >> --- pmacct-0.14.2.orig/src/util.c 2013-05-06 14:50:36.811097741 +1200 >> +++ pmacct-0.14.2/src/util.c 2013-05-06 14:53:46.281424257 +1200 >> @@ -330,6 +330,8 @@ >> } >> } >> >> + /* Tell libc to sort DST rollover when calculating over start/end of >> DST */ >> + rounded->tm_isdst = -1; >> t = mktime(rounded); >> return t; >> } >> @@ -349,6 +351,8 @@ >> howmany--; >> } >> >> + /* Tell libc to sort DST rollover when calculating over start/end of >> DST */ >> + tmt->tm_isdst = -1; >> final = mktime(tmt); >> >> return (final-base); >> ---- >> ! nfacctd configuration >> ! >> ! >> ! >> daemonize: false >> syslog: daemon >> nfacctd_port: 9996 >> pidfile: /var/run/pmacct/nfacctd.pid >> ! Adjust nfacctd core process receiver queue >> plugin_pipe_size: 40960000 >> plugin_buffer_size: 40960 >> plugin_pipe_backlog: 25 >> pre_tag_map: /var/lib/ipac/pretag.map >> refresh_maps: true >> ! Need to use AS file to deconvolute pretag.map - Sheesh, pmacct can >> have a config brain fuck >> networks_file: /var/lib/ipac/pretag-networks.conf >> nfacctd_as_new: file >> ! >> ! storage methods >> plugins: pgsql[1h_ip_in], pgsql[1h_ip_out], pgsql[1d_ip_in], >> pgsql[1d_ip_out], pgsql[1month_ip_in], pgsql[1month_ip_out], >> pgsql[1h_ip_in_country], pgsql[1h_ip_out_country], >> pgsql[1d_ip_in_country], pgsql[1d_ip_out_country], >> pgsql[1month_ip_in_country],pgsql[1month_ip_out_country], >> pgsql[1h_ipip], pgsql[1h_detail], pgsql[1h_type], >> pgsql[5m_ip_in_country], pgsql[5m_ip_out_country], pgsql[5m_ip_in], >> pgsql[5m_ip_out] >> >> ! DB details >> ! use uinix socket as it is most effecient. >> sql_host: /var/run/postgresql >> sql_passwd: 1RAeLODtp01Z508K >> >> ! BGP setup >> !bgp_daemon: true >> !bgp_daemon_ip: 111.65.238.13 >> !bgp_daemon_max_peers: 25 >> !bgp_daemon_msglog: true >> !nfacctd_as_new: false >> !nfacctd_as_new: bgp >> >> ! Default table values >> sql_locking_style: table >> sql_optimize_clauses: true >> aggregate: src_host, src_port, dst_host, dst_port, proto, tag >> nfacctd_time_new: true >> nfacctd_renormalize: true >> ! Only need this for wierd netflow sources >> sampling_map: /etc/ipac/sampling-rate.map >> ! Could be good for deconvolving IPv6 if I knew how it worked. >> !networks_mask: 64 >> geoip_ipv4_file: /usr/share/GeoIP/GeoIP.dat >> geoip_ipv6_file: /usr/share/GeoIP/GeoIPv6.dat >> sql_max_writers: 25 >> sql_delimiter: | >> ! Prevent running out of buckets, and dropped traffic accounting entries >> sql_cache_entries: 193939 >> >> ! BGP hourly table >> !sql_table[1h_bgp]: acct_1h_bgp >> !aggregate[1h_bgp]: src_as, dst_as, peer_src_as, peer_dst_as, >> peer_src_ip, peer_dst_ip, as_path, local_pref, med, tag, tag2 >> !sql_history[1h_bgp]: 1h >> !sql_history_roundoff[1h_bgp]: h >> !sql_refresh_time[1h_bgp]: 300 >> !sql_dont_try_update[1h_bgp]: true >> !sql_use_copy[1h_bgp]: true >> >> ! Per host summary, in, hourly >> sql_table[1h_ip_in]: acct_1h_ip_in >> aggregate[1h_ip_in]: dst_host, tag >> networks_file[1h_ip_in]: /var/lib/ipac/prefix-networks.conf >> sql_history[1h_ip_in]: 1h >> sql_history_roundoff[1h_ip_in]: h >> sql_refresh_time[1h_ip_in]: 300 >> pre_tag2_filter[1h_ip_in]: 1 >> >> ! Per host summary, out, hourly >> sql_table[1h_ip_out]: acct_1h_ip_out >> aggregate[1h_ip_out]: src_host, tag >> networks_file[1h_ip_out]: /var/lib/ipac/prefix-networks.conf >> sql_history[1h_ip_out]: 1h >> sql_history_roundoff[1h_ip_out]: h >> sql_refresh_time[1h_ip_out]: 300 >> pre_tag2_filter[1h_ip_out]: 2 >> >> ! Per host summary, in >> sql_table[1d_ip_in]: acct_1d_ip_in >> aggregate[1d_ip_in]: dst_host, tag >> networks_file[1d_ip_in]: /var/lib/ipac/prefix-networks.conf >> sql_history[1d_ip_in]: 1d >> sql_history_roundoff[1d_ip_in]: d >> sql_refresh_time[1d_ip_in]: 300 >> pre_tag2_filter[1d_ip_in]: 1 >> >> ! Per host summary, out >> sql_table[1d_ip_out]: acct_1d_ip_out >> aggregate[1d_ip_out]: src_host, tag >> networks_file[1d_ip_out]: /var/lib/ipac/prefix-networks.conf >> sql_history[1d_ip_out]: 1d >> sql_history_roundoff[1d_ip_out]: d >> sql_refresh_time[1d_ip_out]: 300 >> pre_tag2_filter[1d_ip_out]: 2 >> >> ! Per host summary, in, monthly >> sql_table[1month_ip_in]: acct_1month_ip_in >> aggregate[1month_ip_in]: dst_host, tag >> networks_file[1month_ip_in]: /var/lib/ipac/prefix-networks.conf >> sql_history[1month_ip_in]: 1M >> sql_history_roundoff[1month_ip_in]: M >> sql_refresh_time[1month_ip_in]: 300 >> pre_tag2_filter[1month_ip_in]: 1 >> >> ! Per host summary, out, monthly >> sql_table[1month_ip_out]: acct_1month_ip_out >> aggregate[1month_ip_out]: src_host, tag >> networks_file[1month_ip_out]: /var/lib/ipac/prefix-networks.conf >> sql_history[1month_ip_out]: 1M >> sql_history_roundoff[1month_ip_out]: M >> sql_refresh_time[1month_ip_out]: 300 >> pre_tag2_filter[1month_ip_out]: 2 >> >> ! Per host summary, in, percentile 5 minutes >> sql_table[5m_ip_in_country]: acct_5m_ip_in_country >> aggregate[5m_ip_in_country]: dst_host, src_host_country, tag >> networks_file[5m_ip_in_country]: /var/lib/ipac/prefix-networks.conf >> sql_history[5m_ip_in_country]: 5m >> sql_history_roundoff[5m_ip_in_country]: m >> sql_refresh_time[5m_ip_in_country]: 300 >> sql_dont_try_update[5m_ip_in_country]: true >> sql_use_copy[5m_ip_in_country]: true >> pre_tag2_filter[5m_ip_in_country]: 1 >> >> ! Per host summary, out, percentile 5 minutes >> sql_table[5m_ip_out_country]: acct_5m_ip_out_country >> aggregate[5m_ip_out_country]: src_host, dst_host_country, tag >> networks_file[5m_ip_out_country]: /var/lib/ipac/prefix-networks.conf >> sql_history[5m_ip_out_country]: 5m >> sql_history_roundoff[5m_ip_out_country]: m >> sql_refresh_time[5m_ip_out_country]: 300 >> sql_dont_try_update[5m_ip_out_country]: true >> sql_use_copy[5m_ip_out_country]: true >> pre_tag2_filter[5m_ip_out_country]: 2 >> >> ! Per host summary, in, domestic hourly >> sql_table[1h_ip_in_country]: acct_1h_ip_in_country >> aggregate[1h_ip_in_country]: dst_host, src_host_country, tag >> networks_file[1h_ip_in_country]: /var/lib/ipac/prefix-networks.conf >> sql_history[1h_ip_in_country]: 1h >> sql_history_roundoff[1h_ip_in_country]: h >> sql_refresh_time[1h_ip_in_country]: 300 >> pre_tag2_filter[1h_ip_in_country]: 1 >> >> ! Per host summary, out, domestic hourly >> sql_table[1h_ip_out_country]: acct_1h_ip_out_country >> aggregate[1h_ip_out_country]: src_host, dst_host_country, tag >> networks_file[1h_ip_out_country]: /var/lib/ipac/prefix-networks.conf >> sql_history[1h_ip_out_country]: 1h >> sql_history_roundoff[1h_ip_out_country]: h >> sql_refresh_time[1h_ip_out_country]: 300 >> pre_tag2_filter[1h_ip_out_country]: 2 >> >> ! Per host summary, in, domestic daily >> sql_table[1d_ip_in_country]: acct_1d_ip_in_country >> aggregate[1d_ip_in_country]: dst_host, src_host_country, tag >> networks_file[1d_ip_in_country]: /var/lib/ipac/prefix-networks.conf >> sql_history[1d_ip_in_country]: 1d >> sql_history_roundoff[1d_ip_in_country]: d >> sql_refresh_time[1d_ip_in_country]: 300 >> pre_tag2_filter[1d_ip_in_country]: 1 >> >> ! Per host summary, out, domestic daily >> sql_table[1d_ip_out_country]: acct_1d_ip_out_country >> aggregate[1d_ip_out_country]: src_host, dst_host_country, tag >> networks_file[1d_ip_out_country]: /var/lib/ipac/prefix-networks.conf >> sql_history[1d_ip_out_country]: 1d >> sql_history_roundoff[1d_ip_out_country]: d >> sql_refresh_time[1d_ip_out_country]: 300 >> pre_tag2_filter[1d_ip_out_country]: 2 >> >> ! Per host summary, in, domestic monthly >> sql_table[1month_ip_in_country]: acct_1month_ip_in_country >> aggregate[1month_ip_in_country]: dst_host, src_host_country, tag >> networks_file[1month_ip_in_country]: /var/lib/ipac/prefix-networks.conf >> sql_history[1month_ip_in_country]: 1M >> sql_history_roundoff[1month_ip_in_country]: M >> sql_refresh_time[1month_ip_in_country]: 300 >> pre_tag2_filter[1month_ip_in_country]: 1 >> >> ! Per host summary, out, domestic monthly >> sql_table[1month_ip_out_country]: acct_1month_ip_out_country >> aggregate[1month_ip_out_country]: src_host, dst_host_country, tag >> networks_file[1month_ip_out_country]: /var/lib/ipac/prefix-networks.conf >> sql_history[1month_ip_out_country]: 1M >> sql_history_roundoff[1month_ip_out_country]: M >> sql_refresh_time[1month_ip_out_country]: 300 >> pre_tag2_filter[1month_ip_out_country]: 2 >> >> ! Hourly IP to IP file >> sql_table[1h_ipip]: acct_1h_ipip >> aggregate[1h_ipip]: src_host, dst_host, tag, tag2 >> ports_file[1h_ipip]: /var/lib/ipac/ports.conf >> sql_history[1h_ipip]: 1h >> sql_history_roundoff[1h_ipip]: h >> sql_refresh_time[1h_ipip]: 3600 >> networks_file[1h_ipip]: /etc/ipac/default-route-networks.conf >> sql_dont_try_update[1h_ipip]: true >> sql_use_copy[1h_ipip]: true >> pre_tag2_filter[1h_ipip]: 1,2,5 >> >> ! Hourly detail table >> sql_table[1h_detail]: acct_1h_detail >> aggregate[1h_detail]: src_host, src_port, dst_host, dst_port, proto, >> tag, tag2 >> ports_file[1h_detail]: /var/lib/ipac/ports.conf >> sql_history[1h_detail]: 1h >> sql_history_roundoff[1h_detail]: h >> sql_refresh_time[1h_detail]: 3600 >> networks_file[1h_detail]: /etc/ipac/default-route-networks.conf >> sql_dont_try_update[1h_detail]: true >> sql_use_copy[1h_detail]: true >> >> sql_table[1h_type]: acct_1h_type >> aggregate[1h_type]: src_port, dst_port, proto, flows, tag2 >> sql_history[1h_type]: 1h >> sql_history_roundoff[1h_type]: h >> sql_refresh_time[1h_type]: 3600 >> networks_file[1h_type]: /etc/ipac/default-route-networks.conf >> sql_dont_try_update[1h_type]: true >> sql_use_copy[1h_type]: true >> pre_tag2_filter[1h_type]: 1,2,5 >> >> ! Per host summary, in, percentile 5 minutes >> sql_table[5m_ip_in_country]: acct_5m_ip_in_country >> aggregate[5m_ip_in_country]: dst_host, src_host_country, tag >> networks_file[5m_ip_in_country]: /var/lib/ipac/prefix-networks.conf >> sql_history[5m_ip_in_country]: 5m >> sql_history_roundoff[5m_ip_in_country]: m >> sql_refresh_time[5m_ip_in_country]: 300 >> sql_dont_try_update[5m_ip_in_country]: true >> sql_use_copy[5m_ip_in_country]: true >> pre_tag2_filter[5m_ip_in_country]: 1 >> >> ! Per host summary, out, percentile 5 minutes >> sql_table[5m_ip_out_country]: acct_5m_ip_out_country >> aggregate[5m_ip_out_country]: src_host, dst_host_country, tag >> networks_file[5m_ip_out_country]: /var/lib/ipac/prefix-networks.conf >> sql_history[5m_ip_out_country]: 5m >> sql_history_roundoff[5m_ip_out_country]: m >> sql_refresh_time[5m_ip_out_country]: 300 >> sql_dont_try_update[5m_ip_out_country]: true >> sql_use_copy[5m_ip_out_country]: true >> pre_tag2_filter[5m_ip_out_country]: 2 >> >> ! Per host summary, in, percentile 5 minutes >> sql_table[5m_ip_in]: acct_5m_ip_in >> aggregate[5m_ip_in]: dst_host, tag >> networks_file[5m_ip_in]: /var/lib/ipac/prefix-networks.conf >> sql_history[5m_ip_in]: 5m >> sql_history_roundoff[5m_ip_in]: m >> sql_refresh_time[5m_ip_in]: 300 >> sql_dont_try_update[5m_ip_in]: true >> sql_use_copy[5m_ip_in]: true >> pre_tag2_filter[5m_ip_in]: 1 >> >> ! Per host summary, out, percentile 5 minutes >> sql_table[5m_ip_out]: acct_5m_ip_out >> aggregate[5m_ip_out]: src_host, tag >> networks_file[5m_ip_out]: /var/lib/ipac/prefix-networks.conf >> sql_history[5m_ip_out]: 5m >> sql_history_roundoff[5m_ip_out]: m >> sql_refresh_time[5m_ip_out]: 300 >> sql_dont_try_update[5m_ip_out]: true >> sql_use_copy[5m_ip_out]: true >> pre_tag2_filter[5m_ip_out]: 2 >> >> >> >> >> >> -- Regards, Matthew Grant | Systems Engineer *Phone:* 0800 5000 24 | +64 3 962 9510 <http://www.voyager.co.nz> Voice & Data www.voyager.co.nz <http://www.voyager.co.nz/> +64 9 444 4444 <http://www.net24.co.nz> Hosting & Cloud www.net24.co.nz <http://www.net24.co.nz/> 0800 5000 24 <http://www.1stdomains.co.nz> Domains & Email www.1stdomains.co.nz <http://www.1stdomains.co.nz/> +64 3 962 9520 _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
