Re: [pmacct-discussion] Pmacct data inconsistencies between tables.

2010-02-16 Thread Paolo Lucente 
Hi Daniel,

Getting through the data and compare traffic figures is,
IHMO, the more practical approach - compared to trying to
reproduce the issue in a controlled environment. Once you
discover a descrepancy, it would be great to receive the
contributing data of each report to see where the issue
comes from.

It's also true that version 0.9.1 is almost 5 years old; i
would highly encourage to refresh it. I should be correct
saying Ubuntu features also version 0.11.4 and 0.11.6 if
you really don't like the idea of compiling 0.12 yourself
(which would be my preferred approach).

Let me know.

Cheers,
Paolo



On Mon, Feb 15, 2010 at 10:41:21AM +, Daniel Levy wrote:
 Hi Paolo,
 
 Thanks for getting back to me. The version of pmacct being used is
 0.9.1-1ubuntu1. I'm not sure how the problem was discovered, but I have
 asked to person who found the problem to tell me and I will forward you
 the response. As for the reports, I'm not entirely sure what you need. I
 am considering going through the database data for each hour and
 comparing the total figures for uploaded and downloaded packets, per IP
 address between the two tables.  Would this give you the information
 you're looking for?
 
 -- 
 Daniel Levy
 
 Aptivate | http://www.aptivate.org/ | +44 (0)1223 760887
 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES
 
 Aptivate is a not-for-profit company registered in England and Wales
 with company number 04980791. 
 
 
 
 Paolo Lucente wrote:
  Hi Daniel,
 
  Unfortunately the configuration doesn't make evident where the
  issue can be. The 'sql_dont_try_update' very well protects against
  duplicate tuples - so i'm rather inclined to exclude that reason. 
 
  Which version are you using? How you did discover the issue - ie.
  did you upgrade recently from a previous version or is a fresh
  installation? Finally, is it possible to get - privately - two
  reports, one from each table, for the same time period? Say, one
  or better two hours? 
 
  Let me know.
 
  Cheers,
  Paolo
 
 
  On Fri, Feb 12, 2010 at 03:19:38PM +, Daniel Levy wrote:

  Hi,
 
  I'm using pmacct to store data in two tables, one containing data
  recorded on a per minute basis, the other containing data recorded on an
  hourly basis. When I get data for the first table over a period of three
  hours, the download traffic (calculated by adding up the bytes field for
  traffic where the ip_dst value is from a machine on the local network)
  for one IP address on the network is 3,719,772,656 bytes. The download
  traffic from the second table for the same IP address over a period of
  one week, including the three hour period mentioned above, is
  significantly smaller (2,114,286,512 bytes) where I would expect it to
  be much larger and I can't figure out why. A slightly modified version
  of the contents of my pmacctd.conf file is given below. Can anyone help?
 
  daemonize: true
  pidfile: /var/run/pmacctd.pid
  syslog: daemon
 
  plugins: mysql[inbound1], mysql[outbound1], mysql[inbound2],
  mysql[outbound2]
 
  aggregate[inbound1]: src_host, src_port, dst_host, dst_port, proto
  aggregate[outbound1]: src_host, src_port, dst_host, dst_port, proto
  aggregate[inbound2]: src_host, src_port, dst_host, dst_port, proto
  aggregate[outbound2]: src_host, src_port, dst_host, dst_port, proto
 
  pcap_filter: not (src and dst net 0.0.0.0/24)
 
 
  sql_db: pmacct
  sql_table[inbound1]: short_data_table
  sql_table[outbound1]: short_data_table
 
  sql_table[inbound2]: long_data_table
  sql_table[outbound2]: long_data_table
 
  sql_history[inbound1]: 1m
  sql_history[outbound1]: 1m
  sql_history[inbound2]: 1h
  sql_history[outbound2]: 1h
 
  sql_history_roundoff[inbound1]: m
  sql_history_roundoff[outbound1]: m
  sql_history_roundoff[inbound2]: h
  sql_history_roundoff[outbound2]: h
  sql_table_version: 6
  sql_host: localhost
  sql_user: auser
  sql_passwd: apass
 
  sql_refresh_time[inbound1]: 60
  sql_refresh_time[outbound1]: 60
  sql_refresh_time[inbound2]: 3600
  sql_refresh_time[outbound2]: 3600
  sql_dont_try_update: true
  sql_optimize_clauses: true
 
  sql_preprocess: minb = 1000
 
  Regards
 
  -- 
  Daniel Levy
 
 
  ___
  pmacct-discussion mailing list
  http://www.pmacct.net/#mailinglists
  
 
  ___
  pmacct-discussion mailing list
  http://www.pmacct.net/#mailinglists


___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] pmacct 0.12.0 released !

2010-02-16 Thread Paolo Lucente 
VERSION.
0.12.0


DESCRIPTION.
pmacct is a small set of passive network monitoring tools to
measure, account, classify, aggregate and export IPv4 and IPv6
traffic; a pluggable and flexible architecture allows to store
collected network data into memory tables or SQL (MySQL, SQLite,
PostgreSQL) databases and export them through NetFlow or sFlow
protocols to remote collectors. pmacct supports fully customizable
historical data breakdown, sampling, filtering and tagging and
triggers. Libpcap, Netlink/ULOG, sFlow v2/v4/v5 and NetFlow v1/
v5/v7/v8/v9 are supported, both unicast and multicast. Also, a
client program makes it easy to export data to tools like RRDtool,
GNUPlot, Net-SNMP, MRTG, and Cacti.


HOMEPAGE.
http://www.pmacct.net/


DOWNLOAD.
http://www.pmacct.net/pmacct-0.12.0.tar.gz


CHANGELOG.
+ 'is_symmetric' aggregation primitive has been implemented: aimed
  at easing detection of asymmetric traffic. It's based on rule
  definitions supplied in a 'bgp_is_symmetric_map' map, reloadable
  at runtime.
+ A new 'bgp_daemon_allow_file' configuration directive allows to
  specify IP addresses that can establish a BGP session with the
  collector's BGP thread. Many thanks to Erik van der Burg for
  contributing the idea.
+ 'nfacctd_ext_sampling_rate' and 'sfacctd_ext_sampling_rate' are
  introduced: they flag the daemon that captured traffic is being
  sampled. Useful to tackle corner cases, ie. the sampling rate
  reported by the NetFlow/sFlow agent is missing or incorrect.
+ The 'bgp_follow_nexthop' feature has been extended so that extra
  IPv4/IPv6 prefixes can be supplied. Up to 32 IP prefixes are now
  supported and a warning message is generated whenever a supplied
  string fails parsing.
+ Pre-Tagging: implemented 'src_local_pref' and 'src_comms' keys.
  These allow tagging based on source IP prefix local_pref (sourced
  from either a map or BGP, ie. 'bgp_src_local_pref_type: map',
  'bgp_src_local_pref_type: bgp') and standard BGP communities.
+ Pre-Tagging: 'src_peer_as' key was extended in order to match on
  BGP-sourced data (bgp_peer_src_as_type: bgp).
+ Pre-Tagging: introduced 'comms' key to tag basing on up to 16
  standard BGP communities attached to the destination IP prefix.
  The lookup is done against the BGP RIB of the exporting router.
  Comparisons can be done in either match-any or match-all fashion;
  xidDocumentation and examples updated.
! fix, util.c: load_allow_file(), empty allow file was granting a
  connection to everybody being confused with a 'no map' condition.
  Now this case is properly recognized and correctly translates in
  a reject all clause.
! fix, sql_common.c: log of NetFlow micro-flows to a SQL database
  (nfacctd_sql_log directive) was not correctly getting committed
  to the backend, when sql_history was disabled.
! fix, mysql|pgsql|sqlite_plugin.c: 'flows' aggregation primitive
  was not suitable to mix-and-match with BGP related primitives
  (ie. peer_dst_as, etc.) due to an incorrect check. Many thanks
  to Zenon Mousmoulas for the bug report.
! fix, pretag_handlers.c: tagging against NetFlow v9 4-bytes in/out
  interfaces was not working properly. Thanks to Zenon Mousmoulas
  for reporting the issue.


NOTES.


Cheers,
Paolo

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists