Re: [pmacct-discussion] monitoring multiple network interfaces
Hi Ross, This same setup works well on a Linux for example. I see you daemonize both pmacctd daemons. Can you please keep them in foreground instead (daemonize: false) so that you can catch what happens? Cheers, Paolo On Thu, Sep 23, 2010 at 01:06:58PM -0700, Ross Lawrie wrote: Hi, I'm trying to set pmacct up on an OpenBSD 4.7 server running PF for its firewall such that I can monitor traffic on each of my networks. From the articles I've read in the wiki and in mailing lists the easiest way to do this is to run multiple instances of pmacctd. The problem I'm having is that processes after the first one don't seem to generate data. For example, using the following two commands to start pmacctd: /usr/local/sbin/pmacctd -f /etc/pmacct/pmacctd_dmz.conf /usr/local/sbin/pmacctd -f /etc/pmacct/pmacctd_internal.conf Here are the associated config files: ! ! pmacctd_dmz.conf ! debug: false daemonize: true interface: em1 plugin_buffer_size: 2048 plugin_pipe_size: 2048000 networks_file: /etc/pmacct/networks.def plugins: memory[dmzin], memory[dmzout] aggregate[dmzin]: dst_host aggregate[dmzout]: src_host imt_path[dmzin]: /tmp/dmz-in.pipe imt_path[dmzout]: /tmp/dmz-out.pipe ! ! pmacct_internal.conf ! debug: false daemonize: true interface: ste0 plugin_buffer_size: 2048 plugin_pipe_size: 2048000 networks_file: /etc/pmacct/networks.def plugins: memory[intin], memory[intout] aggregate[intin]: dst_host aggregate[intout]: src_host imt_path[intin]: /tmp/internal-in.pipe imt_path[intout]: /tmp/internal-out.pipe Using pmacct -s -p pipe I'm able to see data for whichever I start first, however I get errors on the second started. Sometimes one or both of the pipes belonging to the second process aren't created, if they are created I get an error when trying to read from them: pmacct -s -p /tmp/dmz-out.pipe INFO: Connection refused while trying to connect to '/tmp/dmz-out.pipe' Hopefully someone can see whatever I'm missing here. The version of pmacct installed is 0.12.0 in case that's relevant. Thanks, Ross Lawrie ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Source port column name depends on database
Hi Chris,
Agree. I seem to reckon this legacy issue is limited to the TCP/UDP
ports only and i'm thinking perhaps the best way to approach it is
to issue a true/false config switch, ie. sql_table_compat, for the
purpose. But for consistency with the rest, these fields should be
aligned to port_src and port_dst. Agree?
Cheers,
Paolo
On Mon, Sep 13, 2010 at 11:14:18AM +0200, Chris Wilson wrote:
Hi all,
We just had a bug report in pmGraph because it assumed that the source
port database column was called src_port always, as it is in MySQL. The
user is using a postgres database, and it appears that the column is
called port_src there instead:
if (!strcmp(config.type, mysql) || !strcmp(config.type, sqlite3))
{
strncat(insert_clause, src_port, SPACELEFT(insert_clause));
strncat(where[primitive].string, src_port=%u,
SPACELEFT(where[primitive].string));
}
else {
strncat(insert_clause, port_src, SPACELEFT(insert_clause));
strncat(where[primitive].string, port_src=%u,
SPACELEFT(where[primitive].string));
}
I would be much happier writing database-independent code around
pmacct if it didn't do things like this.
I understand that there is a backwards compatibility issue with changing
it, but perhaps it could be done in a new version of the mysql or postgres
schema?
Cheers, Chris.
--
Aptivate | http://www.aptivate.org | Phone: +44 1223 760887
The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES
Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] duplicates...
Hi Thomas, What matters is really how many 'src_host, dst_host' aggregates are generated out of the traffic. Predicting for me is difficult but you can turn sql_dont_try_update to false (so that it doesn't complain about duplicates anymore for the sake of this test) and see whether during the sql_refresh_time time-span you are pushing more than 256K aggregates to the database. Verification is to run a query like the following against the database: SELECT COUNT(*) AS aggregates, stamp_inserted FROM acct GROUP BY stamp_inserted; Cheers, Paolo On Sun, Sep 12, 2010 at 05:59:20PM +, Thomas Raabo - ZitCom A/S wrote: r...@mgmt5 examples]# top top - 19:31:01 up 4 days, 9:24, 5 users, load average: 0.12, 0.08, 0.07 Tasks: 202 total, 1 running, 201 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 24675552k total, 3611064k used, 21064488k free, 312036k buffers Swap: 3124632k total, 260k used, 3124372k free, 1490888k cached We are running about 400mbit of traffik average with netflow exporting 6000 flow/sec What sql_cache_entries do you recommend? Thomas -Oprindelig meddelelse- Fra: pmacct-discussion-boun...@pmacct.net [mailto:pmacct-discussion-boun...@pmacct.net] P? vegne af Paolo Lucente Sendt: 12. september 2010 01:32 Til: pmacct-discussion@pmacct.net Emne: Re: [pmacct-discussion] duplicates... Hi Thomas, This very likely answers why you are not seeing all you expect in the database. Reason you get duplicates is pmacct is trying to insert into the database more than once per 600 seconds (as per your configuration) and sql_dont_try_update is set to true. So quickest work-around is to set sql_dont_try_update to false; but i guess it's good to spend some efforts to understand why pmacct is behaving like that. The cause safe action gives a pointer and it's due to one of the two following reasons: * There are too many 'src_host, dst_host' aggregates for the configured sql_cache_entries value. This should be excluded as only 500-600 entries are to be inserted while the config would allow for sql_cache_entries entries (256K). But, it could be it stops on the first of a series of INSERTs, so there is more data to dump. Knowing your environment, you can ultimately comment whether the 256K value is good or not. * The system is short of memory (or the OS limits the amount of memory the pmacct process(es) can allocate); so at some stage pmacct asks for a new chunk of memory to insert a new entry in the SQL cache and it's not able to get it; this results in pmacct purging the cache content in an effort to make room to new data. Please check whether this can be the case. Let me know. Cheers, Paolo On Sat, Sep 11, 2010 at 05:49:17PM +, Thomas Raabo - ZitCom A/S wrote: Once every 10 min iam getting some errors DEBUG ( default/mysql ): purging process (CAUSE: safe action) ( default/mysql ) *** Purging cache - START *** DEBUG ( default/mysql ): 568 VALUES statements sent to the MySQL server. ERROR ( default/mysql ): Duplicate entry '0-unknown-0:0:0:0:0:0-0:0:0:0:0:0-0-0-0-188.176.68.71-194.182.12' for key 'PRIMARY' DEBUG ( default/mysql ): purging process (CAUSE: safe action) ( default/mysql ) *** Purging cache - START *** DEBUG ( default/mysql ): 569 VALUES statements sent to the MySQL server. ERROR ( default/mysql ): Duplicate entry '0-unknown-0:0:0:0:0:0-0:0:0:0:0:0-0-0-0-91.100.100.103-94.231.97' for key 'PRIMARY' DEBUG ( default/mysql ): purging process (CAUSE: safe action) ( default/mysql ) *** Purging cache - START *** DEBUG ( default/mysql ): 565 VALUES statements sent to the MySQL server. ERROR ( default/mysql ): Duplicate entry '0-unknown-0:0:0:0:0:0-0:0:0:0:0:0-0-0-0-94.231.102.73-95.166.47.' for key 'PRIMARY' My config ! nfacctd configuration example ! ! Did you know CONFIG-KEYS contains the detailed list of all configuration keys ! supported by 'nfacctd' and 'pmacctd' ? ! debug: true !daemonize: true aggregate: tag, src_host, dst_host plugin_pipe_size: 4096 plugin_buffer_size: 40960 nfacctd_port: 9000 ! nfacctd_time_secs: true nfacctd_time_new: true plugins: mysql sql_db: pmacct sql_table: acct sql_table_version: 2 sql_passwd: * sql_user: root sql_refresh_time: 600 sql_multi_values: 10 sql_dont_try_update: true sql_cache_entries: 256000 ! sql_optimize_clauses: true sql_history: 10m sql_history_roundoff: mh Thomas ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http
Re: [pmacct-discussion] duplicates...
Hi Thomas, This very likely answers why you are not seeing all you expect in the database. Reason you get duplicates is pmacct is trying to insert into the database more than once per 600 seconds (as per your configuration) and sql_dont_try_update is set to true. So quickest work-around is to set sql_dont_try_update to false; but i guess it's good to spend some efforts to understand why pmacct is behaving like that. The cause safe action gives a pointer and it's due to one of the two following reasons: * There are too many 'src_host, dst_host' aggregates for the configured sql_cache_entries value. This should be excluded as only 500-600 entries are to be inserted while the config would allow for sql_cache_entries entries (256K). But, it could be it stops on the first of a series of INSERTs, so there is more data to dump. Knowing your environment, you can ultimately comment whether the 256K value is good or not. * The system is short of memory (or the OS limits the amount of memory the pmacct process(es) can allocate); so at some stage pmacct asks for a new chunk of memory to insert a new entry in the SQL cache and it's not able to get it; this results in pmacct purging the cache content in an effort to make room to new data. Please check whether this can be the case. Let me know. Cheers, Paolo On Sat, Sep 11, 2010 at 05:49:17PM +, Thomas Raabo - ZitCom A/S wrote: Once every 10 min iam getting some errors DEBUG ( default/mysql ): purging process (CAUSE: safe action) ( default/mysql ) *** Purging cache - START *** DEBUG ( default/mysql ): 568 VALUES statements sent to the MySQL server. ERROR ( default/mysql ): Duplicate entry '0-unknown-0:0:0:0:0:0-0:0:0:0:0:0-0-0-0-188.176.68.71-194.182.12' for key 'PRIMARY' DEBUG ( default/mysql ): purging process (CAUSE: safe action) ( default/mysql ) *** Purging cache - START *** DEBUG ( default/mysql ): 569 VALUES statements sent to the MySQL server. ERROR ( default/mysql ): Duplicate entry '0-unknown-0:0:0:0:0:0-0:0:0:0:0:0-0-0-0-91.100.100.103-94.231.97' for key 'PRIMARY' DEBUG ( default/mysql ): purging process (CAUSE: safe action) ( default/mysql ) *** Purging cache - START *** DEBUG ( default/mysql ): 565 VALUES statements sent to the MySQL server. ERROR ( default/mysql ): Duplicate entry '0-unknown-0:0:0:0:0:0-0:0:0:0:0:0-0-0-0-94.231.102.73-95.166.47.' for key 'PRIMARY' My config ! nfacctd configuration example ! ! Did you know CONFIG-KEYS contains the detailed list of all configuration keys ! supported by 'nfacctd' and 'pmacctd' ? ! debug: true !daemonize: true aggregate: tag, src_host, dst_host plugin_pipe_size: 4096 plugin_buffer_size: 40960 nfacctd_port: 9000 ! nfacctd_time_secs: true nfacctd_time_new: true plugins: mysql sql_db: pmacct sql_table: acct sql_table_version: 2 sql_passwd: * sql_user: root sql_refresh_time: 600 sql_multi_values: 10 sql_dont_try_update: true sql_cache_entries: 256000 ! sql_optimize_clauses: true sql_history: 10m sql_history_roundoff: mh Thomas ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] [nfacctd] aggregate_filter
Hi Borys, I can confirm you filtering on TCP flags is currently not possible in both nfacctd and sfacctd. Such implementation is not major work and can be done pretty quickly - i can drop you an email privately when the code is available in the CVS so you can test it working? Apart from the above, just to be on the same page, you are already sure TCP flags in NetFlow will work for your scenario? NetFlow ORs all the TCP flags seen for a certain uni-directional flow up to the moment it's set for expiration. So if counting flags, for example, you should not seek for a full match but rather test with a logical AND. Cheers, Paolo On Tue, Aug 24, 2010 at 05:47:33PM +0200, Borys Owczarzak wrote: Hi! I would like to logging some traffic to database: TCP packets with flags SYN or FIN or RST to table kancelaria UDP packets (1-of-100) to table dupa ICMP packets (1-of-100) to table icmp. My test configuration: !START CONFIGURATION daemonize: true pidfile: /var/run/pmacctd.pid syslog: daemon !FOR PMACCTD USE PROMISC: TRUE !promisc: true !FOR NFACCTD USE NFACCTD_PORT: 2055 nfacctd_port: 2055 interface: eth2 aggregate[kancelaria]: src_host, dst_host, src_port, dst_port, tcpflags aggregate[dupa]:src_host, dst_host, src_port, dst_port aggregate[icmpe]:src_host, dst_host, src_port, dst_port aggregate_filter[kancelaria]: tcp[tcpflags] (tcp-syn) != 0 || tcp[tcpflags] (tcp-fin) !=0 || tcp[tcpflags] (tcp-rst) !=0 aggregate_filter[dupa]: udp aggregate_filter[icmpe]: icmp sampling_rate[icmpe]:100 sampling_rate[dupa]:100 plugins: mysql[kancelaria], mysql[dupa], mysql[icmpe] sql_db: pmacct sql_optimize_clauses: true sql_table[dupa]: dupa sql_table[kancelaria]: kancelaria sql_table[icmpe]: icmpe sql_user: root sql_passwd: regedit4 sql_dont_try_update: true sql_refresh_time: 1 sql_history: 1d !STOP CONFIGURATION I try it at pmacctd and nfacctd. When I use pmacct everything works great. But problem is with nfacctd. It does not save data to kancelaria table. When I erase from configuration aggregate_filter[kancelaria]: when it saves packet. Somebody know why nfacct has got problem with these aggregate_filter? PS aggregate_filter[dupa] and aggregate_filter[icmpe] wors correctly. Kind regards Borys Owczarzak ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] cockpit error instrumenting www.pmacct.net/docs/cacti.html?
Hi Larry, Wanted just to say briefly the best person to support you is the author of that document (which btw is not on this list and not really reachable recently). I can't help you with that as i'm not too much into Cacti and pmacct seems to return expected results. I'm certainly available to collect feedback on that document and edit it accordingly. Perhaps is an idea to move it to the wiki so that everybody is potentially able to modify it. Cheers, Paolo On Thu, Aug 12, 2010 at 09:39:38AM -0500, Larry Vaden wrote: First, thanks for the great software. 2nd, feel free to direct me to another list if that should be the case. Following the example at http://www.pmacct.net/docs/cacti.html, the result is the infamous RED X (missing file) rather than the expected graph. Centos 4.8, up to date, using pmacct-0.12.3 and cacti-0.8.7.e (g won't run at the moment on CentOS 4.8). nfacctd is started with (as root) /usr/local/sbin/nfacctd -f /usr/local/etc/pmacctd.conf and the contents of /usr/local/etc/pmacctd.conf follow at the end of this post. The permission on the /tmp/in.pipe and /tmp/out.pipe are as follows [r...@saguaro cacti]# ls -l /tmp total 16 drwxr-xr-x 3 root root 4096 Aug 6 12:27 cactiupgrade srwxrwxrwx 1 root root 0 Aug 12 09:18 in.pipe srwxrwxrwx 1 root root 0 Aug 12 09:18 out.pipe 'pmacct -s -p /tmp/in.pipe' and 'pmacct -s -p /tmp/out.pipe' return the expected results when run as root. [cactiu...@saguaro ~]$ pmacct -s -p /tmp/in.pipe and [cactiu...@saguaro ~]$ pmacct -s -p /tmp/out.pipe and (after 'su - apache') -bash-3.00$ pmacct -s -p /tmp/in.pipe and -bash-3.00$ pmacct -s -p /tmp/out.pipe produce the expected results. There is no sign of the expected files in /var/www/html/cacti/rra: [r...@saguaro cacti]# ls -l /var/www/html/cacti/rra/localhost* -rw-r--r-- 1 cactiuser apache 3651280 Aug 12 09:30 /var/www/html/cacti/rra/localhost_load_1min_5.rrd -rw-r--r-- 1 cactiuser apache 1218368 Aug 12 09:30 /var/www/html/cacti/rra/localhost_mem_buffers_3.rrd -rw-r--r-- 1 cactiuser apache 1218368 Aug 12 09:30 /var/www/html/cacti/rra/localhost_mem_swap_4.rrd -rw-r--r-- 1 cactiuser apache 1218368 Aug 12 09:30 /var/www/html/cacti/rra/localhost_proc_7.rrd -rw-r--r-- 1 cactiuser apache 8517104 Aug 6 12:20 /var/www/html/cacti/rra/localhost_rrdsprocessed_8.rrd -rw-r--r-- 1 cactiuser apache 2434824 Aug 6 12:20 /var/www/html/cacti/rra/localhost_total_9.rrd -rw-r--r-- 1 cactiuser apache 1218368 Aug 12 09:30 /var/www/html/cacti/rra/localhost_users_6.rrd [r...@saguaro cacti]# The contents of /usr/local/etc/pmacctd.conf: [r...@saguaro cacti]# cat /usr/local/etc/pmacctd.conf ! ! pmacct: In + Out ! debug: false daemonize: true interface: eth0 plugin_buffer_size: 2048 plugin_pipe_size: 2048000 networks_file: /usr/local/etc/networks.def ! temporarily remove mysql from plugins: plugins: memory[in], memory[out] aggregate[in]: dst_net aggregate[out]: src_net imt_path[in]: /tmp/in.pipe imt_path[out]: /tmp/out.pipe ! nfacctd_port: 2055 ! sql_db: pmacct sql_table: acct sql_table_version: 7 sql_passwd: redacted sql_user: pmacct ! from wiki OfficialExamples for sql optimization sql_refresh_time: 300 sql_history: 5m sql_history_roundoff: m sql_dont_try_update: true Again, thanks for a great software! Kind regards/ldv Larry Vaden Internet Texoma, Inc. Serving Rural Texomaland Since 1995 We Care About Your Connection! ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] pmpgplay - what does it do?
Hi Chris, On Mon, Jul 26, 2010 at 10:03:15AM +1000, Chris wrote: So I'd be better off making these 2 match? sql_refresh_time: 60 sql_history: 5m Correct. For what it's worth the log seems to be aggregated on the history time. I only see the 5 min increments when I run pmpgplay in debug mode. Yes, this is because you get logged down the exact tuple that was going to be written to the database. And records in the database are fitted in time bins as per the sql_history parameter. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] pmpgplay - what does it do?
Hi Chris, The 'failsafe' mechanism (ie. writing to a backup database or to log files) kicks in upon receipt of an error code from the RDBMS API. So what you see in the log file should never be already in the database. Your specific configuration is tricky because you write to the RDBMS every 60 seconds but aggregate on 5 minutes time-bins and hence it's not possible to determine for sure whether a record is contributing or not to a certain aggregate. Only work-around i can think of for your case is to permanenly enable debug in pmacct (or logging in the RDBMS) so that it's possible, at any moment, to know which queries have been performed. This at the expense of slightly more resources. If using default SQL schemas and an INSERT-only scenario, to protect against duplicates, you could have simply appended the '-i' option to pmpgplay to disable UPDATE SQL queries. Logics is explained here: http://wiki.pmacct.net/CustomizingTheSqlIndexes Cheers, Paolo On Fri, Jul 23, 2010 at 11:53:53AM +1000, Chris wrote: Hi, I'm using pmacct-0.12.0rc3 with postgresql. I've been trying to work out what pmpgplay does when it looks at a log file. I couldn't find much in my searches that go into the details of how it does things. It seems it generates a bunch of sql, but does it check to see if that record exists already before applying the change? I'm worried about replaying the log and it double-recording data (ie the data is already recorded in the db, and then replaying the log does it again), or will it only apply new records that it can't find in the db? I have a missing chunk of data from my database (about 1.5 hrs worth) that I can see in the logs, but the log is surrounded by data that could be in the database (I haven't checked each individual log entry). Relevant pmacct.conf info: plugin_pipe_size:1024000 plugin_buffer_size:8192 imt_buckets: 65537 imt_mem_pools_size: 1024000 sql_table_version:1 sql_refresh_time: 60 sql_history: 5m Any information is much appreciated, thanks! -- Postgresql php tutorials http://www.designmagick.com/ ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] [nfacctd] IPv4/IPv6 Accounting into MySQL
Hi Carsten, A couple of things come to mind: * Was pmacct compiled with --enable-ipv6 ? * Are 'aggregate_filter' directives including the IPv6 subnets you want to account for? * If, in parallel to nfacctd, you fire up a Wireshark (or tshark); by decoding the NetFlow v9 datagrams generated by the probe, can you spot the IPv6 records being sent over to the collector? Basing on the answers to the above questions, we can drill it down further. Cheers, Paolo On Fri, Jul 23, 2010 at 07:00:00PM +0200, InterNetX - Carsten Schoene wrote: Hello List, i've setup a nfacctd instance which writes data into mysql, the configuration looks like this: daemonize: true pidfile: /var/run/nfacctd.pid syslog: daemon plugin_pipe_size: 1024 plugin_buffer_size: 10240 nfacctd_ip: %MY_NFACCTD_HOST% nfacctd_port: 5552 nfacctd_allow_file:/etc/pmacct/nfacctd.allow nfacctd_time_new: true plugins: mysql[in], mysql[out] aggregate[in]: dst_host aggregate[out]: src_host aggregate_filter[in]: dst net %MYNETWORK_A%/18 or dst net %MYNETWORK_B%/19 aggregate_filter[out]: src net %MYNETWORK_A%/18 or src net %MYNETWORK_B%/19 sql_host:localhost sql_user:pmacct sql_passwd:* sql_optimize_clauses: true sql_refresh_time: 300 sql_history: 5m sql_dont_try_update: true sql_history_roundoff: m sql_table_version: 7 sql_table:acct_v7_%Y%m%d_%H00 sql_table_schema:/etc/pmacct/mysql_v7_table.sql sql_recovery_logfile: /raid/nfacctd_recovery_log sql_multi_values: 16384000 CREATE TABLE IF NOT EXISTS acct_v7_%Y%m%d_%H00 ( id INT UNSIGNED NOT NULL AUTO_INCREMENT, ip_src CHAR(45) NOT NULL, ip_dst CHAR(45) NOT NULL, packets INT UNSIGNED NOT NULL, bytes BIGINT UNSIGNED NOT NULL, flows INT UNSIGNED NOT NULL, stamp_inserted DATETIME NOT NULL, stamp_updated DATETIME, PRIMARY KEY (id), KEY `src_dst_stamps` (ip_src, ip_dst, stamp_inserted, stamp_updated), KEY(ip_src), KEY(ip_dst) ) ENGINE=MyISAM; We are running nprobe which exports netflow v9 data using the following netflow template: %IPV6_SRC_ADDR %IPV6_DST_ADDR %IPV4_SRC_ADDR %IPV4_DST_ADDR %LAST_SWITCHED %FIRST_SWITCHED %IN_BYTES %OUT_BYTES %IN_PKTS %OUT_PKTS %L4_SRC_PORT %L4_DST_PORT %PROTOCOL %TCP_FLAGS %IP_PROTOCOL_VERSION %SRC_TOS %SRC_AS %DST_AS %IPV6_SRC_MASK %IPV6_DST_MASK %SRC_MASK %DST_MASK The problem is, that there are no IPv6 entrys written into the tables, only ipv4 addresses. Any idea or help would be great Regards -- Carsten Sch?ne Leiter Rechenzentrum InterNetX GmbH Maximilianstr. 6 93047 Regensburg Tel. +49 941 59559-480 Fax +49 941 59579-051 www.internetx.com www.facebook.com/InterNetX www.twitter.com/InterNetX Gesch?ftsf?hrer/CEO: Thomas M?rz Amtsgericht Regensburg, HRB 7142 ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Netflow and 'ip' key in pretag.map
Hi Damian, Are you running BGP? Would it be feasible for you to past BGP feed(s) into pmacct (granted you run a recent 0.12 release or can upgrade to that)? Idea is you can attach BGP standard communities to IP prefixes as they are advertised or re-distributed into your network. Because comunities are supported as primitives in pmacct (and you can filter what you want to see and what should be thrashed instead), you might even avoid the whole 'id2' section. Just an idea. In case BGP is not an option: total number of mappings defined isn't an issue. In this sense, to store 2K entries you should make use of the pre_tag_map_entries directive (and set it above 2K). Not very nice to see but it works and should be straightforward to automate. It's important though to keep the walks through the maps as short as possible; so, id=1 ip=X jeq=user_X id=1 ip=Y jeq=user_Y ... id=1 ip=Z jeq=user_Z ! id=userid ip=X filter='...' label=user_X jeq=traffic_type_X id=userid ip=Y filter='...' label=user_Y jeq=traffic_type_Y ... id=userid ip=Z filter='...' label=user_Z jeq=traffic_type_Z ! id2=1 ip=X filter='...' label=traffic_type_X jeq=next ... id2=1 ip=Y filter='...' label=traffic_type_Y jeq=next ... id2=1 ip=Z filter='...' label=traffic_type_Z jeq=next ... Cheers, Paolo On Wed, Jul 21, 2010 at 11:33:38AM +1200, Damian Kissick wrote: [ ... ] So finally to the crux of the original question; on the single-router setup, netflow on the appropriate ingress and egress interfaces works and all the traffic is marked appropriately with a userid (tag) and traffic type (tag2). But I know that upon adding the additional routers for the other traffic users, I will currently have to duplicate the id2 mappings for each netflow agent's IP. I am trying to find a way around that to keep the pretag.map efficient (or maybe ~500 networks x 4 netflow agents for 2000 mappings is not actually too bad?) One solution that I am contemplating is to move away from netflow and enable sflow on our core switches which would keep the required duplication of mappings down. Or, in the same way our current system collects traffic data, use mirror ports (on the core switches) and then use the pmacctd daemon instead. I suspect I am overlooking some more obvious solutions so I really appreciate pointers on this and if you or anyone sees other issues or better design tips for this, then I welcome the feedback. Cheers, - Damian ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] How many ips can pmacctd monitorize
Hi, Try performing a locked access to the memory table. This can be done by appending a '-l' option to the command, ie. pmacct -s -l -p Let me know. Cheers, Paolo On Thu, Jul 22, 2010 at 10:35:18AM +, Jose Joaquin Anton Herrerias wrote: I was Reading CONFIG-KEYS and internals documents but im doing something wrong because the pmacct is stolen data. When I look into the table for the traffic of one ip I see that the accumulate is stolen. # pmacct -s -p /tmp/host_in.pipe |grep 184.17 XX.XXX.184.1751277 14929314 # pmacct -s -p /tmp/host_in.pipe |grep 184.17 XX.XXX.184.1753033 15176927 # pmacct -s -p /tmp/host_in.pipe |grep 184.17 XX.XXX.184.172803477449 As you can see the bytes column is minor that before. The last time get 477449 when 10 seconds before has 15176927. My config is: ! ! pmacct: In + Out ! debug: false daemonize: true interface: eth1 promisc: true syslog: kern imt_mem_pools_size: 12000 imt_mem_pools_number: 20 plugin_buffer_size: 20480 networks_file: /etc/pmacct/networks2.def plugins: memory[net_in], memory[net_out], memory[host_in], memory[host_out] aggregate[net_in]: dst_net aggregate[net_out]: src_net aggregate[host_in]: dst_host aggregate[host_out]: src_host imt_path[net_in]: /tmp/net_in.pipe imt_path[net_out]: /tmp/net_out.pipe imt_path[host_in]: /tmp/host_in.pipe imt_path[host_out]: /tmp/host_out.pipe The traffic that I want to monitorize is 70 Mb out and 30 Mb In, is not to much but im getting stolen data. Thanks for the help and sorry for my bad English. Best regards. Jose Joaqu??n Ant??n Herrer??as J jan...@abserver.es ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Error for building pmacct 0.12 from source
Hi Rodolfo, In addition to what Brent correctly said you might want to double check whether you have development kit for MySQL installed at all (libraries and headers). A chance can be you have got installed only, say, MySQL client, server and tools. Cheers, Paolo On Tue, Jun 22, 2010 at 12:01:03PM -0400, rodolfo mondion wrote: I ve downloaded the latest version of pmacct and trying with 2 linux distributions (ubuntu server 9.04 and 10 for i386) i get the same errors related to the librarie libmysqlclient when i make the ./configure --enable-mysql y get: checking default locations for libmysqlclient ... not found checking for mysql_real_connect in lmysqlclient no configure: error: ERROR: missing Mysql client library. Refer to http://www.mysql.com I ve installed at the beginning of the server installation the LAMP option, i dont know if i must install mysql manually or if theres a problem with my mysql version (5.0.75) Thanks for ur help -- Rodolfo Mondion R. ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Help as the sum of certain ips
Hi Rafael, Advice is to check out Q8 from FAQS, EXAMPLES document chapter III, 'sql/README.pgsql' for initial SQL table setup. It should be enough to work out initial configs. Cheers, Paolo On Thu, Jun 17, 2010 at 05:14:12PM -0300, Rafael Stein wrote: Hello to all, I'm starting to use the system now and wanted to see how to generate the sum (input, output) of traffic from certain ips and play in postgres database. My idea is to keep updating the value in the database and not insert values. I appreciate the help. Rafael ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] data only in one direction
Hi Morgan, INTERNALS document is part of the pmacct distribution tarball that you download (ie. pmacct-0.12.2.tar.g). Look into docs/. Cheers, Paolo On Fri, Jun 18, 2010 at 08:35:53AM +0200, Morgan Sellier wrote: Hi, Thanks for your help ! I will test it today but in the documentation I see : Read INTERNALS 'Memory table plugin' chapter for further details. and I really don't know where is this chapter. Cheers, Morgan ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] inet Error
Hi Jonathan, It's a bit hard to say with this information. Can you post your config? Are you using any of the default SQL schemas? Which options did you compile the package? Cheers, Paolo On Fri, Jun 11, 2010 at 01:11:23PM +1000, Jonathan Gleeson wrote: Hi Guys, Anyone got any idea on this one, ERROR ( in/pgsql ): ERROR: invalid input syntax for type inet: LINE 1: ...5201)::Timestamp, ABSTIME(1276164900)::Timestamp, '', 4, 350... Happens every now and then not at a regular interval but a few times a day, when it does occur i loose around 1hours data. it looks like its trying to inset an empty set instead of null. any suggestions would be really helpful Regards, ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] pmacct 0.12.2 released !
VERSION. 0.12.2 DESCRIPTION. pmacct is a small set of passive network monitoring tools to account, classify, aggregate and export IPv4 and IPv6 traffic; a pluggable architecture allows to store collected network data into memory tables or SQL (MySQL, SQLite, PostgreSQL) databases and export them through NetFlow or sFlow protocols to remote collectors. pmacct supports fully customizable historical data breakdown, BGP correlation, sampling, filtering, tagging and triggers. Libpcap, Netlink/ULOG, sFlow v2/v4/v5 and NetFlow v1/ v5/v7/v8/v9 are supported, both unicast and multicast. It also supports replication of incoming NetFlow and sFlow datagrams. A client tool makes it easy to export data to tools like RRDtool, GNUPlot, Net-SNMP, MRTG, and Cacti. HOMEPAGE. http://www.pmacct.net/ DOWNLOAD. http://www.pmacct.net/pmacct-0.12.2.tar.gz CHANGELOG. + A new 'tee' plugin is introduced bringing both NetFlow and sFlow replication capabilities to pmacct. It supports transparent mode (tee_transparent), coarse-grained filtering capabilities via the Pre-Tagging infrastructure. Quickstart guide is included as part of the EXAMPLES file (chapter XII). + nfprobe, sfprobe: introduced support for export of the BGP next-hop information. Source data selection for BGP next-hop is being linked to [pmacctd_as|uacctd_as] configuration directive. Hence it must be set to 'bgp' in order for this feature to work. + nfprobe, sfprobe, BGP daemon: new set of features (nfprobe_ipprec, sfprobe_ipprec, bgp_daemon_ipprec) allows to mark self-originated sFlow, NetFlow and BGP datagrams with the supplied IP precedence value. + peer_src_ip (IP address of the NetFlow emitter, agent ID of the sFlow emitter) and peer_dst_ip (BGP next-hop) can now be filled from NetFlow/sFlow protocols data other than BGP. To activate the feature nfacctd_as_new/sfacctd_as_new have to be 'false' (default value), 'true' or 'file'. + print plugin: introduced support for Comma-Separated Values (CSV) output in addition to formatted-text. A new print_output feature allows to switch between the two. + pmacctd: improved 802.1ad support. While recursing, outer VLAN is always reported as value of the 'vlan' primitive. ! fix, pmacctd: 802.1p was kept integral part of the 'vlan' value. Now a 0x0FFF mask is applied in order to return only the VLAN ID. ! fix, pkt_handlers.c: added trailing '\0' symbol when truncating AS-PATH and BGP community strings due to length constraints. ! fix, sql_common.c: maximum SQL writers warning message was never reached unless a recovery method is specifited. Thanks to Sergio Charpinel Jr for reporting the issue. ! fix, MySQL and PostgreSQL plugins: PGRES_TUPLES_OK (PostgreSQL) and errno 1050 (MySQL) are now considered valid return codes when dynamic tables are involved (ie. sql_table_schema). Thanks to Sergio Charpinel Jr for his support. ! fix, BGP daemon: pkt_bgp_primitives struct has been explicitely 64-bit aligned. Mis-alignment was causing crashes when buffering was enabled (plugin_buffer_size). Verified on Solaris/sparc. NOTES. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] PostgreSQL large database
Hi Sergio, It looks those processes are locked out of the table they want to write to. In MySQL you can check this kind of stuff with a SHOW PROCESSLIST; the PostgreSQL equivalent should be SELECT * FROM pg_stat_activity. Its output might very well shed some light. Just btw, the number of pmacct processes allowed to queue up is settable via the sql_max_writers configuration directive. Cheers, Paolo On Thu, May 13, 2010 at 10:04:14AM -0300, Sergio Charpinel Jr. wrote: Hi, After less than one day running pmacct, my database grew from 1.5G to 15G. At this point I get a lot of postmaster and nfacctd process, and none of them seems to be using CPU and not writting in DB. 6774 root 15 0 160m 103m 916 S 0.0 0.9 0:04.04 nfacctd 6775 postgres 15 0 2210m 1.3g 1.3g S 0.0 11.8 0:31.50 postmaster 6784 root 15 0 160m 103m 916 S 0.0 0.9 0:04.19 nfacctd 6785 postgres 16 0 2210m 1.1g 1.1g S 0.0 9.9 0:23.77 postmaster 6794 root 15 0 160m 103m 916 S 0.0 0.9 0:02.81 nfacctd 6795 postgres 15 0 2210m 1.1g 1.1g S 0.0 9.4 0:20.57 postmaster 6804 root 15 0 160m 103m 916 S 0.0 0.9 0:02.04 nfacctd 6805 postgres 16 0 2210m 870m 867m S 0.0 7.4 0:14.03 postmaster 6819 root 15 0 160m 103m 916 S 0.0 0.9 0:02.64 nfacctd 6820 postgres 15 0 2210m 816m 813m S 0.0 7.0 0:13.14 postmaster 6829 root 15 0 160m 103m 916 S 0.0 0.9 0:01.17 nfacctd 6830 postgres 15 0 2210m 625m 622m S 0.0 5.3 0:09.63 postmaster 6861 root 15 0 160m 103m 916 S 0.0 0.9 0:01.28 nfacctd 6862 postgres 15 0 2210m 558m 556m S 0.0 4.8 0:08.16 postmaster 6889 root 15 0 160m 103m 916 S 0.0 0.9 0:00.33 nfacctd 6890 postgres 16 0 2210m 269m 267m D 0.0 2.3 0:03.16 postmaster [ ... ] ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] MySQL index performance
Hi Chris, I think it would make no semantic difference, but would increase MySQL performance with these table types, if the primary key listed stamp_inserted first instead of last. The change you propose, as you say, would not be impacting - but would you have any testing handy which confirms the hint? Another option to get a monotonically increasing key is to set a synthetic index, ie. a spare field defined as AUTO_INCREMENT, which needs to be complemented by some indexes of choice in order to be effective. I've recently spared some time to write this page on the wiki: http://wiki.pmacct.net/CustomizingTheSqlIndexes Comments? Perhaps an idea is to contribute the Implementation Notes section of the wiki with other takes on those topics (or new topics)? Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Expiring Orphan fragment
Hi, To wrap-up this thread - unless new details emerge: On Tue, May 04, 2010 at 11:53:59AM +, Paolo Lucente wrote: * one capturing some torrent traffic, so that i can replay it in a testbed and see if i can reproduce and validate the behaviour. Orphan fragments are really there. Action point on my side is to mark these log messages as debug rather than informational. * the other capturing some NetFlow packets delivered to nfacctd so that i can look into them with a packet analyzer. It should never happen that nfacctd is unable to parse NetFlow datagrams produced by nfprobe. The box was suffering sustained load due to the pmacctd process acting as NetFlow probe lacking of internal buffering. This in turn generated some packet loss. Adding the following lines to the configuration did help bringing back the load of the box to acceptable levels: plugin_buffer_size: 102400 plugin_pipe_size: 1024 Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Expiring Orphan fragment
Hi Sergio, Good point. I've just marked PGRES_TUPLES_OK as valid return code within the PG_create_dyn_table() function - and committed the change to the CVS. Can you please confirm it works OK for you? Cheers, Paolo On Mon, May 03, 2010 at 10:10:18AM -0300, Sergio Charpinel Jr. wrote: Paolo, I think the problem is in the SELECT. SELECT returns PGRES_TUPLES_OK, which means, a PGRES_COMMAND_OK plus result data. [ ... ] ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Expiring Orphan fragment
Hi Sergio, On Thu, Apr 29, 2010 at 11:23:35AM -0300, Sergio Charpinel Jr. wrote: Does anyone know what this mean? I getting a lot of them in pmacctd.log Expiring orphan fragment: ip_src=210.197.202.84 ip_dst=200.137.66.1 proto=17 id=8885 It means some IP fragments have been staying too long into the pmacctd fragment handler waiting for the missing pieces to come. If not in huge amounts, this can be sympthom of white noise on the public internet (scans, attacks, etc.). Regarding the topic, when using dynamic tables with postgreSQL, How can I verify if a table already exists before trying to create them, to avoid a lot of 'Table already exists' messages in nfacctd.log with dynamic table? I'm not sure there is something already cooked to circumvent that. It should be possible though with a few lines of PL/PGSQL, ie.: IF EXISTS (SELECT relname FROM pg_class WHERE relname='table') THEN ... ELSE ... ENDIF The simplest way should be to get the table name as parameter and embed the actual SQL table schema within the function declaration. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Low performance query
Hi Sergio, The sql_history drives temporal aggregation by splitting traffic data into time-bins. You configured 5 minutes time-bins. See the behaviour of the 'stamp_inserted' field, ie. do SELECT DISTINCT(stamp_inserted) FROM table ORDER BY stamp_inserted, to get this more clear. The roundoff thing does some nice time alignment. So, increasing the size of the time-bins will be certainly beneficial but that's not the entire win: important thing is understanding what you are trying to achieve by putting data into PostgreSQL and choose a suitable spatial aggregation method (the aggregate directive) for your task. Suitable here means: build a data-set fitting your project requirements with the (nearly) least possible amount of records. Then indexing. It gets useful speeding up SQL queries with conditions, ie. WHERE ip_src = 'x.x.x.x'. If that's the kind of query you run the most, try inserting in the SQL table schema a INDEX (ip_src), after the PRIMARY KEY line. Perhaps you might want to do the same with the ip_dst field. Cheers, Paolo On Tue, Apr 27, 2010 at 10:37:28PM -0300, Sergio Charpinel Jr. wrote: I'm getting low perfomance while selecting from psql and from FloX. For example, selecting * WHERE ip_src = 'x.x.x.x' How can I do a more compact aggregation? You mean by getting samples ? Actually, I did not understand very well sql_history and roundoff concepts, but increasing history would do the trick? Thanks for answering. Cheers. 2010/4/27 Paolo Lucente pa...@pmacct.net Hi Sergio, I don't know FloX very well - hence would be good information to know which specific SQL queries are performing bad. Maybe there is room to improve indexing. Is it also your goal to store every micro-flow into the SQL database? Any chance a more compact aggregation method would fit the bill? Cheers, Paolo On Tue, Apr 27, 2010 at 11:36:50AM -0300, Sergio Charpinel Jr. wrote: Hi, I'm using pmacct 0.12.1 with PostgreSQL 8.4 and FloX. i'm running pmacctd and nfacctd daemons (probe and collector). After 1 day running, I'm getting 1+min to make a query. Is there any config in pmacct and PostgreSQL that I can do improve this? Here is my config in pmacct: pmacctd.conf: daemonize: true aggregate: src_host, src_port, dst_host, dst_port, proto, tcpflags plugins: nfprobe nfprobe_receiver: x.x.x.x nfprobe_version: 9 interface: eth1 nfacctd.conf: daemonize: true pidfile: /var/run/nfacctd.pid aggregate: src_host,dst_host, src_port, dst_port, proto, tcpflags plugins: pgsql sql_table_version: 7 sql_db: x sql_host: x.x.x.x sql_passwd: xxx sql_user: x sql_refresh_time: 300 sql_history: 5m sql_history_roundoff: m sql_dont_try_update: true nfacctd_port: nfacctd_ip: In PostgreSQL, I did not change the default config in CentOS 5.4. Thanks in advance! -- Sergio Roberto Charpinel Jr. ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists -- Sergio Roberto Charpinel Jr. ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] sflow in_iface out_iface information wrong
Hi Bernd, If i get it correctly, you should be referring to the big 1073741823 and 2147483648 values in your SQL table. According to the sFlow (v5) specifications, these values are perfectly valid: 1073741823 = 0x3FFF: [ ... ] this is used in describing traffic which is not bridged, routed, or otherwise sent through the device being monitored by the agent, but which rather originates or terminates in the device itself. In the input field, this value is used to indicate packets for which the origin was the device itself (e.g. a RIP request packet sent by the device, if it is acting as an IP router). In the output field, this value is used to indicate packets for which the destination was the device itself (e.g. a RIP response packet (whether unicast or not) received by the device, if it is acting as an IP router). 2147483648 = 0x8000: indicates a packet sent to an unknown number of interfaces greater than 1. - broadcast or multicast come to mind. There are some further codes/information passed through the interface fields; i'd like to point you to [1] for a more complete reference - even though it says new codes might be issued over the time. Of course, with tagging you drive what you expect to see; while dumping information straight from sFlow into a primitive catches such additional things you either don't know about or won't expect to see. Cheers, Paolo [1] http://www.sflow.org/SFLOW-DATAGRAM5.txt On Tue, Apr 27, 2010 at 12:46:44PM +0200, Bernd Boom wrote: Hi everybody, is use the sflow collector and want to collect sflow data depending on the inteface information which should be possible since version 0.12.1. The importat columns are in_iface and out_iface as you can see in the aggregate part. - aggregate: tag,tag2,in_iface, out_iface,flows,tcpflags,vlan,tag,src_host,dst_hos t,src_port,dst_port,proto,src_mac,dst_mac,tos - The problem is that i receive the following (sometimes wrong) interface information, as you can seen from the database output. [ ... ] The correct interfaces are for example agent_id=323 and 24. I compile with --64bit and without this option, but with the same result. Are there any ideas why the interface information is sometimes wrong Although it is to my infomation not possible that there are in and out interface information in the same flow informatione (mostly the switches only support only (in-flowaccounting) as this switches! The version 0.12.0 works correct with these foundry switches with pretagging the interface information! many thanks Bernd Boom ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Low performance query
Hi Sergio, I don't know FloX very well - hence would be good information to know which specific SQL queries are performing bad. Maybe there is room to improve indexing. Is it also your goal to store every micro-flow into the SQL database? Any chance a more compact aggregation method would fit the bill? Cheers, Paolo On Tue, Apr 27, 2010 at 11:36:50AM -0300, Sergio Charpinel Jr. wrote: Hi, I'm using pmacct 0.12.1 with PostgreSQL 8.4 and FloX. i'm running pmacctd and nfacctd daemons (probe and collector). After 1 day running, I'm getting 1+min to make a query. Is there any config in pmacct and PostgreSQL that I can do improve this? Here is my config in pmacct: pmacctd.conf: daemonize: true aggregate: src_host, src_port, dst_host, dst_port, proto, tcpflags plugins: nfprobe nfprobe_receiver: x.x.x.x nfprobe_version: 9 interface: eth1 nfacctd.conf: daemonize: true pidfile: /var/run/nfacctd.pid aggregate: src_host,dst_host, src_port, dst_port, proto, tcpflags plugins: pgsql sql_table_version: 7 sql_db: x sql_host: x.x.x.x sql_passwd: xxx sql_user: x sql_refresh_time: 300 sql_history: 5m sql_history_roundoff: m sql_dont_try_update: true nfacctd_port: nfacctd_ip: In PostgreSQL, I did not change the default config in CentOS 5.4. Thanks in advance! -- Sergio Roberto Charpinel Jr. ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Duplicate entry problem.
Hi Sergey, Duplicates are a clear consequence of the urgent DB writer in conjunction with disabling UPDATE queries (sql_dont_try_update) and the configured (default, i guess) primary key. Urgent writers are elicited by shortage of entries available in the SQL cache. Having memory available, my advice is to increase the value of sql_cache_entries until it matches your traffic footprint (you don't see duplicates anymore). I'd start with simply selecting a prime number (see CONFIG-KEYS, INTERNAL documents) in the area of 256000 and otherwise go one order up. If additional memory is not an option i would advice to change the primary key to a simple auto-increment index. You should also bring back the sql_refresh_time to 60, hence matching the sql_history configured value. Cheers, Paolo On Fri, Apr 16, 2010 at 02:32:06PM +0400, Sergey Veltistov wrote: Hi I get many errors in my log: DEBUG ( default/mysql ): FAILED query follows: INSERT INTO `acct_bgp` (stamp_updated, stamp_inserted, as_src, as_dst, local_pref, peer_as_dst, packets, bytes) VALUES (FROM_UNIXTIME(1271412061), FROM_UNIXTIME(1271411700), 0, 29791, 120, 28809, 1766, 379285) ERROR ( default/mysql ): Duplicate entry '0-0-29791-0-28809-120-0-2010-04-16 13:55:00' for key 1 Nfacctd process during the occurrence of these errors. 54055 p5 S+ 0:27.57 nfacctd: Core Process [default] (nfacctd) 54056 p5 S+ 0:19.70 nfacctd: MySQL Plugin [default] (nfacctd) 54083 p5 S+ 0:04.32 nfacctd: SQL Plugin -- DB Writer (urgent) [default] (nfacctd) Pmacct receives netflow v5 info from cisco router with traffic through interfaces approximately 4.5Gbit/s (sum in/out). Mysql scripts pmacct-create-db_bgp_v1.mysql. Start cmd for testing: ./nfacctd -f mysql.conf My config: -- daemonize: false promisc: false ! !debug: true ! plugins: mysql plugin_pipe_size: 4096 plugin_buffer_size: 40960 ! nfacctd_ip: x.x.x.x nfacctd_port: 9985 nfacctd_time_new: true aggregate: src_as, dst_as, local_pref, peer_dst_as ! pmacctd_as: bgp bgp_daemon: false bgp_daemon_ip: x.x.x.x bgp_daemon_port: 179 bgp_daemon_max_peers: 1 bgp_agent_map: /usr/local/pmacct/sbin/agent_to_peer.map bgp_peer_src_as_type: map bgp_peer_src_as_map: /usr/local/pmacct/sbin/peers.map ! sql_db: pmacct sql_table_version: 1 sql_table_type: bgp sql_passwd: xx sql_user: pmacct sql_optimize_clauses: true sql_dont_try_update: true sql_cache_entries: 256000 sql_multi_values: false ! sql_history_roundoff: mhd sql_history: 1m sql_refresh_time: 30 sql_table: acct_bgp - Any hints? ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] pmacct 0.12.1 released !
VERSION. 0.12.1 DESCRIPTION. pmacct is a small set of passive network monitoring tools to measure, account, classify, aggregate and export IPv4 and IPv6 traffic; a pluggable and flexible architecture allows to store collected network data into memory tables or SQL (MySQL, SQLite, PostgreSQL) databases and export them through NetFlow or sFlow protocols to remote collectors. pmacct supports fully customizable historical data breakdown, sampling, filtering and tagging and triggers. Libpcap, Netlink/ULOG, sFlow v2/v4/v5 and NetFlow v1/ v5/v7/v8/v9 are supported, both unicast and multicast. Also, a client program makes it easy to export data to tools like RRDtool, GNUPlot, Net-SNMP, MRTG, and Cacti. HOMEPAGE. http://www.pmacct.net/ DOWNLOAD. http://www.pmacct.net/pmacct-0.12.1.tar.gz CHANGELOG. + Input/output interfaces (SNMP indexes) have now been implemented natively; it's therefore not required anymore to pass through the (Pre-)tag infrastructure. As a result two aggregation primitives are being introduced: 'in_iface' and 'out_iface'. + Support for source/destination IP prefix masks is introduced via two new aggregation primitives: src_mask and dst_mask. These are populated as defined by the [nf|sf|pm|u]acctd_net directive: NetFlow/sFlow protocols, BGP, Network files (networks_file) or static (networks_mask) being valid data sources. + A generic tunnel inspection infrastructure has been developed to benefit both pmacctd and uacctd daemons. Handlers are defined via configuration file. Once enabled daemons will account basing upon tunnelled headers rather than the envelope. Currently the only supported tunnel protocol is GTP, the GPRS tunnelling protocol (which can be configured as: tunnel_0: gtp, UDP port). Up to 8 different tunnel stacks and up to 4 tunnel layers per stack are supported. First matching stack, first matching layer wins. + uacctd: support for the MAC layer has been added for the Netlink/ ULOG Linux packet capturing framework. + 'nfprobe_source_ip' feature introduced: it allows to select the IPv4/IPv6 address to be used to export NetFlow datagrams to the collector. + nfprobe, sfprobe: network masks are now exported via NetFlow and sFlow. 'pmacctd_net' and its equivalent directives define how to populate src_mask and dst_mask values. ! cleanup, nfprobe/sfprobe: data source for 'src_as' and 'dst_as' primitives is now expected to be always explicitely defined (in line with how 'src_net' and 'dst_net' primitives work). See the UPGRADE doc for the (limited) backward compatibility impact. ! Updated SQL documentation: sql/README.iface guides on 'in_iface' and 'out_iface' primitives; sql/README.mask guides on 'src_mask' and 'dst_mask' primitives; sql/README.is_symmetric guides on 'is_symmetric' primitive. ! fix, nfacctd.h: source and destination network masks were twisted in the NetFlow v5 export structure definition. Affected releases are: 0.12.0rc4 and 0.12.0. ! fix, nfprobe_plugin.c: l2_to_flowrec() was missing some variable declaration when the package was configured for compilation with --disable-l2. Thanks to Brent Van Dussen for reporting the issue. ! fix, bgp.c: bgp_attr_munge_as4path() return code was not defined for some cases. This was causing some BGP messages to be marked as malformed. ! fix, sfprobe: a dummy MAC layer was created whenever this was not included as part of the captured packet. This behaviour has been changed and header protocol is now set to 11 (IPv4) or 12 (IPv6) accordingly. Thanks to Neil McKee for pointing the issue. ! workaround, building sub-system: PF_RING enabled libpcap was not recognized due to missing of pcap_dispatch(). This is now fixed. NOTES. * data source for 'src_as' and 'dst_as' primitives for nfprobe and sfprobe plugins is now expected to be explicitely defined via the [ pmacctd_as | uacctd_as ] directive. All other plugins were already working like that. In terms of backward compatibility the only case affected is getting ASN values out of a Networks File: up to 0.12.0, it was sufficient to define a networks_file to implicitely use it. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] ifindex and netmask records
Hi Richard, On Fri, Apr 02, 2010 at 03:12:23AM -0500, Richard A Steenbergen wrote: * Record (and aggregate on) the address of the router that exported a flow via netflow/sflow. Basically I just want to know which router exported the flow to me, using either the agent address if available (on sflow, etc), or the source address of the netflow packet. As Nitzan correctly mentioned, pre-tagging should be used for this. The idea is you get a tag instead of the IP address of the NetFlow/sFlow exporter. If doesn't suit, just let me know: I would see it as a good feature request. * Record (and aggregate on) the src/dst ifindexes that are exported via sflow/netflow protocols. Obviously this would be paired with the router id mentioned above to give the ifindex meaning, :) As of 0.12.1 (which will be out in roughly a week) or the code currently in the CVS you have the in_iface and out_iface aggregation primitives. The legacy way (up to 0.12.0) to do it was via pre-tagging as per the point before. Of course pre-tagging (so map ifindexes to tags) can still be used when a stricter control (filter out un-needed stuff) is required as part of the aggregation process. * Record the mask that was used in a src/dst_net aggregator. I figured out how to dynamically aggregate by the netmask value exported via netflow/sflow (via the pmacct changelog, it doesn't seem to be in the documentation anywhere I could find), but it doesn't record the netmask that was used. For example, say I receive an export for a flow to [ ... ] As of 0.12.1 (which will be out in roughly a week) or the code currently in the CVS you have the src_mask and dst_mask aggregation primitives :-) You have also a set of [ nfacctd_net | sfacctd_net | pmacctd_net ] config directives which have as values [ netflow | sflow | mask | file | bgp ]. It means the network prefix and the netmask can be explicitely grasped out of: netflow, sflow, bgp, a networks_file: a file where some networks are listed (can be also a dump of the full BGP table) which makes sense going libpcap or ULOG really or a static netwosk_mask directive: ie. aggregate everything to /24: it makes sense once again if going libpcap or ULOG. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] uacctd documentation
Hi Ross, On Thu, Mar 25, 2010 at 09:57:05AM -0400, Ross Vandegrift wrote: Thanks for the examples! I'm having trouble with the iptables piece of the puzzle though. I suspect this is because I'm mirroring traffic to this server and the L2 destination doesn't match any address present on the server. I've tried add ebtables dnat to rewrite the destination MAC of incoming frames to the local interface address, but this doesn't work. I'm curious: if mirroring traffic to the box, why resorting to uacctd and iptables/etables? You have pmacctd (libpcap-based) readily available for such scenario. With uacctd covering the case where traffic is being cleanly routed through (or to) the Linux box. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] uacctd documentation
Hi Ross, On Fri, Mar 26, 2010 at 02:46:51PM -0400, Ross Vandegrift wrote: I'm curious if I can acheive better performance for generating sflow data. My plan was to use the iptables statistics module to do the sampling, to ensure that only sampled packets were being sent to userspace. Since my sampling target is around 1:2000, this has the potential to cut down on a lot of work sending packets to userspace that will be ignored. Neil at InMon suggested this as a possible tactic for more efficient sflow generation than sfacctd when looking at around 3Gb/s and 500kpps. pmacctd appears to be keeping up, but around 55-80% CPU. Traffic will continue to grow, and I'd like to see if I can't get performance better to make the system last longer. I see. It's just not clear whether you are using pmacctd compiled against vanilla libpcap. If this is the case, i can advice to give a try to PF_RING. The kernel module can be configured for sampling by passing the sample_rate parameter. At a glance, it would just seem fitting better your scenario. In case you want to go this way, please use the code from the CVS as the build sub-system (configure script, etc.) has been patched to recognize presence of PF_RING. This will indeed make it in the upcoming release 0.12.1. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] aggregate-filter min packet count and bgp sql table questions
Hi Brent, Good to see progress. The entries stay forever, there is not an aging-out mechanism. Reason being you are supposed to do it yourself, at regular intervals, to build a time reference for the counters. For example a simplistic scenario is a cronjob entry, set up every 5 minutes, that launches a pmacct -s -e counters-time.txt. Such commandline a) fetches the whole content of the table, b) writes it to a file and c) cleans the full table up. Should be all you need to feed such data into 3rd party applications for reporting, graphing, etc. Cheers, Paolo On Thu, Mar 11, 2010 at 02:00:27PM -0800, Brent Van Dussen wrote: Good news, no longer getting the errors and my system hasn't fallen on its face yet. Also have 2-3x more entries in the memory tables so that's excellent as well. My next question is regarding aging of entries in the memory table, how is this done? At first I checked one of the tables and there were over 100k entries, 3 or 4 seconds later when I checked the table it was down to 50k entries and seems to fluctuate +/- 15k entries. I've read the INTERNALS document pertaining to memory tables but it wasn't quite clear in this regard. Thanks! -Brent ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] aggregate-filter min packet count and bgp sql table questions
Hi Brent, Thanks very much for the feedback. I've just corrected the issue and will commit it to the CVS shortly later today. Cheers, Paolo On Thu, Mar 11, 2010 at 09:26:57AM -0800, Brent Van Dussen wrote: Paolo - Just tried to build the latest CVS version and am getting compile errors when configure option --disable-l2 is used. When I re-configured without it everything builds fine. Here's the error I get: -bash-3.2$ make Making all in src gmake[1]: Entering directory `/home/bvd/pmacct-cvs/pmacct/src' Making all in nfprobe_plugin gmake[2]: Entering directory `/home/bvd/pmacct-cvs/pmacct/src/ nfprobe_plugin' gcc -DPACKAGE=\pmacctd\ -DVERSION=\0.12.1-cvs\ -DPROGNAME=1 - DIM_LITTLE_ENDIAN=1 -DHAVE_PCAP_H=1 -DHAVE_LIBPCAP=1 -DPCAP_7=1 - DPCAP_TYPE_linux=1 -DHAVE_DLOPEN=1 -DSTDC_HEADERS=1 -DHAVE_SYS_WAIT_H=1 -DHAVE_GETOPT_H=1 -DHAVE_SYS_SELECT_H=1 -DHAVE_SYS_TIME_H=1 -DHAVE_U_INT64_T=1 -DHAVE_U_INT32_T=1 -DHAVE_U_INT16_T=1 -DHAVE_U_INT8_T=1 -DENABLE_THREADS=1 -DRETSIGTYPE=void -DHAVE_VSNPRINTF=1 -I. -I.. -O2 -DFLOW_SPLAY-DEXPIRY_RB -c -o nfprobe_plugin.o nfprobe_plugin.c nfprobe_plugin.c: In function ???l2_to_flowrec???: nfprobe_plugin.c:312: error: ???p??? undeclared (first use in this function) nfprobe_plugin.c:312: error: (Each undeclared identifier is reported only once nfprobe_plugin.c:312: error: for each function it appears in.) gmake[2]: *** [nfprobe_plugin.o] Error 1 gmake[2]: Leaving directory `/home/bvd/pmacct-cvs/pmacct/src/ nfprobe_plugin' gmake[1]: *** [all-recursive] Error 1 gmake[1]: Leaving directory `/home/bvd/pmacct-cvs/pmacct/src' make: *** [all-recursive] Error 1 Should be able to let you know shortly how the prefix length feature works. Thanks, -Brent On Mar 9, 2010, at 8:28 AM, Brent Van Dussen wrote: Thanks for getting this set up Paolo! We'll get the latest CVS version loaded and tested this week to provide feedback. Cheers, -Brent On Mar 7, 2010, at 1:34 AM, Paolo Lucente wrote: Hi Brent, All, On Sat, Feb 20, 2010 at 01:05:20AM +, Paolo Lucente wrote: Would it also be possible to have the dst_net appended with mask length and a slightly larger DB field to accomodate it? 255.255.255.255/25 would be a CHAR(18) instead of CHAR(15) but it would be really nice to have this information from BGP or the sflow prefix field added in so a network map isn't needed. Netmasks. They are not there but i've recently received two more requests in a row for such a feature. All coming from people making use of the BGP feature. Hence, this is on my todo list and will appear very soon - would say roughly 3-4 weeks. Would such approx timeline match your expectations? To say at this propo that network masks have now been implemented as aggregation primitives. It's all published in the CVS. Network masks lie in a different field from the IP prefix - so that this implementation can get handy for other, ie. statistical, purposes. Indeed, SQL language manipulation can be used to merge masks and IP prefixes together on output. Also, good property of this way of proceeding is that by keeping masks in a separate field, they can be defined as INT(1) which is space-savy compared to a string representation within the IP prefix field. I already see this argument will not particularly impress the PosgreSQL friends but it's done for the sake of selecting a generic approach (not all RDBMS packages have the luxury of 'inet' types ...). Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] aggregate-filter min packet count and bgp sql table questions
Hi Brent, On Thu, Mar 11, 2010 at 10:01:26AM -0800, Brent Van Dussen wrote: I wanted to ask about these messages we're getting in the logs now that we're using mem tables: WARN ( prefixes/memory ): Unable to allocate more memory pools, clear stats manually! WARN ( as_path/memory ): Unable to allocate more memory pools, clear stats manually! What does it mean? It seems like data is still getting into the mem tables. It means the aggregates that are already in the table will still get updated as new traffic comes in. But new aggregates will not make it into the memory structure. Under default settings, it's a protection mechanism to limit memory allocation. It can be fine tuned by reading the INTERNALS document or briefly worked around by configuring: imt_mem_pools_number: 0 If configuring such setting you still see the message above, it means the process is unable to allocate more memory from the system (either software or hardware limit). Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Log output from pmacctd Ive never seen before
Hi Jeff, How often you get this message? Every time you see it, you miss a packet. The way to read that line is: libpcap passed only the first 37 bytes of the datagram to pmacct; L4 appears to start at the 38th octet; and you might have specified src_port or dst_port in your aggregation method. By default pmacct snaplen for IPv4 packets is 68 bytes, so imposing something explicit wouldn't help. It appears the reason for such truncation is external - hence the question at the beginning of this email. Cheers, Paolo On Sun, Feb 28, 2010 at 11:56:50PM -0800, Jeff Welling wrote: Hello again all! Just recently I started seeing this line in my logs every couple of minutes; Feb 28 23:34:57 gir pmacctd[12915]: INFO ( default/core ): short IPv4 packet read (37/38/frags). Snaplen issue ? Gir being the name of my gateway machine. Recent events include installation of an additional internet connection (dual-homed), and configuring pmacctd to monitor the new interface with basically the same setup as my other interface, and setting the interface to not auto-negotiate its speed because that was causing problems. Does this mean pmacct is missing information? Should the snaplen be increased, I dont think Im doing classification and CONFIG-KEYS says the defaults should be ok unless your using classification? Cheers, Jeff. ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Pmacct data inconsistencies between tables.
Hi Daniel, I see the 1 minute table contains duplicates - it would be better to say that _everything_ extracted is repeated twice. It could be handy to add tags to the aggregation method and assign a different 'post_tag' to each plugin so to identify who is generating them. I also wonder: how does the primary key of the 1 min table look like? Is it any different from the 1 hour table? With the sql_don_try_update turned on and the default indexing, duplicates are not possible. Also at a closer look to the configuration you posted i see no aggregate_filter are specified (see EXAMPLES): it means each plugin collects and tries to write to the same table both inbound and outbound traffic. So either you can remove one set of plugins or craft a proper aggregate_filter so that each does only its bit of the job. With regards to the missing tuples, from the few checks i've done, it is always the case that something is in the 1 hour table but can be missing in the 1 minute one. This can very well be the result of a shared 'sql_preprocess: minb = 1000' directive: a flow can accumulate more than 1000 bytes in 1 hour but not in 1 minute - and hence it's accounted in one table and stripped off in the other. Given the sql_preprocess you should never expect counters to match for the same reason as above. To have a comparison more apples to apples, you should consider removing it and when confident everything is allright put it back again. Finally, unrelated to the issue: please for the benefit of public archives, don't send attachments to the list. Cheers, Paolo On Fri, Feb 19, 2010 at 12:11:24PM +, Daniel Levy wrote: Hi Paolo, Here is a report with differences between the two tables. There are a lot of differences between the download figures as well as some instances where IP addresses are only found in one table (see report.txt). I also have some raw data for the time periods between 10:00 and 11:00 on 11/02/2010, which are being sent separately via yousendit. Regards -- Daniel Levy Aptivate | http://www.aptivate.org/ | +44 (0)1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES Aptivate is a not-for-profit company registered in England and Wales with company number 04980791. Paolo Lucente wrote: Hi Daniel, Getting through the data and compare traffic figures is, IHMO, the more practical approach - compared to trying to reproduce the issue in a controlled environment. Once you discover a descrepancy, it would be great to receive the contributing data of each report to see where the issue comes from. It's also true that version 0.9.1 is almost 5 years old; i would highly encourage to refresh it. I should be correct saying Ubuntu features also version 0.11.4 and 0.11.6 if you really don't like the idea of compiling 0.12 yourself (which would be my preferred approach). Let me know. Cheers, Paolo On Mon, Feb 15, 2010 at 10:41:21AM +, Daniel Levy wrote: Hi Paolo, Thanks for getting back to me. The version of pmacct being used is 0.9.1-1ubuntu1. I'm not sure how the problem was discovered, but I have asked to person who found the problem to tell me and I will forward you the response. As for the reports, I'm not entirely sure what you need. I am considering going through the database data for each hour and comparing the total figures for uploaded and downloaded packets, per IP address between the two tables. Would this give you the information you're looking for? -- Daniel Levy Aptivate | http://www.aptivate.org/ | +44 (0)1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES Aptivate is a not-for-profit company registered in England and Wales with company number 04980791. Paolo Lucente wrote: Hi Daniel, Unfortunately the configuration doesn't make evident where the issue can be. The 'sql_dont_try_update' very well protects against duplicate tuples - so i'm rather inclined to exclude that reason. Which version are you using? How you did discover the issue - ie. did you upgrade recently from a previous version or is a fresh installation? Finally, is it possible to get - privately - two reports, one from each table, for the same time period? Say, one or better two hours? Let me know. Cheers, Paolo On Fri, Feb 12, 2010 at 03:19:38PM +, Daniel Levy wrote: Hi, I'm using pmacct to store data in two tables, one containing data recorded on a per minute basis, the other containing data recorded on an hourly basis. When I get data for the first table over a period of three hours, the download traffic (calculated by adding up the bytes field for traffic where the ip_dst value is from a machine on the local network) for one IP address on the network is 3,719,772,656 bytes. The download traffic from the second table for the same IP address over a period
Re: [pmacct-discussion] aggregate-filter min packet count and bgp sql table questions
Hi Brent, On Fri, Feb 19, 2010 at 10:51:21AM -0800, Brent Van Dussen wrote: I was curious if there was a way to have sfacctd only insert into the database if a certain number of packets and/or bytes threshold is reached. It seems you are looking for the sql_preprocess directive - and minb and/or minp keys. It allows to put some conditions for row insertion/ update at every refresh interval. Would it also be possible to have the dst_net appended with mask length and a slightly larger DB field to accomodate it? 255.255.255.255/25 would be a CHAR(18) instead of CHAR(15) but it would be really nice to have this information from BGP or the sflow prefix field added in so a network map isn't needed. Netmasks. They are not there but i've recently received two more requests in a row for such a feature. All coming from people making use of the BGP feature. Hence, this is on my todo list and will appear very soon - would say roughly 3-4 weeks. Would such approx timeline match your expectations? Question to you (and others who might be interested indeed): given the use you would make out of them; how would you envision netmasks implementation in the DB structure? What would be handy: a separate field in the table or the mask appended to the IP prefix string? I do have myself an idea but don't want to biase answer(s) at this stage. One final thing was that I saw erratic behavior with the as-path feature when creating my own custom sql table from scratch, pasted below...anything look like it shouldn't work? I would comment out the sql_table_version and sql_table_type and uncomment sql_table to define the new table name to use and everything inserted ok just most of my as-path's were then missing create table custom_llnw ( agent_id INT(4) UNSIGNED NOT NULL, as_path CHAR(50) NOT NULL, packets INT UNSIGNED NOT NULL, bytes BIGINT UNSIGNED NOT NULL, stamp_inserted DATETIME NOT NULL, stamp_updated DATETIME, PRIMARY KEY (agent_id, stamp_inserted) ); Configuration and schema look OK (except for a primary key missing the as_path, but perhaps that is intentional). In the table above, as_path is the only primitive being grasped from BGP - it looks there might be an underlying issue correlating sFlow data to BGP. Did you already clear out other BGP primitives, ie. local_pref, are grasped OK? For some debug you can enable the 'bgp_daemon_msglog' in order to verify the prefixes are received OK. To see which router is successfully connecting, you can locally enable debug. I can advice to give a name to the 'mysql' plugin and then enable debug on the 'default' (core) process, ie.: debug[default]: true plugins: mysql[blabla] If daemonizing, it would be convenient to add a 'logfile' directive to get all such output nailed down somewhere persistent. Thanks and let me know if there's any other data I can provide. Great work on getting BGP implemented into the daemon!!! Thanks! Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] unable to compile latest pmacct with pfring enabled libpcap ...
Hi Sebastien, It's not clear to me if this was working for you before (some earlier pmacct release) and it doesn't instead with the latest. I've just tried myself to compile pmacct 0.12.0 against a libpcap 1.0.0 (vanilla) and it works fine. I'm also fairly sure this worked up to 0.12.0rc4 for others - Yuriy should be among them. Cheers, Paolo On Thu, Feb 18, 2010 at 01:01:14PM +0100, S?bastien Cramatte wrote: It doesn't works :( ... same error ./configure --with-pcap-includes=/usr/src/pf_ring/PF_RING/userland/libpcap-1.0.0-ring/pcap --with-pcap-libs=/usr/local/lib #ls 2010/2/18 Yavetskiy Yuriy yavets...@kpi.ua Hello. Try to use --with-pcap-includes=/path/to/userland/libpcap-1.0.0-ring/pcap --with-pcap-libs=/path/to/lib S?bastien Cramatte wrote: Hello, I've donwloaded, compiled and installed the latest pf_ring patch. Ntop works as expected but unfortunately I'm unable to compile pmacct-0.12.0 : checking whether to disable L2 features... no checking whether to enable IPv6 code... no checking default locations for pcap.h... found in /usr/local/include checking default locations for libpcap... found in /usr/local/lib checking for pcap_dispatch in -lpcap... no configure: error: ERROR: missing pcap library. Refer to: http://www.tcpdump.org/ ... Any ideas of what occurs ? I suspect that I should add somewhre LDFLAGS with -lpfring ... Thank you for your help ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists -- WBR Yavetskiy Yuriy ULTI-RIPE ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] unable to compile latest pmacct with pfring enabled libpcap ...
Hi Sebastien, Unfortunately it's not something quick to reproduce as i'm myself mostly based on Solaris. I will gladly have a look on your box instead. Anyway: what the configure script was complaining is the checking for pcap_dispatch in -lpcap check. Meaning the libpcap.a doesn't exist or doesn't appear to be valid. The issue doesn't look connected to PF_RING specifically. Consider also pmacct doesn't have any native support for PF_RING (which ntop might have instead), so I don't think you should ever link with -lpfring - but I can indeed be mistaken. Cheers, Paolo On Thu, Feb 18, 2010 at 02:12:18PM +0100, S?bastien Cramatte wrote: Hi, It's true ... was working with an old release with debian eth on x86 platform. Now we have upgraded to debian squeeze on x86_64 platform. Note that we are using pf_ring and it works perfectly with ntop ... Could you try to build it using libpcap from pf_ring svn ? svn co https://svn.ntop.org/svn/ntop/trunk/PF_RING/ 2010/2/18 Paolo Lucente pa...@pmacct.net Hi Sebastien, It's not clear to me if this was working for you before (some earlier pmacct release) and it doesn't instead with the latest. I've just tried myself to compile pmacct 0.12.0 against a libpcap 1.0.0 (vanilla) and it works fine. I'm also fairly sure this worked up to 0.12.0rc4 for others - Yuriy should be among them. Cheers, Paolo On Thu, Feb 18, 2010 at 01:01:14PM +0100, S?bastien Cramatte wrote: It doesn't works :( ... same error ./configure --with-pcap-includes=/usr/src/pf_ring/PF_RING/userland/libpcap-1.0.0-ring/pcap --with-pcap-libs=/usr/local/lib #ls 2010/2/18 Yavetskiy Yuriy yavets...@kpi.ua Hello. Try to use --with-pcap-includes=/path/to/userland/libpcap-1.0.0-ring/pcap --with-pcap-libs=/path/to/lib S?bastien Cramatte wrote: Hello, I've donwloaded, compiled and installed the latest pf_ring patch. Ntop works as expected but unfortunately I'm unable to compile pmacct-0.12.0 : checking whether to disable L2 features... no checking whether to enable IPv6 code... no checking default locations for pcap.h... found in /usr/local/include checking default locations for libpcap... found in /usr/local/lib checking for pcap_dispatch in -lpcap... no configure: error: ERROR: missing pcap library. Refer to: http://www.tcpdump.org/ ... Any ideas of what occurs ? I suspect that I should add somewhre LDFLAGS with -lpfring ... Thank you for your help ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists -- WBR Yavetskiy Yuriy ULTI-RIPE ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Pmacct data inconsistencies between tables.
Hi Daniel, Getting through the data and compare traffic figures is, IHMO, the more practical approach - compared to trying to reproduce the issue in a controlled environment. Once you discover a descrepancy, it would be great to receive the contributing data of each report to see where the issue comes from. It's also true that version 0.9.1 is almost 5 years old; i would highly encourage to refresh it. I should be correct saying Ubuntu features also version 0.11.4 and 0.11.6 if you really don't like the idea of compiling 0.12 yourself (which would be my preferred approach). Let me know. Cheers, Paolo On Mon, Feb 15, 2010 at 10:41:21AM +, Daniel Levy wrote: Hi Paolo, Thanks for getting back to me. The version of pmacct being used is 0.9.1-1ubuntu1. I'm not sure how the problem was discovered, but I have asked to person who found the problem to tell me and I will forward you the response. As for the reports, I'm not entirely sure what you need. I am considering going through the database data for each hour and comparing the total figures for uploaded and downloaded packets, per IP address between the two tables. Would this give you the information you're looking for? -- Daniel Levy Aptivate | http://www.aptivate.org/ | +44 (0)1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES Aptivate is a not-for-profit company registered in England and Wales with company number 04980791. Paolo Lucente wrote: Hi Daniel, Unfortunately the configuration doesn't make evident where the issue can be. The 'sql_dont_try_update' very well protects against duplicate tuples - so i'm rather inclined to exclude that reason. Which version are you using? How you did discover the issue - ie. did you upgrade recently from a previous version or is a fresh installation? Finally, is it possible to get - privately - two reports, one from each table, for the same time period? Say, one or better two hours? Let me know. Cheers, Paolo On Fri, Feb 12, 2010 at 03:19:38PM +, Daniel Levy wrote: Hi, I'm using pmacct to store data in two tables, one containing data recorded on a per minute basis, the other containing data recorded on an hourly basis. When I get data for the first table over a period of three hours, the download traffic (calculated by adding up the bytes field for traffic where the ip_dst value is from a machine on the local network) for one IP address on the network is 3,719,772,656 bytes. The download traffic from the second table for the same IP address over a period of one week, including the three hour period mentioned above, is significantly smaller (2,114,286,512 bytes) where I would expect it to be much larger and I can't figure out why. A slightly modified version of the contents of my pmacctd.conf file is given below. Can anyone help? daemonize: true pidfile: /var/run/pmacctd.pid syslog: daemon plugins: mysql[inbound1], mysql[outbound1], mysql[inbound2], mysql[outbound2] aggregate[inbound1]: src_host, src_port, dst_host, dst_port, proto aggregate[outbound1]: src_host, src_port, dst_host, dst_port, proto aggregate[inbound2]: src_host, src_port, dst_host, dst_port, proto aggregate[outbound2]: src_host, src_port, dst_host, dst_port, proto pcap_filter: not (src and dst net 0.0.0.0/24) sql_db: pmacct sql_table[inbound1]: short_data_table sql_table[outbound1]: short_data_table sql_table[inbound2]: long_data_table sql_table[outbound2]: long_data_table sql_history[inbound1]: 1m sql_history[outbound1]: 1m sql_history[inbound2]: 1h sql_history[outbound2]: 1h sql_history_roundoff[inbound1]: m sql_history_roundoff[outbound1]: m sql_history_roundoff[inbound2]: h sql_history_roundoff[outbound2]: h sql_table_version: 6 sql_host: localhost sql_user: auser sql_passwd: apass sql_refresh_time[inbound1]: 60 sql_refresh_time[outbound1]: 60 sql_refresh_time[inbound2]: 3600 sql_refresh_time[outbound2]: 3600 sql_dont_try_update: true sql_optimize_clauses: true sql_preprocess: minb = 1000 Regards -- Daniel Levy ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] pmacct 0.12.0 released !
VERSION. 0.12.0 DESCRIPTION. pmacct is a small set of passive network monitoring tools to measure, account, classify, aggregate and export IPv4 and IPv6 traffic; a pluggable and flexible architecture allows to store collected network data into memory tables or SQL (MySQL, SQLite, PostgreSQL) databases and export them through NetFlow or sFlow protocols to remote collectors. pmacct supports fully customizable historical data breakdown, sampling, filtering and tagging and triggers. Libpcap, Netlink/ULOG, sFlow v2/v4/v5 and NetFlow v1/ v5/v7/v8/v9 are supported, both unicast and multicast. Also, a client program makes it easy to export data to tools like RRDtool, GNUPlot, Net-SNMP, MRTG, and Cacti. HOMEPAGE. http://www.pmacct.net/ DOWNLOAD. http://www.pmacct.net/pmacct-0.12.0.tar.gz CHANGELOG. + 'is_symmetric' aggregation primitive has been implemented: aimed at easing detection of asymmetric traffic. It's based on rule definitions supplied in a 'bgp_is_symmetric_map' map, reloadable at runtime. + A new 'bgp_daemon_allow_file' configuration directive allows to specify IP addresses that can establish a BGP session with the collector's BGP thread. Many thanks to Erik van der Burg for contributing the idea. + 'nfacctd_ext_sampling_rate' and 'sfacctd_ext_sampling_rate' are introduced: they flag the daemon that captured traffic is being sampled. Useful to tackle corner cases, ie. the sampling rate reported by the NetFlow/sFlow agent is missing or incorrect. + The 'bgp_follow_nexthop' feature has been extended so that extra IPv4/IPv6 prefixes can be supplied. Up to 32 IP prefixes are now supported and a warning message is generated whenever a supplied string fails parsing. + Pre-Tagging: implemented 'src_local_pref' and 'src_comms' keys. These allow tagging based on source IP prefix local_pref (sourced from either a map or BGP, ie. 'bgp_src_local_pref_type: map', 'bgp_src_local_pref_type: bgp') and standard BGP communities. + Pre-Tagging: 'src_peer_as' key was extended in order to match on BGP-sourced data (bgp_peer_src_as_type: bgp). + Pre-Tagging: introduced 'comms' key to tag basing on up to 16 standard BGP communities attached to the destination IP prefix. The lookup is done against the BGP RIB of the exporting router. Comparisons can be done in either match-any or match-all fashion; xidDocumentation and examples updated. ! fix, util.c: load_allow_file(), empty allow file was granting a connection to everybody being confused with a 'no map' condition. Now this case is properly recognized and correctly translates in a reject all clause. ! fix, sql_common.c: log of NetFlow micro-flows to a SQL database (nfacctd_sql_log directive) was not correctly getting committed to the backend, when sql_history was disabled. ! fix, mysql|pgsql|sqlite_plugin.c: 'flows' aggregation primitive was not suitable to mix-and-match with BGP related primitives (ie. peer_dst_as, etc.) due to an incorrect check. Many thanks to Zenon Mousmoulas for the bug report. ! fix, pretag_handlers.c: tagging against NetFlow v9 4-bytes in/out interfaces was not working properly. Thanks to Zenon Mousmoulas for reporting the issue. NOTES. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Pmacct data inconsistencies between tables.
Hi Daniel, Unfortunately the configuration doesn't make evident where the issue can be. The 'sql_dont_try_update' very well protects against duplicate tuples - so i'm rather inclined to exclude that reason. Which version are you using? How you did discover the issue - ie. did you upgrade recently from a previous version or is a fresh installation? Finally, is it possible to get - privately - two reports, one from each table, for the same time period? Say, one or better two hours? Let me know. Cheers, Paolo On Fri, Feb 12, 2010 at 03:19:38PM +, Daniel Levy wrote: Hi, I'm using pmacct to store data in two tables, one containing data recorded on a per minute basis, the other containing data recorded on an hourly basis. When I get data for the first table over a period of three hours, the download traffic (calculated by adding up the bytes field for traffic where the ip_dst value is from a machine on the local network) for one IP address on the network is 3,719,772,656 bytes. The download traffic from the second table for the same IP address over a period of one week, including the three hour period mentioned above, is significantly smaller (2,114,286,512 bytes) where I would expect it to be much larger and I can't figure out why. A slightly modified version of the contents of my pmacctd.conf file is given below. Can anyone help? daemonize: true pidfile: /var/run/pmacctd.pid syslog: daemon plugins: mysql[inbound1], mysql[outbound1], mysql[inbound2], mysql[outbound2] aggregate[inbound1]: src_host, src_port, dst_host, dst_port, proto aggregate[outbound1]: src_host, src_port, dst_host, dst_port, proto aggregate[inbound2]: src_host, src_port, dst_host, dst_port, proto aggregate[outbound2]: src_host, src_port, dst_host, dst_port, proto pcap_filter: not (src and dst net 0.0.0.0/24) sql_db: pmacct sql_table[inbound1]: short_data_table sql_table[outbound1]: short_data_table sql_table[inbound2]: long_data_table sql_table[outbound2]: long_data_table sql_history[inbound1]: 1m sql_history[outbound1]: 1m sql_history[inbound2]: 1h sql_history[outbound2]: 1h sql_history_roundoff[inbound1]: m sql_history_roundoff[outbound1]: m sql_history_roundoff[inbound2]: h sql_history_roundoff[outbound2]: h sql_table_version: 6 sql_host: localhost sql_user: auser sql_passwd: apass sql_refresh_time[inbound1]: 60 sql_refresh_time[outbound1]: 60 sql_refresh_time[inbound2]: 3600 sql_refresh_time[outbound2]: 3600 sql_dont_try_update: true sql_optimize_clauses: true sql_preprocess: minb = 1000 Regards -- Daniel Levy ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] pre_tag_map issue + questions
Hi Zenon, On Mon, Feb 08, 2010 at 02:43:49PM +0200, Zenon Mousmoulas wrote: records. If it's in there, then i'd like to give it a look myself: i would ask you to produce a trace and send it to me privately so that i can have a look. We can then summarize findings here. OK. I will send you the capture privately. It turned to be another bug related to 4-bytes ifIndex values exported via NetFlow v9. Fix is already committed to the CVS. Thanks for your cooperation on this. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] pmacct count only 5% of SYN packets.
Hi Yuriy, You have also other means to get a count of the TCP/SYN packets out of pmacct. I would suggest one for troubleshooing purposes with the goal to check where the issue lies: * keep the 'tcpflags' primitive out of the 'aggregate' directive * add a 'pcap_filter' directive to the config; it supports filters in tcpdump syntax and you can feed it with the same filter you use in tcpdump to count TCP/SYN packets. I would like to know if counting TCP/SYN packets this way makes pmacct matching the numbers you get out of tcpdump. Btw, if you like this strategy, instead of resorting to tcpflags, it can be refined so to make it co-existing with other things you might want to do with the tool (ie. by using tagging or replacing the 'pcap_filter' with an 'aggregate_filter'). Cheers, Paolo On Tue, Feb 09, 2010 at 12:39:52PM +0200, Yavetskiy Yuriy wrote: Hello. I'm running Debian with vanila 2.6.32 kernel. I've compiled pmacct 0.11.6 with pf_ring (transparent mode 1). Packets pass through brigde of 2 interfaces, one interface in promisc mode (both interfaces are intel 82576, NAPI, LRO, RSS). Load on this brigde is 800mbps and 100 kpps (90% idle on each of 8 cores). I'm using pmacct to count traffic through brigde. I've noticed, that snmp data and pmacct's data are the same (99% similar in MB). But if I use tcpflags in aggregation (src_host, dst_host, dst_port, proto, tcpflags), I see (compare pmacct's data on bridge with tcpdump on packet's destination host) that only 5% of packets with flag 2 (SYN) are counted. What could be the problem? -- WBR Yavetskiy Yuriy ULTI-RIPE ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Missing information in mysql table
Hi Jonas, On Tue, Feb 09, 2010 at 04:55:52PM +0100, Jonas Nylund wrote: [ ... ] mysql select * from acct_v5_06 limit 10; +--+--+-+-+--+-+-+--+--+--+-+-+---+---+-+-+ | agent_id | class_id | mac_src | mac_dst | vlan | ip_src | ip_dst | src_port | dst_port | ip_proto | tos | packets | bytes | flows | stamp_inserted | stamp_updated | +--+--+-+-+--+-+-+--+--+--+-+-+---+---+-+-+ |0 | unknown | 0:0:0:0:0:0 | 0:0:0:0:0:0 | 200 | 0.0.0.0 | 0.0.0.0 |0 |0 | ip | 104 | 67 | 36641 | 0 | [ ... ] I probably have missed something in my config. But i cant find what. I also need help undestanding the TOS values? What is tos 67? in VLAN 200 i only should see DSCP AF31, and DSCP 47. I see 67 appearing in the 'packets' field. TOS appears to be 104, which should match your expectations of seeing DSCP AF31 populating such field. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] The accounting of flows requires SQL table v4?
Hi Zenon, Good to see you around again. Please add to your config: sql_table_version[sqltest]: 4 Reason being most of the primitives are still connected to the SQL table versioning concept (in essence: pmacct expects you to make explicit which SQL schema you are running). This is in the process of being dismantled, which is already the case with the plethora of BGP-related primitives (i know you come from there and hence it sounds new ...). Cheers, Paolo On Mon, Feb 08, 2010 at 10:34:50AM +0200, Zenon Mousmoulas wrote: Hi, plugins: mysql[sqltest] sql_optimize_clauses[sqltest]: true sql_table[sqltest]: acct ... aggregate: src_as, [...] flows, tag I also tried aggregate[sqltest]: src_as, [...] flows, tag It is also in my schema: create table acct ( id INT(4) UNSIGNED NOT NULL AUTO_INCREMENT, agent_id INT(4) UNSIGNED NOT NULL, as_src INT(4) UNSIGNED NOT NULL, [...] flows INT UNSIGNED NOT NULL, packets INT UNSIGNED NOT NULL, bytes BIGINT UNSIGNED NOT NULL, stamp_inserted DATETIME NOT NULL, stamp_updated DATETIME, PRIMARY KEY (id), INDEX idxall (agent_id, as_src, as_dst, stamp_inserted) ) TYPE=MyISAM AUTO_INCREMENT=1; No matter what, I always get this: ERROR ( sqltest/mysql ): The accounting of flows requires SQL table v4. Exiting. Using latest cvs code (2010-01-29). Any ideas? Thanks, Z. ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] pre_tag_map issue + questions
Hi Zenon, On Mon, Feb 08, 2010 at 10:37:54AM +0200, Zenon Mousmoulas wrote: I have a netflow v9 feed to nfacctd from a juniper router (JUNOS 9.6R2.11), using a service pic. According to a packet capture, records include ingress and egress interface and they seem to be properly defined in the corresponding template. I've tried to use the snmp ifindex numbers in pre_tag_map keys, but they never match. Matching in pre_tag_map with other keys seems to work fine. Any ideas how to debug? You can start by checking (ie. with Wireshark) whether the input/output interface fields are part of the NetFlow v9 template other than the records. If it's in there, then i'd like to give it a look myself: i would ask you to produce a trace and send it to me privately so that i can have a look. We can then summarize findings here. I also noticed that proto and ToS are not available as pre_tag_map keys. Any particular reason for that? DSCP matching would be handy in my case. For this task you can use a 'filter' keyword within the pre_tag_map, which accepts a filter in libpcap syntax. Give it a try and let me know. On a somewhat different note: the particular juniper can also export ipv6 flows, using a different template. I've noticed it includes an IP_PROTOCOL_VERSION (60) field in this template. If we send the ipv6 feed to the same instance of nfacctd which receives the ipv4 feed, how can we tell apart ipv6 from ipv4 traffic if we're doing AS aggregation? It would be handy to have an ip_proto aggregation primitive, or at least to be able to match by 'IPVersion' in a pre_tag_map. Sure. Once again you have to resort to a filter in libpcap format, this time the 'aggregate_filter'. You can configure it as follows: aggregate_filter[ip_traffic]: ip aggregate_filter[ip6_traffic]: ip6 Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] The accounting of flows requires SQL table v4?
HI Zenon, It turned to be a bug. It's now fixed and change is already committed to the CVS. Please give it a try and let me know. Cheers, Paolo On Mon, Feb 08, 2010 at 02:22:41PM +0200, Zenon Mousmoulas wrote: Hi Paolo, it's nice to be back :) I had already tried adding this directive (and tried it one more time just before writing this). Unfortunately it made no difference... Z. On 08 ?? 2010, at 11:51 , Paolo Lucente wrote: Hi Zenon, Good to see you around again. Please add to your config: sql_table_version[sqltest]: 4 Reason being most of the primitives are still connected to the SQL table versioning concept (in essence: pmacct expects you to make explicit which SQL schema you are running). This is in the process of being dismantled, which is already the case with the plethora of BGP-related primitives (i know you come from there and hence it sounds new ...). Cheers, Paolo On Mon, Feb 08, 2010 at 10:34:50AM +0200, Zenon Mousmoulas wrote: Hi, plugins: mysql[sqltest] sql_optimize_clauses[sqltest]: true sql_table[sqltest]: acct ... aggregate: src_as, [...] flows, tag I also tried aggregate[sqltest]: src_as, [...] flows, tag It is also in my schema: create table acct ( id INT(4) UNSIGNED NOT NULL AUTO_INCREMENT, agent_id INT(4) UNSIGNED NOT NULL, as_src INT(4) UNSIGNED NOT NULL, [...] flows INT UNSIGNED NOT NULL, packets INT UNSIGNED NOT NULL, bytes BIGINT UNSIGNED NOT NULL, stamp_inserted DATETIME NOT NULL, stamp_updated DATETIME, PRIMARY KEY (id), INDEX idxall (agent_id, as_src, as_dst, stamp_inserted) ) TYPE=MyISAM AUTO_INCREMENT=1; No matter what, I always get this: ERROR ( sqltest/mysql ): The accounting of flows requires SQL table v4. Exiting. Using latest cvs code (2010-01-29). Any ideas? Thanks, Z. ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Not working pmacct -N [matching data['; ' ... ]]
Hi Slava, Can you confirm which version you are running into this issue? Can you also post your config - just in case? Labbing this scenario up, i see it working fine for me. One thing i can suggest is: append a -l to your query to ensure a locked access to the memory table; if it's a concurrency issue, you should see it working after that. Let me know. Cheers, Paolo On Wed, Feb 03, 2010 at 03:09:00AM +0200, sl...@tangramltd.com wrote: Hi. [r...@stat-ca ~]# pmacct -c src_net -N 91.72.146.32 36575 [r...@stat-ca ~]# pmacct -c src_net -N 91.72.146.30 208 [r...@stat-ca ~]# pmacct -c src_net -N 91.72.146.32;91.72.146.30 36575 0 Why last command show 0 for 91.72.146.30 ? And [r...@stat-ca ~]# pmacct -c src_net -N 91.72.146.32;91.72.146.30 -S 36575 instead 36575+208 Seems it's bug. WBR, Slava ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Enterasys nfacctd expecting flow error
Hi Marc, I would ask you if you can send me privately a packet capture (in tcpdump format, full payload) so that i can have a look into it and possibly replay in lab. This should very well give an insight on the sequence jumps; and might also give an hint why not all the traffic is accounted for ie. if there is a cause-effect relationship between the twos. Let me know if this is acceptable to you. Cheers, Paolo On Thu, Jan 14, 2010 at 02:54:55PM -0800, marc slice wrote: They appear frequently. Every 10-15 secs. We have between 25-80Mbps running across the interfaces recording netflow data on the enterasys through out the day. Not all the traffic is getting recorded when compared to port statistics. No real pattern that we have found. We have 1Gbps connections from the enterasys to the collector and the collector is a HP 2 CPU opteron box with 8GB of memory. CPU is seeing very little use at all times. - Original Message From: Paolo Lucente pa...@pmacct.net To: pmacct-discussion@pmacct.net Sent: Wed, January 13, 2010 3:33:36 PM Subject: Re: [pmacct-discussion] Enterasys nfacctd expecting flow error Hi Marc, Such messages tell it has been detected some issues with NetFlow datagram sequence numbers. This can be caused by packet loss between an agent and the collector, mistakes in the sequencing encoding among the others. Besides the warning messages, which can be turned off, NetFlow datagrams reaching pmacct are processed as usual. Do you see such messages appearing regularly or occasionally? Can you spot a pattern (ie. only a subset of the devices are affected, jumps repeat the same way, etc.)? Cheers, Paolo On Wed, Jan 13, 2010 at 11:52:07AM -0800, marc slice wrote: I have setup netflow from an Enterasys N series switch and receive the following when running nfacctd. WARN: expecting flow '2727940030' but received '2727940026' collector=0.0.0.0:2055 agent=172.16.32.2:513 WARN: expecting flow '11226450' but received '11226438' collector=0.0.0.0:2055 agent=172.16.32.2:769 WARN: expecting flow '2727940052' but received '2727940056' collector=0.0.0.0:2055 agent=172.16.32.2:513 WARN: expecting flow '11226456' but received '11226468' collector=0.0.0.0:2055 agent=172.16.32.2:769 WARN: expecting flow '2727952866' but received '2727952852' collector=0.0.0.0:2055 agent=172.16.32.2:513 WARN: expecting flow '11226618' but received '11226617' collector=0.0.0.0:2055 agent=172.16.32.2:769 WARN: expecting flow '2727952868' but received '2727952882' collector=0.0.0.0:2055 agent=172.16.32.2:513 Couldn't find much info on this problem and was wondering if someone could help? [ ... ] ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] pmacct can't create mysql table, but I can with the same schema
Hi Jeff, If i'm getting this correctly, you see everything working except for the table creation error you get back. If this is the case, i wouldn't say you are doing something wrong; pmacct tries to create the table every time the SQL cache scanner kicks in (sql_refresh_time interval); table name is a string, just it gets compiled in at runtime with dynamic tables. So, the failure to create the table derives from the fact the table name changes monthly in your case whereas data is inserted every 5 minutes. In summary: this is an harmless and not impacting issue. Why in the end this log message is labelled by pmacct as ERROR? pmacct just reports what the SQL library says - it has no clue of the severity of the content itself. This should really be an informational one. Cheers, Paolo On Tue, Dec 22, 2009 at 03:44:16AM -0800, Jeff Welling wrote: Hi again, So it seems I spoke too soon . It looks like its accumulating information in the database in the right tables, but I don't know that the information is correct or why its trying to create the table even though it already exists..? I'm seeing this in my syslogs, one per table but here's the output from just one; Dec 22 03:35:15 Gir pmacctd[10365]: ERROR ( incoming/mysql ): PRIMARY 'mysql' backend trouble. Dec 22 03:35:15 Gir pmacctd[10365]: ERROR ( incoming/mysql ): The SQL server says: Table 'pmacct_v7_incoming_200912' already exists It shows up, I'm guessing, every time it tries to update the sql database. Any idea what I did wrong? Anything at all will be appreciated :) Thanks, Jeff. ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] pmacct 0.12.0rc4 released !
VERSION.
0.12.0rc4
DESCRIPTION.
pmacct is a small set of passive network monitoring tools to
measure, account, classify, aggregate and export IPv4 and IPv6
traffic; a pluggable and flexible architecture allows to store
collected network data into memory tables or SQL (MySQL, SQLite,
PostgreSQL) databases and export them through NetFlow or sFlow
protocols to remote collectors. pmacct supports fully customizable
historical data breakdown, sampling, filtering and tagging and
triggers. Libpcap, Netlink/ULOG, sFlow v2/v4/v5 and NetFlow v1/
v5/v7/v8/v9 are supported, both unicast and multicast. Also, a
client program makes it easy to export data to tools like RRDtool,
GNUPlot, Net-SNMP, MRTG, and Cacti.
HOMEPAGE.
http://www.pmacct.net/
DOWNLOAD.
http://www.pmacct.net/pmacct-0.12.0rc4.tar.gz
CHANGELOG.
+ BGP-related source primitives are introduced, namely: src_as_path,
src_std_comm, src_ext_comm, src_local_pref and src_med. These add
to peer_src_as which was already implemented. All can be resolved
via reverse BGP lookups; peer_src_as, src_local_pref and src_med
can also be resolved via lookup maps which support checks like:
bgp_nexthop (RPF), peer_dst_as (RPF), input interface and source
MAC address. Many thanks to Zenon Mousmoulas and GRNET for their
fruitful cooperation.
+ Memory structures to store BGP-related primitives have been
optimized. Memory is now allocated only for primitives part of
the selected aggregation profile ('aggregate' config directive).
+ A new 'bgp_follow_nexthop' configuration directive is introduced
to follow the BGP next-hop up to the edge of the routing domain.
This is particularly aimed at networks not running MPLS, where
hop-by-hop routing is in place.
+ Lookup maps for BGP-related source primitives (bgp_src_med_map,
bgp_peer_src_as_map, bgp_src_local_pref_map): result of check(s)
can now be the keyword 'bgp', ie. 'id=bgp' which triggers a BGP
lookup. This is thought to handle exceptions to static mapping.
+ A new 'bgp_peer_as_skip_subas' configuration directive is being
introduced. When computing peer_src_as and peer_dst_as, returns
the first ASN which is not part of a BGP confederation; if only
confederated ASNs are on the AS-Path, the first one is returned
instead.
+ Pre-Tagging: support has been introduced for NetFlow v9 traffic
direction (ingress/egress).
+ Network masks part of NetFlow/sFlow export protocols can now be
used to compute src_net, dst_net and sum_net primitives. As a
result a set of directives [nfacctd|sfacctd|pmacctd|uacctd]_net
allows to globally select the method to resolve such primitives,
valid values being: netflow, sflow, file (networks_file), mask
(networks_mask) and bgp (bgp_daemon).
+ uacctd: introduced support for input/output interfaces, fetched
via NetLink/ULOG API; interfaces are available for Pre-Tagging,
and inclusion in NetFlow and sFlow exports. The implementation
is courtesy of Stig Thormodsrud.
+ nfprobe, sfprobe: new [nfprobe|sfprobe]_peer_as option to set
source/destination ASNs, part of the NetFlow and sFlow exports,
to the peer-AS rather than origin-AS. This feature depends on a
working BGP daemon thread setup.
! A few resource leaks were detected and fixed. Patch is courtesy
of Eric Sesterhenn.
! bgp/bgp.c: thread concurrency was detected upon daemon startup
under certain conditions. As a solution the BGP thread is being
granted a time advantage over the traffic collector thread.
! bgp/bgp.c: fixed a security issue which could have allowed a
malicious user to disrupt established working BGP sessions by
exploiting the implemented concept of BGP session replenishment;
this has been secured by a check against the session holdtime.
Many thanks to Erik van der Burg for spotting the issue.
! bgp/bgp.c: BGP listener socket now sets SO_REUSEADDR option for
quicker turn around times while stopping/starting the daemon.
! net_aggr.c: default route (0.0.0.0/0) was considered invalid;
this is now fixed.
NOTES.
Cheers,
Paolo
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] sflow renormalize not working correctly (sql_preprocess does)
Hi Igor, I'm not aware of any issues with the 'sfacctd_renormalize' feature; perfect you already did some debug with sflowtool - that would have been my first suggestion. I would ask you, if possible, to send me privately a brief capture of some sFlow datagrams (pcap format, full packets) so that i can have a look myself and replay them in a lab environment. This would also help understanding very quickly if the load of the system can be a viable explanation or not. Cheers, Paolo On Mon, Dec 21, 2009 at 09:19:34PM +0100, Igor Ybema wrote: Hi, We are using pmacct (sfacctd) for 2 years now. We are using sflow agent on HP procurve 3500 switches with sampling rate of 1024. Before we didn't use the renormalize function and no preprocessing. We just used kbytes in stead of bytes in our applications which worked fine. Since a month we started to use the sflow_renormalize function but since then we see the bytes counted for our 'incoming' table multiplied by a factor of 1.5 to 2 (not accurate). The 'outgoing' table does not show this behaviour. To figure out where the problem is, I tried a couple of things: - using sflowtool to debug the packets - meanSkip = 1024 and counting MB's myself (with a little help of bash) gives me the total bytes I have send to the test host - using another sflow agent (foundry XMR router) on other sflow server with renormalize on also gives me the correct values - turning of renormalize and using sql_preprocess: usrf=1024 also gives me the correct values - using the newest 0.12.0rc4 does not help So,.. using same sflow server and agents, same data etc... only switching from renormalize to preprocess works for me right now. What could be the problem? The sflow server (combined mysql server) is a busy server hitting high cpu and io waits. Could this cause issue's? regars, Igor Ybema ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Aggregate traffic count with BGP community match
Hi Andrew, On Fri, Dec 11, 2009 at 01:32:34PM +1300, Andrew Thrift wrote: I have a requirement to count clients traffic passing through our border router by classes, e.g. local (iBGP), national (BGP marked with specific community) and international (anything not covered by local/national), after a bit of research I found pmacct/nfacct and from what I can tell it looks like it can do what I want. That's correct. More sophisticated considerations (ie. whether leveraging communities is the best choice) can be made only by knowing the overall picture. Have aggregation periods per client IP of 1 Hour, 1 Day and 1 Month (1st day to last day) Yes. Temporal aggregation is fully flexible, so you can really choose whichever time period makes sense to you. Indeed per IP or per subnet accounting is possible. Use match filters similar to: [ ... ] Match filters: I guess you would like to base your solution on a SQL backend. BGP communities, IP addresses, etc. are all visible in the database: all revolves around crafting the right SQL queries. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Layer 7 classification problem.
Hi, Mike kindly offered access to his box. The classification engine was found to work just fine - issue being related to some catch-all patterns included with the L7-filter classifiers package (namely unknown.pat and unset.pat). By removing them, classification was working much better. Indeed, I don't enter the discussion of the precision of the 3rd party regex classifiers. Action i've taken is to add a note in the documentation. Cheers, Paolo On Wed, Dec 02, 2009 at 10:53:04AM +, Paolo Lucente wrote: Hi Buddhike, On Wed, Dec 02, 2009 at 02:36:20PM +0530, Buddhike wrote: http://www.mail-archive.com/pmacct-discussion@pmacct.net/msg01397.html Any chance you can give it a try and post some feedback, ie. whether it yeah I've tried that with pmacct-0.12.0rc3 and still the result is same. Thanks very much for your feedback; i believe Mike will then be on the same lines. Now, I'd like to help you but as i don't manage to reproduce the problem with my resources, to make it effective at this stage i would need access to your installations. If this is a viable option for either of you, please contact me privately to arrange this up. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Not save data in DB when exit
Hi Slava, On Tue, Dec 01, 2009 at 10:01:28PM +0200, Slava Dubrovskiy wrote: Yes, I remove sql_multi_values and it working. But as I understand this this give more performance. Was suspecting so correct. Is it possible use it? And it working good. Problem only when I stop it. It has to be identified what is causing it. The problem it flags is duplicates within the primary key. This is because the UPDATE query cannot be used as part of the sql_dont_try_update (which, once again gives better performances, so good it is enabled). You should get a few examples of which rows are clashing (take one duplicate error message and search for that row in the database). If stopping-starting-stopping pmacct in a very short sequence you might obvioulsy have to clean up the database for the test to be consistent. Verify whether the clash makes sense or not - given your schema and configuration. One last note: last time you posted your config, i recall you did enable 'nfacctd_time_new' which is recommended when using NetFlow in conjunction with these SQL directives. Is it still the case, right? Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Layer 7 classification problem.
Hi Buddhike, On Wed, Dec 02, 2009 at 02:36:20PM +0530, Buddhike wrote: http://www.mail-archive.com/pmacct-discussion@pmacct.net/msg01397.html Any chance you can give it a try and post some feedback, ie. whether it yeah I've tried that with pmacct-0.12.0rc3 and still the result is same. Thanks very much for your feedback; i believe Mike will then be on the same lines. Now, I'd like to help you but as i don't manage to reproduce the problem with my resources, to make it effective at this stage i would need access to your installations. If this is a viable option for either of you, please contact me privately to arrange this up. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Not save data in DB when exit
Hi Slava, On Mon, Nov 30, 2009 at 03:03:41PM +0200, Slava Dubrovskiy wrote: I see such errors: [ ... ] DEBUG ( t2/mysql ): 3 VALUES statements sent to the MySQL server. ERROR ( t2/mysql ): Duplicate entry '0-5-2009-11-28 02:00:00' for key 1 [ ... ] DEBUG ( t1/mysql ): 400 VALUES statements sent to the MySQL server. ERROR ( t1/mysql ): Duplicate entry '0-0.0.0.0-208.94.173.101-0-0-udp-2009-11-28 02:00:00' for key 1 Thanks for the extensive logging. You should be referring to the two errors above, right? To better pin-point where the issue is, it's good rule to remove the sql_multi_values: this allows to see precisely which SQL statement is causing the issue. From there you can check whether effectively this is something already in the database and why. The fact that you don't see anything written into the database when such errors pop up is normal due to the combination of the sql_multi_values and sql_dont_try_update directives. A multi-value query is effectively a single statement; so if one component fails it negatively affects all the others. If this doesn't help; feel free to send me further SQL output, commenting out the sql_multi_values. Perhaps this will get too extensive; in such a case, send it privately and then we can summarize over here. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] peer-AS calculation problem
Hi Zenon, On Wed, Nov 25, 2009 at 12:59:04PM +0200, Zenon Mousmoulas wrote: I am not sure if this affects nfacctd or, perhaps, if it overrides this information by looking up the next-hop (and perhaps also the dst peer AS) in the BGP RIB? If i'm not mistaken you are not using the 'nfacctd_as_new' configuration directive in your config - or it's not set to 'bgp'. If this is the case, that key causes fields that can be grasped from multiple sources, ie. BGP and NetFlow, like src_as, dst_as, bgp_nexthop to be looked up against BGP. Default value is to look them up in the export protocol (NetFlow, sFlow). Still if this holds, can you give it a try? Let me know. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] pmacct-contribs 20091125 released
VERSION. 20091125 DESCRIPTION. pmacct is a set of network tools to gather, filter and tag IP traffic; it is able to store collected data either into a DB or a memory table. We see any monitoring, billing or accounting environment as a stack where data are picked from the network, get processed in a meaningful way, and optionally formatted to be user-friendly. In the context of this layered approach we also see pmacct residing at its lowest layer (that is, just upon the network); once collected, often data need to be pulled to upper layers of the environment for a range of applications like time-based reporting, pricing model application, policy enforcement, network thresholds handling, nice output to end-users. pmacct-contribs aims to be over the time a collection, though not exaustive, of scripts and works revolved around or topped over pmacct. README. http://www.pmacct.net/contribs-README DOWNLOAD. http://www.pmacct.net/pmacct-contribs-20091125.tar.gz CHANGELOG. + (Stig Thormodsrud) quagga_gen_as_network.pl: a script to generate a AS number/network file (networks_file) in pmacct format from Quagga. NOTES. README contains the contributions list. New contributions are welcome; support to existing contributions is also appreciated. Questions, critics, suggestions, advices to existing contributions must be sent to respective authors. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Not save data in DB when exit
Hi Slava, On Wed, Nov 25, 2009 at 09:04:24PM +0200, Slava Dubrovskiy wrote: Seems when I make kill INT PID_OF_CORE_PROCESS it down, but plugins do not write to database. I see delay before off for plugins, but not see that they change command line to DB writer. And not see data for period. Can you confirm that bug? You should send a SIGINT to the plugins you want to write to the database, not only to the core process (just wondering if it's written this way in any part of the documentation). A 'killall INT pmacctd' should do it; or if you need better granularity use the 'pidfile' directive to be able to retrieve the PID for the plugins aswell. Let me know how it goes. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] A strange thing with one traffic direction
Hi Peter, On Mon, Nov 23, 2009 at 10:45:38PM +0100, Peter Franzel wrote: I thing RX Traffic is brilliant, but why is there such a big difference between TX bytes?! Is there something I am going wrong or where is the fault? I would essentially suggest to go in a couple of directions; first do a bit of comparisons with tcpdump; pmacct would see what tcpdump sees; except a) tcpdump considers non-IP traffic (ie. ARP) and b) pmacct considers IPv6 traffic if configured with the --enable-ipv6 switch. If that still doesn't help, simplify your configuration for the duration of the troubleshooting phase; then proceed step by step. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Layer 7 classification problem.
Hi Mike,
On Mon, Nov 23, 2009 at 02:00:04PM +0300, Mike Lykov wrote:
By the way, L7-filter have two types of filter:
The first speed shown for a pattern in the tables below is the speed when
used in the kernel (with the old V8 regular expression library). The second
is the speed when used in userspace (with the modern GNU library).
i.e.userspace version uses GNU regex(7) library. Are pmacct using it also?
pmacct only embeds the first one. I've written for you a quick patch
(attached) which leverages regex coming with the SO (typically POSIX
compliant). It works for me on a Linux (x86_64) and a Solaris (sparc)
with the http classifier from L7-filter project - but it's also true
everything was working fine at some quick testing before.
Let me know if you see any improvements. If this is the case, i can
either replace the current implementation or add a knob in the coming
release. The patch should apply fine to 0.12.0rc3; if not, please use
code in the CVS.
Cheers,
Paolo
diff -ur pmacct/src/classifier.c pmacct.class/src/classifier.c
--- pmacct/src/classifier.c 2009-10-31 12:46:10.0 +0100
+++ pmacct.class/src/classifier.c 2009-11-23 13:57:11.0 +0100
@@ -157,7 +157,13 @@
payload[y] = '\0';
while (class[j].id j max) {
- if (class[j].pattern) ret = pm_regexec(class[j].pattern, payload);
+ char match;
+
+ if (class[j].pattern.buffer) {
+ ret = regexec(class[j].pattern, payload, 0, (regmatch_t *) match, 0);
+ if (!ret) ret = 1;
+ else ret = 0;
+ }
else if (*class[j].func) {
cc_node = search_context_chain(fp, idx, class[j].protocol);
cc_rev_node = search_context_chain(fp, reverse, class[j].protocol);
@@ -370,11 +376,12 @@
Log(LOG_ERR, ERROR: Pattern in %s too long. A maximum of %d chars is allowed.\n, fname, MAX_PATTERN_LEN);
return 0;
}
- css-pattern = pm_regcomp(pre_process(line), linelen);
- if (!css-pattern) {
- Log(LOG_ERR, ERROR: Failed compiling regular expression for protocol '%s'\n, css-protocol);
- return 0;
- }
+ memset(css-pattern, 0, sizeof(css-pattern));
+ ret = regcomp(css-pattern, pre_process(line), REG_EXTENDED|REG_ICASE|REG_NOSUB);
+ if (ret) {
+Log(LOG_ERR, ERROR: Failed compiling regular expression for protocol '%s' (%u)\n, css-protocol, ret);
+return 0;
+ }
datatype = done;
break;
@@ -452,7 +459,7 @@
}
link_conntrack_helper(css);
- css-pattern = NULL;
+ memset(css-pattern, 0, sizeof(css-pattern));
return 1;
#else
return 0;
diff -ur pmacct/src/classifier.h pmacct.class/src/classifier.h
--- pmacct/src/classifier.h 2006-11-19 16:16:07.0 +0100
+++ pmacct.class/src/classifier.h 2009-11-23 13:57:41.0 +0100
@@ -1,5 +1,5 @@
#include dirent.h
-#include regexp.h
+#include regex.h
#include conntrack.h
/* defines */
@@ -27,7 +27,7 @@
struct pkt_classifier {
pm_class_t id;
char protocol[MAX_PROTOCOL_LEN];
- regexp *pattern;
+ regex_t pattern;
pm_class_t (*func)(struct pkt_classifier_data *, int, void **, void **, void **);
conntrack_helper ct_helper;
void *extra;
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] MySQL options
Hi Joel, On Sat, Nov 21, 2009 at 12:43:19PM +1100, Joel Roberts wrote: I need to setup traffic accounting (in and out) for each IP address, and then export that data to an EXTERNAL mysql database on a separate machine accessible via IP address. How do I go about setting up pmacct to do this? On the pmacct side, look at the sql_host configuration directive. Look at MySQL documentation on how to a) let the daemon listen on a TCP/IP port and b) setup user accounts which are going to connect remotely (GRANT queries). There are plenty of examples on how to account in/out traffic per-IP address around - some as part of the EXAMPLES document, some on Google. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Layer 7 classification problem.
Hi, On Fri, Nov 20, 2009 at 05:06:25PM +0530, Buddhike wrote: I'm testing pmacct on my network, and pmacct runs on a box inbetween my LAN switch and My ADSL router, and i'm using layer 7 classifires for classify traffic. But when testing I observed that the traffic is not correctly displayed according to the classification some of the problems i've seen are listed below. 1. Most of the http traffic are classifed as finger. 2. All the other traffic are also shown either as unknown or in some other non-relevant protocols. There was a very similar thread going on few days ago - and still not solved; you can check it out at the link below: http://www.mail-archive.com/pmacct-discussion@pmacct.net/msg01374.html You can perform some troubleshooting as per that thread and see if it helps. Otherwise, it has to be debugged and at a glance feeling is that the issue is not lying in the pmacct domain. FYI, pmacct leverages both regexp engine and classifiers of the L7-filter project. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] invalid network mask?
Hi Charlie,
It appears you didn't put the table in the correct format first. Is
it the case? If yes, to make that table pmacct-friendly you have to
pre-process it as follows:
cat bgptable | sed 's/\([0-9a-f:][0-9a-f\.\/:]*\).* \([0-9][0-9]*\)[
0-9,{}]*$/\2,\1/' | uniq networks.lst
At least it works fine with GNU sed. And do consider this was really
a 5 minutes job, so double-check accuracy of the output.
Btw, indeed yes line numbers should be included in logging messages
upon detecting errors; this is the case for other checks, apparently
not for this one. Just fixed it, will commit a patch to the CVS soon.
Cheers,
Paolo
On Thu, Nov 19, 2009 at 11:54:05AM +, Charlie Allom wrote:
Hi Paolo,
would it be useful to include the line number that causes the error
here?
I've put the file here: http://capslock.playlouder.com/bgptable.gz
11:50 pmacct:~# md5sum bgptable
578e6138f9e5b3d84c09415ce9d1d019 bgptable
NetFlow Accounting Daemon, nfacctd 0.12.0rc3
11:48 pmacct:~# nfacctd -f /etc/nfacctd.conf
INFO ( default/pgsql ): 131070 bytes are available to address shared memory
segment; buffer size is 1024 bytes.
INFO ( default/pgsql ): Trying to allocate a shared memory segment of
33553408 bytes.
ERROR ( /home/charlie/bgptable ): Invalid network mask '2147483647'.
[ ... ]
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Problem with aggregate_filter
Hi, On Thu, Nov 19, 2009 at 04:00:33PM +0530, Buddhike wrote: I've searched on google but didn't find any examples with mac adresses but with src/dst networks. But i didn't seen any records saying that aggrigate_filter can't be used with src mac and dst mac option. anyway I tested it with the following config file but it didn't worked. Remember filters in pmacct follow tcpdump/libpcap syntax - so you can always check tcpdump man page when in doubt. The correct syntax is ether src and ether dst. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] create my own mysql table
Hi, On Mon, Nov 16, 2009 at 04:45:57PM -0600, fedora fedora wrote: DEBUG ( default/mysql ): INSERT INTO `test_1` (stamp_updated, stamp_inserted, ip_src, ip_dst, as_src, as_dst, src_port, dst_port, tcp_flags, ip_proto, packets, bytes, flows) VALUES (FROM_UNIXTIME(1258410661), FROM_UNIXTIME(1258410600), 'x.x.x.34', 'x.x.x.2', xx8, xx9, 443, 2608, 24, 'tcp', 1, 1353, 140733193388033) Thanks, for the output. So, only the 'flows' primitive is involved. Hopefully last bit of information i need is: which NetFlow version are your routers exporting to pmacct? If v8, which profile? If v9, you doing anything fancy with it (ie. aggregated NetFlow)? Roughly a week ago i committed to the CVS a minor patch to initialize some variables used at some stage to convert values as counters; can you please see if the version currently in the CVS behaves any better? So pmacct keeps tracking the traffic count and and the end of the given minutes(hours..etc) it calculates the summary and then writes it to the backend database, am I right? Yes pmacct SQL plugins feature a cache to accumulate counters. Then the scanner kicks in at regular intervals (by default 60 secs) and writes to the database. If sql_history matches sql_refresh_time (or is a multiple) then each aggregate is written with a SQL INSERT query; otherwise UPDATE queries are involved. If I am correct, how does pmacct treat netflow data? since all the data it gets already get aggregated by netflow protocol. Will pmacct do something extra? I guess for sflow, it will act differently and do the calculations. Basic thing to consider is pmacct is not a packet/sample/flow logger. This is partly highlighted by Q5 in FAQS. It performs data reduction, ie. temporal aggregation, spatial aggregation, filtering, sampling (or sub-sampling). Timestamps part of the export protocol, ie. NetFlow or sFlow, are used to assign data to a time-bin, when using a SQL plugin (as the in-memory table plugin doesn't have such a concept; you simply grabclean data at regular intervals). Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] protocol classification don't detect http
Hi Mike, On Tue, Nov 17, 2009 at 02:27:06PM +0300, Mike Lykov wrote: I would suggest a couple of checks: * see if HTTP traffic is reaped by some other classifier, but i guess you might have already checked that. if class_id = unknown, i think it's not this case. Yes, correct. But are you getting all the web traffic? I mean, I see you are a) not collecting TCP/UDP ports and b) using an aggregate_filter. Is it web traffic the one left as unknown or something else? Any chance some web traffic is being filtered out, ie. because some mirrored data is VLAN-tagged? You can test this by commenting out the aggregate_filter. * see if the HTTP classifier is written correctly. Not referring only to the regexp but to the overall syntax. The implemented format is *veeery* sensible to tabs, spaces, white lines, etc. So try to keep it essential. Strip comments and empty lines out. I delete all comments from file [r...@router ~]# cat /var/local/pmacct/classifiers/http.pat http http/(0\.9|1\.0|1\.1) [1-5][0-9][0-9] [\x09-\x0d -~]*(connection:|content-type:|content-length:|date:)|post [\x09-\x0d -~]* http/[01]\.[019] What else may I try to? Try with a simplified (and polished up) filter. See if the memory table plugin behaves any differently/better compared to the SQL one (this is an always-good troubleshooting step). Increase classification tentatives although with http traffic it should make no difference. After all, if http-marked traffic makes it in the database, as per your previous email, it means the regexp engine itself is working. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] protocol classification don't detect http
Hi Mike, I see all of those signatures actually working by picking some sites randomly with wget. This is with 0.12.0rc3 but honestly speaking there has not been any major work related to the classification part for the past 3-4 years. I would suggest a couple of checks: * see if HTTP traffic is reaped by some other classifier, but i guess you might have already checked that. * see if the HTTP classifier is written correctly. Not referring only to the regexp but to the overall syntax. The implemented format is *veeery* sensible to tabs, spaces, white lines, etc. So try to keep it essential. Strip comments and empty lines out. Let me know. Cheers, Paolo On Mon, Nov 16, 2009 at 01:13:03PM +0300, Mike Lykov wrote: Hi all on this list. I am try to install pmacct + protocol classification feature and want to ask some question about it. pmacct + pmacct_v5 base + set of .pat files from l7filter site. See results: successfully detect ftp,nntp,subversion,jabber,ssh,dns,pop3,smtp detect connection to jabber-icq gate as rtp detect ntp as edonkey don't detect http and http-ssl at all don't detect irc (tested on irc.freenode.org + irssi), whois (whois.ripn.net + console whois) False detections rtp/edonkey is a little inconvenience, but not to detect http at all is a big disappointment! I try some variants of regexp: simple HTTP/(0\.9|1\.0|1\.1) [1-5][0-9][0-9] default http/(0\.9|1\.0|1\.1) [1-5][0-9][0-9] [\x09-\x0d -~]*(connection:|content-type:|content-length:|date:)|post [\x09-\x0d -~]* http/[01]\.[019] second default http/(0\.9|1\.0|1\.1) [1-5][0-9][0-9]|post [\x09-\x0d -~]* http/[01]\.[019] [ ... ] Anybody here with http classification working? ;) ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] create my own mysql table
Hi, On Mon, Nov 16, 2009 at 11:58:14AM -0600, fedora fedora wrote: I still see all flow records having the same number 4294967295 in my mysql table, and debug does not seem to tell me why this happens. How do you mean? You see that number appearing in the debug? As writing to the database is done through a library (MySQL client library), I'm trying to establish what is generating the issue. Can you also say what version of MySQL you are using, which OS and which architecture you are running pmacct on? And I have another question about the sql_history, what exactly is this? I have read the configuration keys several time but I am still not sure. If I set it to 5m, does it mean pmacct will hold and count the data internally until the end of the 5 minutes then send the final statistic to mysql? It means you have 5 minutes traffic counters (temporal aggregation). The SQL cache scanner sets how often you write to the database. You can have a sql_history set to 5 minutes but have the SQL scanner to kick in every minute (sql_refresh_time: 60). There are more advanced considerations to do at this propo - which are useless to start with and that you can read of in CONFIG-KEYS, sql_dont_try_update feature. Overall, best is to start blank with relevant sections of the sql/README.mysql and EXAMPLES documents in the distribution tarball - and then complicate things bit by bit. Btw, sql_history_roundoff is also a very useful directive to be coupled almost always with sql_history, for example: sql_history: 5m sql_history_roundoff: h It nicely aligns the time bins to the hour boundary. Again, perhaps give it a try to see the effect. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] create my own mysql table
Hi, On Fri, Nov 13, 2009 at 04:21:26PM -0600, fedora fedora wrote: Thanks for the reply, i disable the daemon option and here are the error messages, i cannot put src_ip and src_as together? INFO ( default/mysql ): 131070 bytes are available to address shared memory segment; buffer size is 128 bytes. INFO ( default/mysql ): Trying to allocate a shared memory segment of 2097024 bytes. INFO ( default/core ): waiting for NetFlow data on 0.0.0.0:1 ERROR ( default/mysql ): SQL tables v6 are unable to mix IP addresses and AS numbers (ie. src_ip, src_as). INFO: connection lost to 'default-mysql'; closing connection. Simply add to your config a 'sql_table_version: 6' line. Then, the 'sql_optimize_clauses' directive ensures you can customize your table. Q16 in the FAQS document explains the concept of SQL table versioning in pmacct. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] create my own mysql table
Hi, On Fri, Nov 13, 2009 at 05:10:35PM -0600, fedora fedora wrote: sorry, one more question, when preparing the aggregate, does the order of the values matter? right now my aggregate is like the following No, doesn't matter. One more question, how can i get pmacct to show the flow number in the database? I got all 0 ,and when i change to aggregate to aggregate:proto,src_host, dst_host, src_port, dst_port, src_as, dst_as, tcpflags,flows all the numbers are one same pretty big number MMM, sounds strange. Also, don't manage to reproduce either of the two scenarios by playing your configuration on a test box. The same pretty big number smells like dirty data? Enable debug and you should see on the screen which SQL queries are generated by pmacct and sent to the MySQL database. Also i will have to read what exactly does stamp_inserted , and stamp_update mean.. pmacct has a concept of time bins - which you are currently not using in your config (but you can easily add the line sql_history: 5m and see). sql_inserted is the base time of the time bin; sql_history directive in the configuration sets the length of such time bins; stamp_updated says about when a specific SQL record was touched for the last time by pmacct. Useful for troubleshooting purposes and general reference - not to infer time bins' length. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] NAT question
Hi JF, As Karl said, libpcap looks what's on the wire and pmacct doesn't get further up in the packet layering. You can always do a quick check by verifying what tcpdump sees. While on NAT Linux, and perhaps not related to this specific issue: the uacctd daemon has been introduced in pmacct as of 0.12.0rc3: it relies on the ULOG framework for packet capturing and should give increased flexibility (prerouting, postrouting, etc.) in scenarios where one does accounting on the same Linux box which is also doing the NAT. Cheers, Paolo On Thu, Nov 12, 2009 at 12:24:34AM -0500, JF Cliche wrote: Maybe a newbie question, so I'll be brief: I am behind two NAT routers (Linksys running DD-WRT) with port forwarding up to the machine running pmacct, and yet pmacct reports SSH traffic to the forwarded port with the public (external, non-NATed) addresses. I thought all traffic should be seen as coming from the second router private address. Is pmacct (or underlying pcab library) getting the public address from extra data encapsulated in the TCP packets by the routers or in the SSH protocol? I've seen the opposite problem being discussed in this forum, but not this... JF -- Jean-Fran?ois Cliche, Ph.D., P. Eng ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] No Sqlite3 logging
Hi JF, On Mon, Nov 09, 2009 at 05:09:40AM -0500, JF Cliche wrote: debug: true syslog: kern plugin_buffer_size: 10240 plugin_pipe_size: 10240 interface: egiga0 daemonize: false promisc: false sql_cache_entries: 2 plugins: sqlite3[in] sql_max_writers: 2 pidfile: /vol1/pmacct.pid sql_db: /vol1/pmacct.db aggregate[in]: dst_host aggregate_filter[in]: dst net 11.0.3.0/24 sql_table[in]: acct sql_table_version: 1 sql_refresh_time: 5 From the debug output you posted, it looks everything is OK (database exists, file permissions are fine, etc.) - except pmacct doesn't have any data to report. Can you please comment the 'aggregate_filter' out? Another chance is that the daemon is bound to an interface with almost no traffic and hence the plugin_buffer_size/plugin_pipe_size couple is buffering and hence hiding traffic out. While here, i would suggest to review the following things from the above config - perhaps good if/when you go in production: * remove the 'sql_cache_entries: 2' line or perhaps explain the goal you want to achieve. This sets the number of buckets of the hashed SQL cache structure and can lead to sustained CPU utilization, if configured incorrectly. * remove the 'sql_max_writers: 2' line to at least 10; or increase sql_refresh_time * increase sql_refresh_time to something more sensible 60-120 secs; unless indeed this is done on purpose. * Try to keep plugin_pipe_size in the 1000*plugin_buffer_size area Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] No Sqlite3 logging
Hi JF, On Mon, Nov 09, 2009 at 10:26:35AM -0500, JF Cliche wrote: In any case, I cleaned- up my config file and made sure I filter nothing (see config below). I rechecked pmacctd using the memory plugin and data is being gathered. Then I relaunched with the sqlite3 plugin. 'pmacct -s' still generate an error: the pipe file is not created. And a large number of sqlite3 writer processes are created. The pmacct client tool is only useful to interact with the memory plugin; when you are using only the sqlite3 plugin, it tries to see if a memory plugin is listening on the pipe file - but nothing is there and hence it triggers the error. I would say this is all OK. ... ( in/sqlite3 ) *** Purging cache - START *** ( in/sqlite3 ) *** Purging cache - END (QN: 0, ET: 0) *** WARN ( in/sqlite3 ): Maximum number of SQL writer processes reached (10). ( in/sqlite3 ) *** Purging cache - START *** ( in/sqlite3 ) *** Purging cache - END (QN: 0, ET: 0) *** ... Now, this part is interesting. As you said, writer processes seem to not be replenished - while they should be shortly after each END debug message you see above; and also it seems the writer process is unable to dump the SQL cache content into the database (as the query number, QN, stays at zero while you produce some traffic; moreover such traffic is reaching the sqlite3 plugin no problems as you can get the buffering error ERROR ( in/sqlite3 ): We are missing data from the plugin). To rule out some configuration issue, i've copy/pasted your config on a testbed - and i see it working fine. This and the fact that the memory plugin works fine (it has a different way of working compared the SQL plugins) suggests me the issue could be at a lower level. I can propose some solutions: 1) allow me to troubleshoot the issue; this would require an access to the NAS (and feel free to bring this over privately); 2) if the above is not possible a clean alternative might be let pmacct to write to a remote SQL database, either MySQL or PostgreSQL; it has to be checked you don't run in the same SQLite issue first; 3) check whether what you want to achieve, can be done with the memory plugin. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] tracking traffic flowing through a specific AS-path
Hi Zenon, On Thu, Oct 29, 2009 at 10:09:16AM +0200, Zenon Mousmoulas wrote: BGP RIB lookups. However, for traffic flowing from sources behind that path (inbound for us), how would one go about doing just that? [ ... ] This is quite similar in concept to the mechanics of calculating the source peer AS: an RPF lookup would be required again, which should return the AS-path instead of the next-hop. Of course once again we have the assumption that routing is symmetric, which however in internet environments is far from the truth, but it's still better than nothing. How difficult would it be to implement this so that we could have a src_as_path aggregation primitive? Implementing a trivial src_as_path is quick and easy ... but is that any useful? It's something non-deterministic and it's also not always possible to infer whether it's correct or pure rubbish. When i posed myself that question some time ago, i thought that even if a trivial implementation takes just a couple of hours, bottom line it's time wasted. I guess on a topic like src_as_path there is margin for a study, some research or at least some ample discussion maybe bringing in further interested people (coming from the NREN world, you perhaps can think of somebody at this propo?) with the goal of coming up with something clever. Recently, for example, having spoken to somebody on similar topics i started wondering if further information about ingress traffic is better inferred from the network itself (say: BGP, NetFlow, TTL values, etc.), from out-of-band data (say: AS macros, RIPE data, etc.) which would be all part of a map or a mix of the two. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Long sflow BGP configuration
Dear Maxence, On Wed, Oct 28, 2009 at 10:56:20AM +0100, Maxence Rousseau wrote: I didn't understood all the documentation... Do I need to use pmacctd ? It seems not necessary because sfacctd is able to launch BGP thread and log to mysql. What's the need of the pmacctd process in my case ? Correct. Getting data via sFlow, you have to deal with sfacctd only. The logging to the MySQL db is working correctly but a lot of information are missing : DEBUG ( default/mysql ): INSERT INTO `acct_bgp` (stamp_updated, stamp_inserted, as_src, as_dst, as_path, local_pref, med, peer_as_src, peer_as_dst, peer_ip_src, peer_ip_dst, agent_id, comms, packets, bytes) VALUES (FROM_UNIXTIME(1256724001), FROM_UNIXTIME(1256723400), 34383, 0, '', 100, 0, 0, 0, '192.168.5.1', 'WAN_IP', 0, '', 2, 509) As you can see, in my configuration below, i've set the bgp update-source IP the same than sflow agent-ip. Is it enough to able to match informations between sflow bgp ? How can I see than the match is working ? The correlation between sFlow and BGP is working as, as it can be seen from the query above src_as, local_pref and peer_ip_dst (BGP next-hop) are resolved fine. All of that comes from BGP. As what is missing above is roughly as_dst, peer_as_dst, as_path (and there are no communities attached): is it not possible this is something redistributed in BGP (ie. a static or a connected route or a route injected from some other protocol)? Two troubleshooting steps that helped recently (thanks to Zenon from GRNET for his cooperation at this propo) are: * temporarily set to true 'bgp_daemon_msglog' to have a debug of all the BGP routes (and their attributes) being exported from the peer(s) * temporarily increase resolution of the aggregation, ie. add src_host and/or dst_host, so that specific issues can be pin-pointed. (I cannot change it), in the documentation the match between sflow BGP is done with the source IP address from BGP sflow, so the router-id seems not useful except for more clear log messages. Correct ? Correct. Some information are available in sflow (viewed with sflowtool) like my_as,src_peer_as but are not logged. Why ? Because we wait them from the BGP ? Correct. Disabling the 'sfacctd_as_new' makes as_src and dst_as populated from sFlow and the rest from BGP. peer_as_src is instead populated via a map: see pages 15-18 of http://www.pmacct.net/lucente_pmacct_uknof14.pdf Your pmacct and router configs also appears correct. When all will be up, I will be able to post on the wiki a complete configuration step by step for this exact configuration. Thanks, that would indeed be much appreciated. You (or I if you give me the link at some stage) can cross-link it from the pmacct wiki to give it even further visibility. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Long sflow BGP configuration
Hi Maxence, On Wed, Oct 28, 2009 at 03:38:59PM +, Maxence Rousseau wrote: The information seems perfect, sample : INFO ( default/core/BGP ): [Id: ROUTER-ID] u Prefix: '208.74.16.0/21' Path: '174 3549 32952' Comms: '174:21000 174:22013' EComms: '' LP: '100' MED: '82011' Nexthop: 'WAN_IP_ROUTER_TRANSIT_AS_174' [ ... ] 192.168.5.1 PEER_DST_IP SRC_IPDST_IP 0 0 0 ip 0 1 0 200 What is not clear to me: is DST_IP part of the IP prefix listed above or the two examples are unrelated? Otherwise: what are the BGP attributes related to the IP prefix of which DST_IP is part of? If you feel you want to go private, in order to give me a clear example, and then summarize on-list feel free. Btw, for troubleshooting purposes, it can also be useful to know what is the role of the sFlow/BGP exporter in the network, which interfaces/direction sFlow is enabled (is it on the IP transit ports, ingress direction or what?) and a brief description of the network topology. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] pmacct 0.12.0rc3 released !
VERSION. 0.12.0rc3 DESCRIPTION. pmacct is a small set of passive network monitoring tools to measure, account, classify, aggregate and export IPv4 and IPv6 traffic; a pluggable and flexible architecture allows to store collected network data into memory tables or SQL (MySQL, SQLite, PostgreSQL) databases and export them through NetFlow or sFlow protocols to remote collectors. pmacct supports fully customizable historical data breakdown, sampling, filtering and tagging and triggers. Libpcap, Netlink/ULOG, sFlow v2/v4/v5 and NetFlow v1/ v5/v7/v8/v9 are supported, both unicast and multicast. Also, a client program makes it easy to export data to tools like RRDtool, GNUPlot, Net-SNMP, MRTG, and Cacti. HOMEPAGE. http://www.pmacct.net/ DOWNLOAD. http://www.pmacct.net/pmacct-0.12.0rc3.tar.gz CHANGELOG. + Support for NetFlow v9 sampling via Option templates and data is introduced; this is twofold: a) 'nfacctd_renormalize' configuration directive is now able to renormalize NetFlow v9 data on-the-fly by performing Option templates management; b) 'nfprobe', the NetFlow probe plugin, is able to flag sampling rate (either internal or external) when exporting flows to the collector. + '[pm|u]acctd_ext_sampling_rate' directives are introduced to support external sampling rate scenarios: packet selection is performed by the underlying packect capturing framework, ie. ULOG, PF_RING. Making the daemon aware of the sampling rate, allows to renormalize or export such information via NetFlow or sFlow. + pmacctd: the IPv4/IPv6 fragment handler engine was reviewed to make it sampling-friendly. The new code hooks get enabled when external sampling (pmacctd_ext_sampling_rate) is defined. + A new 'uacctd' daemon is added to the set; it is based on the Netlink ULOG packet capturing framework; this implies it works only on Linux and can be optionally enabled when compling by defining the '--enable-ulog' switch. The implementation is fully orthogonal with the existing feature set. Thanks very much to: A.O. Prokofiev for contributing the original idea and code; Stig Thormodsrud for his support and review. + The 'tag2' primitive is introduced. Its aim is to support traffic matrix scenarios by giving a second field dedicated to tag traffic. In a pre_tag_map this can be employed via the 'id2' key. See examples in the 'examples/pretag.map.example' document. SQL plugins write 'tag2' content in the 'agent_id2' field. Read 'sql/README.agent_id2' document for reference. + Some new directives to control and re-define file attributes written by the pmacct daemons, expecially when launched with increased priviledges, are introduced: file_umask, files_uid, files_gid. Files to which these apply include, ie. pidfile, logfile and BGP neighbors file. ! fix, bgp/bgp.c: upon reaching bgp_daemon_max_peers threshold, logs were flooded by warnings even when messages were coming from a previously accepted BGP neighbor. Warnings are now sent only when a new BGP connection is refused. ! fix, nfprobe/netflow9.c: tags (pre_tag_map, post_tag) were set per pair of flows, not respecting their uni-directional nature. It was generating hiding of some tags. ! fix, nfprobe/netflow9.c: templates were (wrongly) not being included in the count of flows sent in NetFlow v9 datagrams. While this was not generating any issues with parsing flows, it was originating visualization issues in Wireshark. ! fix, SQL plugins: CPU hitting 100% has been determined when sql_history is disabled but sql_history_roundoff is defined. Thanks to Charlie Allom for reporting the issue. ! fix, sfacctd.c: input and output interfaces (non-expaneded format) were not correcly decoded creating issues to Pre- tagging. Thanks to Jussi Sjostrom for reporting the issue. NOTES. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] peer-AS calculation (plus netflow v9 sampling) questions
Hi Zenon, On Tue, Oct 27, 2009 at 01:16:06PM +0200, Zenon Mousmoulas wrote: 1. Does bgp_peer_src_as_map apply both to src and dst AS or only the first? In any case, I don't understand how bgp_nexthop can be used for calculating the peer src AS, since that field applies to the destination recorded in the flow record? As the name says, bgp_peer_src_as_map applies to the src peer AS only. bgp_nexthop check in the context of bgp_peer_src_as_map is employed to do a reverse lookup on the source IP prefix and check the bgp_nexthop (ie. where the ingress router would route to the source IP prefix) is what you expect (ie. traffic is symmetric). Makes sense? 2. Does bgp_peer_src_as_map work with as-aggregated netflow? Actually a broader question is if any of BGP lookup features work when using any form of aggregated flow records where src/dst IP address is not available. Broader question is good: correlation of NetFlow and BGP is done by looking up source and destination IP prefixes in NetFlow into the BGP RIB. If you aggregate IP addresses logically to the network boundaries - then it's still OK. If you aggregate by removing the IP layer at all and export only AS numbers, well, it will not work anymore ... but you are raising a very valid point. IHMO, it would be good to put some efforts in this direction - specifically, before even developing something general-purpose, it should be seen how much accuracy is affected by rather common de-aggregation or prefix load-balancing practices on the internet: ie. i have one AS, two /18 allocations, two upstream provider: i'll advertise a /18 here and one there. 3. In BGP peerings it is quite common practice to advertise next-hop self for a number of different, valid reasons. I am right to assume such a practice effectively renders useless any next-hop based lookups? iBGP or eBGP? Can you bring an example? On an different topic: I have seen that nfacctd detects and supports netflow v5 sampling, but I am not sure if this is also true for v9. It may have been mentioned somewhere/discussed previously, but I have been unable to dig it up. I am fully aware however that detecting sampling in netflow v9 PDUs does require a lot more work... NetFlow v9 sampling is supported as of 0.12.0rc3 (which is currently in the CVS and due to be released ... soonish). Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] dumping stored netflow into nfacctd
Hi Charlie, The 'nfacctd_time_new' directive is by default set to false, meaning original timestamps would be honoured (and aggregates inserted in the correct time-bin). Would also suggest to ensure keeping the 'sql_dont_try_update' to false (which is the default) - as you are going to do something which is not real-time (hence the 'sql_startup_delay' can't be of help) and you might incur into the situation something has to be UPDATEd. Cheers, Paolo On Mon, Oct 26, 2009 at 02:39:40PM +, Charlie Allom wrote: Hi, I have multiple gigs of stored netflow data from nfacpd. Would 'replaying' it back into nfacctd cause the process to recognise any old timestamps or would it think it's new traffic? eg. # for i in * ; do nfreplay -H 192.168.2.111 -p 3001 -r $i ; done # C. -- Media Service Provider Ltd. http://blog.playlouder.com/ http://as47998.net/ 020 7729 4797 ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Productivity Pre-Tagging [was] Traffic count only for certain networks
Hi Slava, Although tagging can play in your case a key (negative) role under sustained loads, i wouldn't know if it is the prime contributor to such hang ups. The log below tells that either the router itself is unable to export all the NetFlow data or such data gets lost before making it to the collector (network, kernel buffers, etc.). Such sequence checks can be avoided with the aim of avoid massive logging and in turn relief CPU load: nfacctd_disable_checks set to true. What occurs when the Core Process has not time to handle all traffic? Well, nfacctd reads data from a socket; and a socket at the very end manages a buffer of a certain size. If nfacctd is too slow to pick data out of the buffer compared to the arrival rate, there will be some data loss. At this propo: is buffering enabled within nfacctd (ie. plugin_pipe_size, plugin_buffer_size) ? Is it not also an idea, if possible (depends on the router) and for the benefit of the whole solution, to introduce sampled NetFlow? Cheers, Paolo On Mon, Oct 26, 2009 at 07:57:17PM +0200, Slava Dubrovskiy wrote: [ ... ] Throughout our conversation about traffic accounting has noticed that periodically the daemon nfacctd hangs up. It happens when the quantity of packages strongly increases more then 50kpps (during DDoS). In log I see: Oct 25 18:59:48 stat nfacctd[3379]: WARN: expecting flow '4333297' but received '403' collector=???^^B:8818 agent=192.168.21.1:129 Oct 25 18:59:48 stat nfacctd[3379]: WARN: expecting flow '405' but received '406' collector=???^^B:8818 agent=192.168.21.1:129 Oct 25 18:59:48 stat nfacctd[3379]: WARN: expecting flow '407' but received '420' collector=^H^_B:8818 agent=192.168.21.1:129 Oct 25 18:59:48 stat nfacctd[3379]: WARN: expecting flow '421' but received '432' collector=^T^_B:8818 agent=192.168.21.1:129 Oct 25 18:59:48 stat nfacctd[3379]: WARN: expecting flow '433' but received '446' collector=^_B:8818 agent=192.168.21.1:129 Oct 25 18:59:48 stat nfacctd[3379]: WARN: expecting flow '447' but received '456' collector=,^_B:8818 agent=192.168.21.1:129 Oct 25 18:59:48 stat nfacctd[3379]: WARN: expecting flow '457' but received '463' collector=3^_B:8818 agent=192.168.21.1:129 Oct 25 18:59:48 stat nfacctd[3379]: WARN: expecting flow '464' but received '478' collector=B^_B:8818 agent=192.168.21.1:129 Oct 25 18:59:48 stat nfacctd[3379]: WARN: expecting flow '479' but received '4333400' collector=X^_B:8818 agent=192.168.21.1:129 After this nfacct stop listen port and not working. Question: What occurs, when the Core Process has not time to handle all traffic? How it is possible to increase productivity Pre-Tagging? ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] BGP-related fields do not show up in the memory table
Hi Zenon, Thanks very much for your feedback first of all; please follow my replies in-line. On Fri, Oct 23, 2009 at 02:23:34AM +0300, Zenon Mousmoulas wrote: I am exporting netflow v9 (non-aggregated, for the time being) from a Cisco router (12000/PRP with 12.0S) to nfacctd (0.12.0rc2). I have Can i ask you which 12.0S IOS version the C12k is running precisely? However, I can not see this information in the memory table, for example: Your configuration, bgp_agent_map and overall setup appears correct. I've double-checked by reproducing the scenario on a testbed before replying. I find two possible explanations on what's happening: * if you compiled the package with support for IPv6 (--enable-ipv6) - doesn't appear so, but better ask - the bgp_agent_map should be rewritten as: id=x.x.x.7 ip=:::x.x.x.10 * the BGP Router-ID is set as x.x.x.7 but effectively BGP session is established by using a different IP address, ie. you didn't impose the neighbor ... update-source interface or you did but the interface has multiple IP addresses assigned and another one is picked. Let me know on this. Also, in the table above, AS 0 should be the exporting router's own AS (5408) but it isn't, probably because the corresponding prefixes are known via the IGP. Is it possible to translate with pre_tag_map? Any other ideas? I am reluctant to use 'nfacctd_as_new: bgp' RIB lookups since we probably have this information already (exporter is setup for origin-as). I see two possible cases for the AS 0, IHMO one more likely the other slightly less: * It could be static or connected routes redistributed in BGP; in such a case you can use communities to assign a fictious ASN to people on your own IP address space (see section XIc of the EXAMPLES document, the bgp_stdcomm_pattern_to_asn entry in the CONFIG-KEYS document and pages 19-20 of the following presentation: http://www.pmacct.net/lucente_pmacct_uknof14.pdf * It could be, as you said, a prefix lying in the IGP; in such a case you have two options: - as you said, pre_tag_map. Note rc3, which will hopefully be out very soon (by end of the month), will include a tag2 field (ie. a second field dedicated to tagging) - very useful when building traffic matrices. - You might re-distribute these routes in BGP; network-wise it will cost slightly more memory (you shouldn't have that many routes in the IGP, do you? Would expect in the order of a few thousands if not less) while from a pure routing perspective, the IGP will always win due to the higher protocol preference. Having the prefixes in BGP will enable you to get back to the previous case and use the bgp_stdcomm_pattern_to_asn feature. Very open to feedback, privately or here on list, on this matter. Finally I should note that I am seeing some occasional warnings in the debug log of nfacctd about unknown templates: DEBUG ( default/core ): Discarded NetFlow V9 packet (R: unknown template 257 [195.251.27.10:259]) The exporter is supposed to be resending the template every 20 packets (the default); I did a packet capture and it looks like it is regularly doing so. Would you mind sending me privately a brief capture of the template and possibly a few NetFlow packets containing flowsets that match such template? Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] BGP-related fields do not show up in the memory table
Hi Zenon, One additional point to my previous reply. On Fri, Oct 23, 2009 at 02:23:34AM +0300, Zenon Mousmoulas wrote: I am reluctant to use 'nfacctd_as_new: bgp' RIB lookups since we probably have this information already (exporter is setup for origin-as). Very true. And it depends on your goals whether that is sufficient or not. An important piece of information, for peering purposes for example, is correlating peer-as and origin-as. Getting ASN info straight from BGP enables you to do that. Perhaps also add BGP next-hop if peering with the same people at multiple places or for a bit of traffic engineering - granted that the network is is running MPLS; but because you run NetFlow v9, you should be able to get BGP next-hop from there aswell. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Using nfacctd as netflow probe with high traffic levels
Hi Chris, Was wondering if it could be a counter-rollover issue. Did you configure pmacct to compile with the --enable-64bit knob? Apart from the above - given you are aggregating on src_host,dst_host in pmacct, to save some precious space on your Sup720 NetFlow TCAM, i would recommend to use the mls flow ip interface-destination-source profile instead. Just curiosity: did you enable that mls netflow interface on purpose or it's there by itself? Cheers, Paolo On Sun, Oct 11, 2009 at 07:46:53PM +0200, Chris Hellkvist wrote: Hi, i just set up pmacct and try to use nfacctd as netflow collector with a mysql backend for accounting purposes. The netflow v5 data is generated by a Cisco device with a sup720-3bxl card and several GigE interfaces and sent to a FreeBSD box running nfacctd. nfacctd is configured to store the data in a msysql datase on a remote server. After playing with some config options everything runs fine on the first tests with only one vlan-interface generating netflow data (a single customer with about 1000 mbits traffic on the vlan-interface). nfacctd receives the data, summarizes it according to the networks list and puts it into the mysql database every minute. The daily value for this customer looks like this when selected every minute from the database: 2051 GB, 2058 GB, 2069 GB, 2074 GB, 2079 GB, 2085 GB, 2091 GB This values are coherent for the traffic load on the interface. So i started configuring other interfaces to generate netflow data on the Cisco device (about 1800mbits traffic). After this, the system logs only about 1GB every minute for the same customer: 2106 GB, 2107 GB, 2108 GB, 2109 GB, 2110 GB, 2112 GB, 2113 GB (I'd say that there is some lack of traffic ;-) ) Now i'm trying to figure out where the data gets lost. The load on the nfacct-box and the database servers is little (not more that 0.3 at all time). I ran nfacctd in debug mode, but i did not find any messages giving me a hint. Also on the cisco box everything looks fine, no high cpu or memory load. For the records, a snippet of my nfacctd-config: daemonize: true #debug: true networks_file: /root/conf/agtv-nets.list aggregate: src_host,dst_host nfacctd_port: 9787 plugins: mysql sql_optimize_clauses: true sql_table_schema: /root/conf/dbs-dat.schema sql_refresh_time: 60 sql_history: 1d sql_history_roundoff: d sql_db: accounting sql_table: tkue_%Y_%m_%d sql_host: X sql_passwd: Y sql_user: Z Also the netflow-config on the cisco box: ip flow-cache entries 52 ip flow-cache timeout active 1 mls netflow interface mls flow ip interface-full mls nde sender version 5 ip flow-export version 5 ip flow-export destination 172.16.80.10 Maybe somebody on the list can give me a hint if there are any options to optimize or if there is any bottleneck i did not think about - any help is appreciated. Thanks, Chris ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] MySQL connection issues
Hi Jeremy, On Mon, Oct 05, 2009 at 01:14:46PM +1000, Jeremy Lee wrote: I'm now discovering that there's often a huge time lag before the data goes into the database, ranging from a few minutes to up to an hour. I've got debug going, 20 seconds between SQL refreshes, and I get several: Quoting only introduction to the issue for the sake of brevity; you got the reason why you get nothing for minutes, then suddenly it's all there: buffering. Perhaps try with incremental steps if you not have already done that - instead of jumping from 1024 to 10240. Get the trade-off which better suits your scenario. Overall, what peak Mbps is this installation about? Any pps figure? What i'm trying to figure is why by using buffers of 1K you loose data. Any chance there is a concurrent process leaking full CPU cycles for a substantial amount of time which doesn't allow the daemon to cope with peaks of traffic? On the persistency of the database connection; i'm open to discussion and comments on this. I also see it would apply just fine to you. But let me say some forewords: * pmacct comes from a persistent connection implementation (many years ago); this was dropped because too fragile when adopted as a general purpose solution. Hence migrating to a more stateless approach. This was for a mix of reasons, mainly: a) some conditions hard to detect: server was shut down not properly, firewall, NAT or load-balancers in the middle timing out the session or restarting, etc. b) communications with the database server always passing through 3rd party APIs; this easily translates in not having full control on things. * Adding a clean option in this sense might require quite some work to make it generally applicable, ie. not speaking about a quick fix but something which has to be ported (and tested working fine) across the multiple database software supported by pmacct. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Traffic count only for certain networks
Hi Slava, On Tue, Sep 22, 2009 at 11:42:37PM +0300, Slava Dubrovskiy wrote: I wish to count only traffic from/to local IP. I.e. that in base were only my local IP. For this purpose I specify my local IP in networks_file and I aggregate by src_host, dst_host. I don't wish to store not local IP since then the base will be very big. Thus there are some networks (ua-list.txt) the traffic with which I need to count separately. So I make: aggregate[hourly]: src_host, dst_host aggregate[hourly_ua]: src_host, dst_host aggregate_filter[hourly_ua]: net 173.194.0.0/24 188.163.0.0/24 193.0.227.0/24 plugins: mysql[hourly], mysql[hourly_ua] sql_table[hourly]: acct_base sql_table[hourly_ua]: acct_base_ua networks_file: /etc/pmacct/networks.list But how can I separate traffic from/to UA-IX? I can use aggregate_filter for this. And it work fine. But networks list is very big :( And I can't add full list to aggregate_filter. OK: accounting for local IP addresses traffic and UA-IX networks is two different things - no traffic matrix required. Hence, i would propose you to change your config as follows: aggregate[hourly]: src_host, dst_host aggregate[hourly_ua]: src_host, dst_host plugins: mysql[hourly], mysql[hourly_ua] sql_table[hourly]: acct_base sql_table[hourly_ua]: acct_base_ua networks_file[hourly]: /etc/pmacct/networks-local.list networks_file[hourly_ua]: /etc/pmacct/networks-ua-ix.list The effect will be that the hourly_ua plugin will give you a breakdown of the traffic at the UA-IX, plus you will find a catch all entry every hour, ie. ip_src = 0, ip_dst = 0, which is the hourly sum of all the traffic between your local IP addresses and networks besides UA-IX. I did explain advantages of usin a networks_file rather than an aggregate_filter in this scenario in a previous email (reloadable at runtime and more efficient on long list of IP). Makes sense? Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] reloading config accuracy
Hi Tony, On Sun, Sep 20, 2009 at 06:03:18PM -0700, Tony wrote: I haven't upgraded yet, I will be doing that now, but I wanted to give you some feedback on what I'm seeing in the old version and we can see if it persists to the new version. [ ... ] 10306644 10306462182 0.00177% 83631880 83631698182 0.00022% 1016473 1016265 208 0.02047% 3318523 3318341 182 0.00548% 48220490 48220308182 0.00038% I just thought I'd alert you to the above and see if the changes you have committed make a difference. Hopefully I will be able to report back again in a few days. To confirm you that that behaviour was precisely target of a bugfix which was issued a few weeks ago to the CVS and now is already part of 0.12.0rc2 release. Let me know how it goes after the upgrade. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Format of storage IP addresses in database.
Hi Slava, On Wed, Sep 09, 2009 at 09:38:03PM +0300, Slava Dubrovskiy wrote: Why for store IP it is used CHAR type? It's better use INT UNSIGNED and use INET_ATON() and INET_NTOA() for convert. http://dev.mysql.com/doc/refman/5.0/en/miscellaneous-functions.html#function_inet-aton This is correct observation. The topic did already get some (although minor) attention in the past and as a result some patches were produced (for releases 0.7.9 and 0.10.x; i can find them again and point them to you); they never managed to get merged into mainstream due to requiring major coding for a clean job and in contrast lack of general interest. My impression is that using INTs instead of CHARs has overall more consistent cons than pros. Pros: it saves space and allows for more efficient range lookups avoiding LIKEs (but, in turn, if one needs that, the *_net primitives are readily available in pmacct). Cons: it doesn't look like there is any support for IPv6, data conversion does not come for free: has performance impact on each INSERT and UPDATE query which need instead to be processed as fast as possible and some SELECTs (ie. SELECT INET_NTOA(ip_src) ...). Two extra considerations: a) if one has a particular eye for this, probably the best choice is to consider using PostgreSQL rather than MySQL as it features the 'inet' (IPv4, IPv6) and 'macaddr' data types: IHMO, they did a true clean job with it; b) the real performance loss heppans when using variable CHAR fields - but this is never the case with pmacct; static length CHAR fields might be slightly less efficient in terms of size than INTs but, you know, overall it has never got that bad to justify going a different way. Indeed i'm open to discussion. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Traffic count only for certain networks
Hi Slava, On Thu, Sep 10, 2009 at 02:24:22PM +0300, Slava Dubrovskiy wrote: [ ... ] It is necessary to count the traffic with filtering on certain networks. For this purpose I need write ALL this networks to aggregate_filter. But this list of networks is big and it periodically varies. I cannot specify in aggregate_filter a file with the list of networks. How it can be made? It looks like to be a task for networks_file - which is also reloadable at runtime. I see you are already using it in your config, is it not producing the expected results (ie. local networks accounted for, all the rest zeroed)? For long, long lists of IPv4/IPv6 prefixes this might get also preferrable from a pure performance point of view given that aggregate_filter employs a BPF filter which would check elements one-by-one, in the order you inserted them, until a match is found whereas the networks_file employs a merge-sort algorithm. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] not build (was pmacct 0.12.0rc2 released !)
Hi Slava, Don't know precisely where the problem lies but i've tried the same on a Linux and a Solaris box and it works no problems: [pa...@linone ~]$ tar xfz pmacct-0.12.0rc2.tar.gz [pa...@linone ~]$ cd pmacct-0.12.0rc2 [pa...@linone pmacct-0.12.0rc2]$ ./configure --build=i586-alt-linux --host=i586-alt-linux --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib --libexecdir=/usr/lib --localstatedir=/var/lib --sharedstatedir=/usr/com --mandir=/usr/share/man --infodir=/usr/share/info --disable-dependency-tracking --without-included-gettext --with-pgsql-includes=/usr/include/pgsql --enable-64bit --enable-threads --enable-mysql --disable-pgsql --disable-sqlite3 [ ... ] updating cache ./config.cache creating ./config.status creating Makefile creating src/Makefile creating src/nfprobe_plugin/Makefile creating src/sfprobe_plugin/Makefile creating src/bgp/Makefile [pao...@linone pmacct-0.12.0rc2]$ make [ ... ] And then compiles fine. Anybody else reading able to report issue while configuring for compilation? Or can you give it another try? Cheers, Paolo On Wed, Sep 09, 2009 at 10:41:39PM +0300, Slava Dubrovskiy wrote: 09.09.2009 20:00, Paolo Lucente ??: VERSION. 0.12.0rc2 it not build. [ ... ] rc1 is building without problems. Do you have any advices? -- WBR, Dubrovskiy Vyacheslav ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] reloading config accuracy
Hi Tony, On Sat, Sep 05, 2009 at 09:01:01PM -0700, Tony wrote: I have tested the above suggested configuration and it is working. There is data going into the SQL table now! I am going to let it run in parallel with the unadjusted data (which is going into another table) and then compare the two of them and also compare to the stats being reported by the packeteer. I've just managed to commit to the CVS repository some code to remove a dependency between actions and checks in the sql_preprocess layer (so that you can roll-back to your original config, which did make sense). I also went through an overall review of the feature - which resulted in a couple of fixes (one right to the 'adjb' section) and some cleanups. Hence I would highly invite you to make your assessment against the version currently in the CVS or alternatively wait until the rc2 release is out, later in the week. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] nfacctd_renormalize and sql_preprocess: usrf=10 not working
Hi Slava, On Fri, Sep 04, 2009 at 10:10:46PM +0300, Slava Dubrovskiy wrote: I trying use nfacctd_renormalize option and use sql_preprocess: usrf=10 [ ... ] And after as I add sql_preprocess: usrf=10 it stop write to database. If I comment sql_preprocess: usrf=10 - working, but not renormalize traffic. What is wrong? Is it necessary set sql_preprocess: usrf=10 or nfacctd_renormalize: true enough? Part of this topic (what is not making the usrf=10 working) was touched in the email i've just sent out, please give it a look. In short: usrf is defined as an action: you need to prepend to it a check, even a dummy one which can be passed by every entry on the queue (ie. minp=1). Give it a try. With regards to renormalization and which way to go, it depends on which NetFlow version you are using. pmacct supports natively only renormalization of NetFlow v5 flows (it means you can set the 'nfacctd_renormalize: true' and that's it). Whereas NetFlow v9 sampling is not natively supported (yet): hence you are forced to go the sql_preprocess (= static) way by explicitely defining the sampling rate by which flows have to be renormalized (and remember to change this parameter at the collector whenever you change it at the exporting devices). Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] timestamp rounding bug
Hi, I've just committed into the CVS some code to slightly facilitate life of who uses pmacct on systems which automagically get the clock changed as effect of DST (Daylight Saving Time). Shifts forward are handled smoothly: they are timely recognized and as a result the current time-bin (sql_history) and SQL cache scanner deadline (sql_refresh_time) are adjusted by catching up the time difference; an if (now X) then X += Y was modified in a while (now X) then X += Y. Shifts backward were not a reason of prime concern (essentially as they get solved transparently); for the way DST works, it means having a day of 25 hours, with one hour repeated twice: whichever way these backward shifts are handled, they would still generate spourious results and hence i felt to not act on them - but as usual comments are welcome. As of now, what would happen is a 02:00AM entry accounting for the real time bin (say, 5mins, 1hr basing on the sql_history config) plus the full extra hour. The tricky alternative would be to bring the clock back and having the time-bins between 01:00AM to 02:00AM polluted with double data. Caveat for this approach being: it would not work whenever sql_dont_try_update is enabled - as it would just generate lots of duplicates. I've lab-tested the shifts forward working (for me); any additional testing from you is more than welcome. Cheers, Paolo On Mon, Aug 03, 2009 at 10:12:30AM +0300, alex wrote: Hello Paolo, Could you please tell about current state of this question. Alex On 04/19/2009 01:00:48 PM, Paolo Lucente wrote: Hi Alex, DST is not supported. Timezones are. The idea behind this was that a backend application (like pmacct is) should ideally work only with UTC (even if timezones are supported) and then front-ends should localize the time as required. I don't understand where the conversion to local time is happening regarding this problem, but what makes sense to me is to collect timestamps in UTC, store them in UTC when storing them in a database, and let whatever's pulling the data out of the db present the data to the user in whatever fashion makes sense. Any other approach, i.e. working in local time or DST, makes working across time zones difficult, and computing intervals (in the case of DST) impossible. (Because when you look at a time during the hour when DST moves the clock back you don't know which hour you are in.) As, it seems, timezones are supported, then those that want to struggle with an approach other than the above can already do so because timezones account for DST. I don't know what it means to support timezones and not support DST so don't know what DST support would mean. I would think that all the support for timezones, DST, and the like is already in the localization facilities supplied by the OS and that's where it should stay. Karl k...@meme.com --- ??? ?? ??? - http://pogoda.tut.by ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] pmacctd commands to match cisco ip flow-cache timeout
Hi Stig, That is correct. Align all other available timeouts (tcp, udp and icmp other than the general one which you already mentioned) to the inactive value. Active timeout goes definitely mapped to maxlife instead. Cheers, Paolo On Wed, Aug 19, 2009 at 07:59:45PM -0700, Stig Thormodsrud wrote: I'm using pmacctd to export netflow v5 and am experimenting with a netflow collector that suggests the following cisco flow timeout values: ip flow-cache timeout active 1 ip flow-cache timeout inactive 15 I'm trying to figure out which nfprobe_timeout vars are the equivalent. Would it be: nfprobe_timeouts: maxlife=60:general=15 thanks, stig ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] segv with memory,sfprobe plugins
Hi Stig,
thanks very much for having reported the issue. This is now solved
in the CVS. I managed to reproduce it.
It was lying in the fact that initialization of the sfprobe plugin
was explicitely disabling the IP fragment handler in pmacctd; this
was causing the IMT plugin, configured with L4 primitives (src_port
for example), to crash because it expects the IP fragment handler
to be there.
The one-liner fix basically avoids sfprobe to turn the IP fragment
handler off in case it was previously turned on (hence you see the
position of the plugins was relevant) as part of operations of a
concurrent plugin.
Cheers,
Paolo
On Mon, Aug 17, 2009 at 08:27:51PM -0700, Stig Thormodsrud wrote:
I'm getting a segv fault when using the following conf file:
s...@io:~/git/pmacct-0.11.4/src$ cat pm.conf
daemonize: false
debug: true
promisc: true
pidfile: /var/run/pmacctd-eth0.pid
imt_path: /tmp/pmacctd-eth0.pipe
aggregate: src_host,dst_host,proto,src_port,dst_port,tos,flows
interface: eth0
!syslog: daemon
pcap_filter: !ether src 00:15:17:0b:d2:16
plugins: memory,sfprobe
sfprobe_agentsubid: 5
sfprobe_receiver: 172.16.117.25:6343
s...@io:~/git/pmacct-0.11.4/src$ sudo ./pmacctd -f pm.conf
INFO ( default/memory ): 131070 bytes are available to address shared
memory segment; buffer size is 132 bytes.
INFO ( default/memory ): Trying to allocate a shared memory segment of
4325244 bytes.
INFO ( default/sfprobe ): Pipe size obtained: 131070 / 49348.
OK ( default/core ): link type is: 1
DEBUG ( default/sfprobe ): Creating sFlow agent.
INFO ( default/sfprobe ): Exporting flows to [172.16.117.25]:6343
INFO ( default/sfprobe ): Sampling at: 1/1
DEBUG ( default/memory ): allocating a new memory segment.
DEBUG ( default/memory ): allocating a new memory segment.
OK ( default/memory ): waiting for data on: '/tmp/pmacctd-eth0.pipe'
DEBUG ( default/memory ): Selecting bucket 16151.
Segmentation fault
In gdb it stops at:
(gdb) run -f pm.conf
Starting program: /home/stig/git/pmacct-0.11.4/src/pmacctd -f pm.conf
[Thread debugging using libthread_db enabled]
INFO ( default/memory ): 131070 bytes are available to address shared
memory segment; buffer size is 132 bytes.
INFO ( default/memory ): Trying to allocate a shared memory segment of
4325244 bytes.
INFO ( default/sfprobe ): Pipe size obtained: 131070 / 49348.
DEBUG ( default/memory ): allocating a new memory segment.
DEBUG ( default/sfprobe ): Creating sFlow agent.
INFO ( default/sfprobe ): Exporting flows to [172.16.117.25]:6343
INFO ( default/sfprobe ): Sampling at: 1/1
DEBUG ( default/memory ): allocating a new memory segment.
OK ( default/memory ): waiting for data on: '/tmp/collect.pipe'
OK ( default/core ): link type is: 1
[New Thread 0xb788fa90 (LWP 23213)]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb788fa90 (LWP 23213)]
0x080649f3 in src_port_handler (chptr=0x80c3ce0, pptrs=0xbf90dca8,
data=0xbf90dc6c) at pkt_handlers.c:353
(gdb)
(gdb) where
#0 0x080649f3 in src_port_handler (chptr=0x80c3ce0, pptrs=0xbf90dca8,
data=0xbf90dc6c) at pkt_handlers.c:353
#1 0x0805d218 in exec_plugins (pptrs=0xbf90dca8) at plugin_hooks.c:219
#2 0x08059b72 in pcap_cb (user=0xbf90de8c \031\, pkthdr=0xbf90dd88,
buf=0x883d1ba ) at pmacctd.c:665
#3 0xb7ebbd45 in ?? () from /usr/lib/libpcap.so.0.8
#4 0xbf90de8c in ?? ()
#5 0xbf90dd88 in ?? ()
#6 0x0883d1ba in ?? ()
#7 0x0020 in ?? ()
#8 0xbf90dd74 in ?? ()
#9 0xbf90dd98 in ?? ()
#10 0x in ?? ()
(gdb)
(gdb) p *pptrs
$1 = {pkthdr = 0xbf90dd88, f_agent = 0xb7e52219 SMP, f_header = 0x0,
f_data = 0x1 Address 0x1 out of bounds, f_tpl = 0x0, f_status = 0x1
Address 0x1 out of bounds, idtable = 0x0, bpas_table = 0x756e694c
Address 0x756e694c out of bounds, bta_table = 0xbf90e09c \220\223,
packet_ptr = 0x883d1ba , mac_ptr = 0x883d1ba , l3_proto = 2048,
l3_handler = 0x8059c77 ip_handler, l4_proto = 6, tag = 0, bpas = 0, bta
= 0, bgp_src = 0xb78900f0 \003\210\020ii\r, bgp_dst = 0x1 Address 0x1
out of bounds, bgp_peer = 0x1 Address 0x1 out of bounds, pf = 0,
new_flow = 0 '\0', tcp_flags = 0 '\0', vlan_ptr = 0x0, mpls_ptr = 0x0,
iph_ptr = 0x883d1c8 E, tlh_ptr = 0x29370 Address 0x29370 out of
bounds, payload_ptr = 0x0, class = 0, cst = {tentatives = 20 '\024',
stamp = {tv_sec = 0, tv_usec = 0}, ba = 3213942184, pa = 25312, fa = 240
''}, shadow = 0 '\0', tag_dist = 1 '\001'}
(gdb)
void src_port_handler(struct channels_list_entry *chptr, struct
packet_ptrs *pptrs, char **data)
{
struct pkt_data *pdata = (struct pkt_data *) *data;
if (pptrs-l4_proto == IPPROTO_UDP || pptrs-l4_proto == IPPROTO_TCP)
pdata-primitives.src_port = ntohs(((struct my_tlhdr *)
pptrs-tlh_ptr)-src_port);
else pdata-primitives.src_port = 0;
}
Seems like the problem is dereferencing pptrs-tlh_ptr in
src_port_handler(). If I reverse the plugins to sfprobe,memory or
remove the memory plugin, the it works. Could the
Re: [pmacct-discussion] Bug in pmacct-0.12.0rc1?
Hi Johannes, To get more verbosity you have to disable the sql_multi_values feature; that's what causing the rather compact and un-helpful output. Would that be an option for you? Also, was wondering: any chance the 0.0.0.0 bit bucket entry is collecting 4GB traffic in 5 minutes time? In such a case, the solution is to go with 64-bit counters. Let me know. Cheers, Paolo PS: very verbose outputs are very much welcome but go in private email (perhaps compressed) :-) On Thu, Aug 06, 2009 at 07:33:57PM +0200, Johannes Formann wrote: Paolo Lucente pa...@pmacct.net wrote: Hi Paolo, If it's not something in this sense, i would ask you to turn on debugging on one of the two plugins (debug[inbound]: true for example) and check what SQL queries are causing such duplication. If it's not too much sensible data, you can let me have a look aswell at the debug log by sending it privately. I'd turn up debug, ca I make ist more verbose, I'll think this is not so helping: Aug 6 17:21:53 traffic pmacctd[19363]: INFO ( default/core ): Start logging ... Aug 6 17:21:53 traffic pmacctd[19363]: INFO ( inbound/mysql ): Pipe size obtained: 110592 / 4000. Aug 6 17:21:53 traffic pmacctd[19365]: DEBUG ( /etc/pmacct/networks ): (networks table IPv4) AS: 0, net: 5426, mask: Aug 6 17:21:53 traffic pmacctd[19365]: DEBUG ( /etc/pmacct/networks ): (networks table IPv4) AS: 0, net: c32a7200, mask: ff00 Aug 6 17:21:53 traffic pmacctd[19365]: DEBUG ( /etc/pmacct/networks ): (networks table IPv4) AS: 0, net: c32a7300, mask: ff00 Aug 6 17:21:53 traffic pmacctd[19365]: DEBUG ( /etc/pmacct/networks ): IPv4 Networks Cache successfully created: 1 entries. Aug 6 17:21:53 traffic pmacctd[19365]: DEBUG ( /etc/pmacct/networks ): (networks table IPv6) AS: 0, net: 0:0:0:0, mask: 0:0:0:0 Aug 6 17:21:53 traffic pmacctd[19365]: DEBUG ( /etc/pmacct/networks ): IPv6 Networks Cache successfully created: 32771 entries. Aug 6 17:21:53 traffic pmacctd[19363]: OK ( default/core ): link type is: 1 Aug 6 17:25:06 traffic pmacctd[19372]: ( inbound/mysql ) *** Purging cache - START *** Aug 6 17:25:06 traffic pmacctd[19372]: DEBUG ( inbound/mysql ): 1933 VALUES statements sent to the MySQL server. Aug 6 17:25:06 traffic pmacctd[19372]: ( inbound/mysql ) *** Purging cache - END (QN: 1934, ET: 0) *** Aug 6 17:30:06 traffic pmacctd[19800]: ( inbound/mysql ) *** Purging cache - START *** Aug 6 17:30:06 traffic pmacctd[19800]: DEBUG ( inbound/mysql ): 1976 VALUES statements sent to the MySQL server. Aug 6 17:30:06 traffic pmacctd[19800]: ERROR ( inbound/mysql ): Duplicate entry '2009-08-06 17:25:00-0.0.0.0' for key 1#012 Aug 6 17:30:06 traffic pmacctd[19800]: ( inbound/mysql ) *** Purging cache - END (QN: 1977, ET: 0) *** Aug 6 17:30:11 traffic pmacctd[19801]: ERROR ( outbound/mysql ): Duplicate entry '2009-08-06 17:25:00-0.0.0.0' for key 1#012 greetings Johannes ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Bug in pmacct-0.12.0rc1?
Hi Johannes, The only idea coming to the mind is that two pmacctd instances (each with two plugins) are fired up in parallel and trying to write to the same MySQL tables. If it's not something in this sense, i would ask you to turn on debugging on one of the two plugins (debug[inbound]: true for example) and check what SQL queries are causing such duplication. If it's not too much sensible data, you can let me have a look aswell at the debug log by sending it privately. I assume this an upgrade of a working installation? Let me know. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] timestamp rounding bug
On Tue, Aug 04, 2009 at 09:18:22PM -0500, Karl O. Pinc wrote: On 08/04/2009 04:35:31 AM, Chris Wilson wrote: Is any real-world system set to UTC? I'm certainly not going to run my firewall (where I run pmacct currently) on UTC. All my logs would be screwed up and much harder to interpret. Setting the system clock to UTC is traditional in Unix, AFAIK. The obvious reason being that it makes it easy to compare times across systems. It's up to the logging application to decide whether to output UTC, localtime without time zones, localtime with time zones etc. Likewise other programs that produce times for human consumption. The Internet (e.g. rfc2822 and, iirc, rfc5424) tends to solve the human consumption problem by logging in localtime but including an offset from utc in the timestamp. Chiming briefly into this to add to the thread that is common to have systems set on UTC even in large networks that span across multiple timezones and having local times wouldn't help the NOC at all (given that pretty much all works on UTC, ie. maintenance announcements). Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] multiple interfaces uni-directional flows
Hi Stig, Very briefly to confirm: a) you are correct, libpcap captures both inbound and outbound traffic and b) the workaround you have put in place not only makes sense but is also by far the most efficient way to filter traffic out of pmacctd. Cheers, Paolo On Tue, Aug 04, 2009 at 10:39:00AM -0700, Stig Thormodsrud wrote: I notice with multiple interfaces that I get duplicate flows. If I recall correctly a cisco router does netflow only on input while it seems pcap captures both inbound outbound packets. My work around to filter out the output flows was to use a pcap_filter such as: ! daemonize: true promisc: false pidfile: /var/run/pmacctd-eth0.pid imt_path: /tmp/pmacctd-eth0.pipe plugins: nfprobe, memory aggregate: src_host,dst_host,src_port,dst_port,proto,tos,flows,tag interface: eth0 syslog: daemon ! filter out packets with the mac address of eth0 pcap_filter: !ether src 00:0c:29:8c:53:7c nfprobe_receiver: 172.16.117.25:2100 nfprobe_version: 5 nfprobe_engine: 1:2 post_tag: 2 Is this the approach others are using with multiple interfaces or is there a better way? Thanks, stig ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] dynamic sql tables not created
Hi Christian, I'm not sure how a change to the frontend can influence that way the backend. Two things to check: 'sql_history' is in use (as it generates the timestamp used at a later stage to work out the name of the dynamic table) and sql_table_schema file is readable. Moreover, something related to dynamic SQL tables was fixed in just released 0.12.0rc1 version - and it's been in the CVS for quite a while. If you are not using either of these versions, another chance is to upgrade and see if this solves the issue. Otherwise we may have to dig deeper into it. Let me know. Cheers, Paolo On Mon, Aug 03, 2009 at 11:51:13AM +0200, christian wrote: hello world, since i have changed my 'frontend' to slack64 on a e5520, pmacctd doesn't create weekly tables any more on my 'backend' server. it seems it doesn't even try.. when i create them manually, they get populated. a typical sunday/monday switch looks like this: __cut__ Aug 3 00:00:04 ns pmacctd[7614]: ERROR ( default/mysql ): PRIMARY 'mysql' backend trouble. Aug 3 00:00:04 ns pmacctd[7614]: ERROR ( default/mysql ): The SQL server says: Table 'pmacct.acct_ppp0_200931' doesn't exist __cut__ no other error messages. i definitely *can* create these tables when doing `mysql` on the same machine as user pmacct (just tested)... what am i possibly doing wrong, and how could i find or fix it? thanks for reading! regards, christian ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
