Re: [pmacct-discussion] monitoring multiple network interfaces

2010-09-23 Thread Paolo Lucente
Hi Ross, This same setup works well on a Linux for example. I see you daemonize both pmacctd daemons. Can you please keep them in foreground instead (daemonize: false) so that you can catch what happens? Cheers, Paolo On Thu, Sep 23, 2010 at 01:06:58PM -0700, Ross Lawrie wrote: Hi, I'm

Re: [pmacct-discussion] Source port column name depends on database

2010-09-14 Thread Paolo Lucente
Hi Chris, Agree. I seem to reckon this legacy issue is limited to the TCP/UDP ports only and i'm thinking perhaps the best way to approach it is to issue a true/false config switch, ie. sql_table_compat, for the purpose. But for consistency with the rest, these fields should be aligned to

Re: [pmacct-discussion] duplicates...

2010-09-12 Thread Paolo Lucente
-Oprindelig meddelelse- Fra: pmacct-discussion-boun...@pmacct.net [mailto:pmacct-discussion-boun...@pmacct.net] P? vegne af Paolo Lucente Sendt: 12. september 2010 01:32 Til: pmacct-discussion@pmacct.net Emne: Re: [pmacct-discussion] duplicates... Hi Thomas, This very likely

Re: [pmacct-discussion] duplicates...

2010-09-11 Thread Paolo Lucente
Hi Thomas, This very likely answers why you are not seeing all you expect in the database. Reason you get duplicates is pmacct is trying to insert into the database more than once per 600 seconds (as per your configuration) and sql_dont_try_update is set to true. So quickest work-around is to

Re: [pmacct-discussion] [nfacctd] aggregate_filter

2010-08-24 Thread Paolo Lucente
Hi Borys, I can confirm you filtering on TCP flags is currently not possible in both nfacctd and sfacctd. Such implementation is not major work and can be done pretty quickly - i can drop you an email privately when the code is available in the CVS so you can test it working? Apart from the

Re: [pmacct-discussion] cockpit error instrumenting www.pmacct.net/docs/cacti.html?

2010-08-13 Thread Paolo Lucente
Hi Larry, Wanted just to say briefly the best person to support you is the author of that document (which btw is not on this list and not really reachable recently). I can't help you with that as i'm not too much into Cacti and pmacct seems to return expected results. I'm certainly available to

Re: [pmacct-discussion] pmpgplay - what does it do?

2010-07-26 Thread Paolo Lucente
Hi Chris, On Mon, Jul 26, 2010 at 10:03:15AM +1000, Chris wrote: So I'd be better off making these 2 match? sql_refresh_time: 60 sql_history: 5m Correct. For what it's worth the log seems to be aggregated on the history time. I only see the 5 min increments when I run pmpgplay in debug

Re: [pmacct-discussion] pmpgplay - what does it do?

2010-07-23 Thread Paolo Lucente
Hi Chris, The 'failsafe' mechanism (ie. writing to a backup database or to log files) kicks in upon receipt of an error code from the RDBMS API. So what you see in the log file should never be already in the database. Your specific configuration is tricky because you write to the RDBMS every 60

Re: [pmacct-discussion] [nfacctd] IPv4/IPv6 Accounting into MySQL

2010-07-23 Thread Paolo Lucente
Hi Carsten, A couple of things come to mind: * Was pmacct compiled with --enable-ipv6 ? * Are 'aggregate_filter' directives including the IPv6 subnets you want to account for? * If, in parallel to nfacctd, you fire up a Wireshark (or tshark); by decoding the NetFlow v9 datagrams generated

Re: [pmacct-discussion] Netflow and 'ip' key in pretag.map

2010-07-22 Thread Paolo Lucente
Hi Damian, Are you running BGP? Would it be feasible for you to past BGP feed(s) into pmacct (granted you run a recent 0.12 release or can upgrade to that)? Idea is you can attach BGP standard communities to IP prefixes as they are advertised or re-distributed into your network. Because

Re: [pmacct-discussion] How many ips can pmacctd monitorize

2010-07-22 Thread Paolo Lucente
Hi, Try performing a locked access to the memory table. This can be done by appending a '-l' option to the command, ie. pmacct -s -l -p Let me know. Cheers, Paolo On Thu, Jul 22, 2010 at 10:35:18AM +, Jose Joaquin Anton Herrerias wrote: I was Reading CONFIG-KEYS and internals

Re: [pmacct-discussion] Error for building pmacct 0.12 from source

2010-06-22 Thread Paolo Lucente
Hi Rodolfo, In addition to what Brent correctly said you might want to double check whether you have development kit for MySQL installed at all (libraries and headers). A chance can be you have got installed only, say, MySQL client, server and tools. Cheers, Paolo On Tue, Jun 22, 2010 at

Re: [pmacct-discussion] Help as the sum of certain ips

2010-06-18 Thread Paolo Lucente
Hi Rafael, Advice is to check out Q8 from FAQS, EXAMPLES document chapter III, 'sql/README.pgsql' for initial SQL table setup. It should be enough to work out initial configs. Cheers, Paolo On Thu, Jun 17, 2010 at 05:14:12PM -0300, Rafael Stein wrote: Hello to all, I'm starting to use the

Re: [pmacct-discussion] data only in one direction

2010-06-18 Thread Paolo Lucente
Hi Morgan, INTERNALS document is part of the pmacct distribution tarball that you download (ie. pmacct-0.12.2.tar.g). Look into docs/. Cheers, Paolo On Fri, Jun 18, 2010 at 08:35:53AM +0200, Morgan Sellier wrote: Hi, Thanks for your help ! I will test it today but in the documentation I

Re: [pmacct-discussion] inet Error

2010-06-11 Thread Paolo Lucente
Hi Jonathan, It's a bit hard to say with this information. Can you post your config? Are you using any of the default SQL schemas? Which options did you compile the package? Cheers, Paolo On Fri, Jun 11, 2010 at 01:11:23PM +1000, Jonathan Gleeson wrote: Hi Guys, Anyone got any idea on this

[pmacct-discussion] pmacct 0.12.2 released !

2010-05-27 Thread Paolo Lucente
VERSION. 0.12.2 DESCRIPTION. pmacct is a small set of passive network monitoring tools to account, classify, aggregate and export IPv4 and IPv6 traffic; a pluggable architecture allows to store collected network data into memory tables or SQL (MySQL, SQLite, PostgreSQL) databases and export them

Re: [pmacct-discussion] PostgreSQL large database

2010-05-13 Thread Paolo Lucente
Hi Sergio, It looks those processes are locked out of the table they want to write to. In MySQL you can check this kind of stuff with a SHOW PROCESSLIST; the PostgreSQL equivalent should be SELECT * FROM pg_stat_activity. Its output might very well shed some light. Just btw, the number of pmacct

Re: [pmacct-discussion] MySQL index performance

2010-05-13 Thread Paolo Lucente
Hi Chris, I think it would make no semantic difference, but would increase MySQL performance with these table types, if the primary key listed stamp_inserted first instead of last. The change you propose, as you say, would not be impacting - but would you have any testing handy which

Re: [pmacct-discussion] Expiring Orphan fragment

2010-05-04 Thread Paolo Lucente
Hi, To wrap-up this thread - unless new details emerge: On Tue, May 04, 2010 at 11:53:59AM +, Paolo Lucente wrote: * one capturing some torrent traffic, so that i can replay it in a testbed and see if i can reproduce and validate the behaviour. Orphan fragments are really

Re: [pmacct-discussion] Expiring Orphan fragment

2010-05-03 Thread Paolo Lucente
Hi Sergio, Good point. I've just marked PGRES_TUPLES_OK as valid return code within the PG_create_dyn_table() function - and committed the change to the CVS. Can you please confirm it works OK for you? Cheers, Paolo On Mon, May 03, 2010 at 10:10:18AM -0300, Sergio Charpinel Jr. wrote:

Re: [pmacct-discussion] Expiring Orphan fragment

2010-04-30 Thread Paolo Lucente
Hi Sergio, On Thu, Apr 29, 2010 at 11:23:35AM -0300, Sergio Charpinel Jr. wrote: Does anyone know what this mean? I getting a lot of them in pmacctd.log Expiring orphan fragment: ip_src=210.197.202.84 ip_dst=200.137.66.1 proto=17 id=8885 It means some IP fragments have been staying too long

Re: [pmacct-discussion] Low performance query

2010-04-28 Thread Paolo Lucente
. For example, selecting * WHERE ip_src = 'x.x.x.x' How can I do a more compact aggregation? You mean by getting samples ? Actually, I did not understand very well sql_history and roundoff concepts, but increasing history would do the trick? Thanks for answering. Cheers. 2010/4/27 Paolo

Re: [pmacct-discussion] sflow in_iface out_iface information wrong

2010-04-27 Thread Paolo Lucente
Hi Bernd, If i get it correctly, you should be referring to the big 1073741823 and 2147483648 values in your SQL table. According to the sFlow (v5) specifications, these values are perfectly valid: 1073741823 = 0x3FFF: [ ... ] this is used in describing traffic which is not bridged, routed,

Re: [pmacct-discussion] Low performance query

2010-04-27 Thread Paolo Lucente
Hi Sergio, I don't know FloX very well - hence would be good information to know which specific SQL queries are performing bad. Maybe there is room to improve indexing. Is it also your goal to store every micro-flow into the SQL database? Any chance a more compact aggregation method would fit

Re: [pmacct-discussion] Duplicate entry problem.

2010-04-16 Thread Paolo Lucente
Hi Sergey, Duplicates are a clear consequence of the urgent DB writer in conjunction with disabling UPDATE queries (sql_dont_try_update) and the configured (default, i guess) primary key. Urgent writers are elicited by shortage of entries available in the SQL cache. Having memory available, my

[pmacct-discussion] pmacct 0.12.1 released !

2010-04-07 Thread Paolo Lucente
VERSION. 0.12.1 DESCRIPTION. pmacct is a small set of passive network monitoring tools to measure, account, classify, aggregate and export IPv4 and IPv6 traffic; a pluggable and flexible architecture allows to store collected network data into memory tables or SQL (MySQL, SQLite, PostgreSQL)

Re: [pmacct-discussion] ifindex and netmask records

2010-04-02 Thread Paolo Lucente
Hi Richard, On Fri, Apr 02, 2010 at 03:12:23AM -0500, Richard A Steenbergen wrote: * Record (and aggregate on) the address of the router that exported a flow via netflow/sflow. Basically I just want to know which router exported the flow to me, using either the agent address if available

Re: [pmacct-discussion] uacctd documentation

2010-03-26 Thread Paolo Lucente
Hi Ross, On Thu, Mar 25, 2010 at 09:57:05AM -0400, Ross Vandegrift wrote: Thanks for the examples! I'm having trouble with the iptables piece of the puzzle though. I suspect this is because I'm mirroring traffic to this server and the L2 destination doesn't match any address present on the

Re: [pmacct-discussion] uacctd documentation

2010-03-26 Thread Paolo Lucente
Hi Ross, On Fri, Mar 26, 2010 at 02:46:51PM -0400, Ross Vandegrift wrote: I'm curious if I can acheive better performance for generating sflow data. My plan was to use the iptables statistics module to do the sampling, to ensure that only sampled packets were being sent to userspace. Since

Re: [pmacct-discussion] aggregate-filter min packet count and bgp sql table questions

2010-03-12 Thread Paolo Lucente
Hi Brent, Good to see progress. The entries stay forever, there is not an aging-out mechanism. Reason being you are supposed to do it yourself, at regular intervals, to build a time reference for the counters. For example a simplistic scenario is a cronjob entry, set up every 5 minutes, that

Re: [pmacct-discussion] aggregate-filter min packet count and bgp sql table questions

2010-03-11 Thread Paolo Lucente
On Mar 9, 2010, at 8:28 AM, Brent Van Dussen wrote: Thanks for getting this set up Paolo! We'll get the latest CVS version loaded and tested this week to provide feedback. Cheers, -Brent On Mar 7, 2010, at 1:34 AM, Paolo Lucente wrote: Hi Brent, All, On Sat, Feb 20, 2010 at 01:05:20AM

Re: [pmacct-discussion] aggregate-filter min packet count and bgp sql table questions

2010-03-11 Thread Paolo Lucente
Hi Brent, On Thu, Mar 11, 2010 at 10:01:26AM -0800, Brent Van Dussen wrote: I wanted to ask about these messages we're getting in the logs now that we're using mem tables: WARN ( prefixes/memory ): Unable to allocate more memory pools, clear stats manually! WARN ( as_path/memory ):

Re: [pmacct-discussion] Log output from pmacctd Ive never seen before

2010-03-03 Thread Paolo Lucente
Hi Jeff, How often you get this message? Every time you see it, you miss a packet. The way to read that line is: libpcap passed only the first 37 bytes of the datagram to pmacct; L4 appears to start at the 38th octet; and you might have specified src_port or dst_port in your aggregation method.

Re: [pmacct-discussion] Pmacct data inconsistencies between tables.

2010-02-19 Thread Paolo Lucente
760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES Aptivate is a not-for-profit company registered in England and Wales with company number 04980791. Paolo Lucente wrote: Hi Daniel, Getting through the data and compare traffic figures is, IHMO, the more

Re: [pmacct-discussion] aggregate-filter min packet count and bgp sql table questions

2010-02-19 Thread Paolo Lucente
Hi Brent, On Fri, Feb 19, 2010 at 10:51:21AM -0800, Brent Van Dussen wrote: I was curious if there was a way to have sfacctd only insert into the database if a certain number of packets and/or bytes threshold is reached. It seems you are looking for the sql_preprocess directive - and

Re: [pmacct-discussion] unable to compile latest pmacct with pfring enabled libpcap ...

2010-02-18 Thread Paolo Lucente
Hi Sebastien, It's not clear to me if this was working for you before (some earlier pmacct release) and it doesn't instead with the latest. I've just tried myself to compile pmacct 0.12.0 against a libpcap 1.0.0 (vanilla) and it works fine. I'm also fairly sure this worked up to 0.12.0rc4 for

Re: [pmacct-discussion] unable to compile latest pmacct with pfring enabled libpcap ...

2010-02-18 Thread Paolo Lucente
it using libpcap from pf_ring svn ? svn co https://svn.ntop.org/svn/ntop/trunk/PF_RING/ 2010/2/18 Paolo Lucente pa...@pmacct.net Hi Sebastien, It's not clear to me if this was working for you before (some earlier pmacct release) and it doesn't instead with the latest. I've just

Re: [pmacct-discussion] Pmacct data inconsistencies between tables.

2010-02-16 Thread Paolo Lucente
looking for? -- Daniel Levy Aptivate | http://www.aptivate.org/ | +44 (0)1223 760887 The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES Aptivate is a not-for-profit company registered in England and Wales with company number 04980791. Paolo Lucente wrote: Hi Daniel

[pmacct-discussion] pmacct 0.12.0 released !

2010-02-16 Thread Paolo Lucente
VERSION. 0.12.0 DESCRIPTION. pmacct is a small set of passive network monitoring tools to measure, account, classify, aggregate and export IPv4 and IPv6 traffic; a pluggable and flexible architecture allows to store collected network data into memory tables or SQL (MySQL, SQLite, PostgreSQL)

Re: [pmacct-discussion] Pmacct data inconsistencies between tables.

2010-02-12 Thread Paolo Lucente
Hi Daniel, Unfortunately the configuration doesn't make evident where the issue can be. The 'sql_dont_try_update' very well protects against duplicate tuples - so i'm rather inclined to exclude that reason. Which version are you using? How you did discover the issue - ie. did you upgrade

Re: [pmacct-discussion] pre_tag_map issue + questions

2010-02-09 Thread Paolo Lucente
Hi Zenon, On Mon, Feb 08, 2010 at 02:43:49PM +0200, Zenon Mousmoulas wrote: records. If it's in there, then i'd like to give it a look myself: i would ask you to produce a trace and send it to me privately so that i can have a look. We can then summarize findings here. OK. I will send you

Re: [pmacct-discussion] pmacct count only 5% of SYN packets.

2010-02-09 Thread Paolo Lucente
Hi Yuriy, You have also other means to get a count of the TCP/SYN packets out of pmacct. I would suggest one for troubleshooing purposes with the goal to check where the issue lies: * keep the 'tcpflags' primitive out of the 'aggregate' directive * add a 'pcap_filter' directive to the config; it

Re: [pmacct-discussion] Missing information in mysql table

2010-02-09 Thread Paolo Lucente
Hi Jonas, On Tue, Feb 09, 2010 at 04:55:52PM +0100, Jonas Nylund wrote: [ ... ] mysql select * from acct_v5_06 limit 10;

Re: [pmacct-discussion] The accounting of flows requires SQL table v4?

2010-02-08 Thread Paolo Lucente
Hi Zenon, Good to see you around again. Please add to your config: sql_table_version[sqltest]: 4 Reason being most of the primitives are still connected to the SQL table versioning concept (in essence: pmacct expects you to make explicit which SQL schema you are running). This is in the

Re: [pmacct-discussion] pre_tag_map issue + questions

2010-02-08 Thread Paolo Lucente
Hi Zenon, On Mon, Feb 08, 2010 at 10:37:54AM +0200, Zenon Mousmoulas wrote: I have a netflow v9 feed to nfacctd from a juniper router (JUNOS 9.6R2.11), using a service pic. According to a packet capture, records include ingress and egress interface and they seem to be properly defined

Re: [pmacct-discussion] The accounting of flows requires SQL table v4?

2010-02-08 Thread Paolo Lucente
(and tried it one more time just before writing this). Unfortunately it made no difference... Z. On 08 ?? 2010, at 11:51 , Paolo Lucente wrote: Hi Zenon, Good to see you around again. Please add to your config: sql_table_version[sqltest]: 4 Reason being most of the primitives

Re: [pmacct-discussion] Not working pmacct -N [matching data['; ' ... ]]

2010-02-04 Thread Paolo Lucente
Hi Slava, Can you confirm which version you are running into this issue? Can you also post your config - just in case? Labbing this scenario up, i see it working fine for me. One thing i can suggest is: append a -l to your query to ensure a locked access to the memory table; if it's a

Re: [pmacct-discussion] Enterasys nfacctd expecting flow error

2010-01-15 Thread Paolo Lucente
is seeing very little use at all times. - Original Message From: Paolo Lucente pa...@pmacct.net To: pmacct-discussion@pmacct.net Sent: Wed, January 13, 2010 3:33:36 PM Subject: Re: [pmacct-discussion] Enterasys nfacctd expecting flow error Hi Marc, Such messages tell it has been

Re: [pmacct-discussion] pmacct can't create mysql table, but I can with the same schema

2009-12-22 Thread Paolo Lucente
Hi Jeff, If i'm getting this correctly, you see everything working except for the table creation error you get back. If this is the case, i wouldn't say you are doing something wrong; pmacct tries to create the table every time the SQL cache scanner kicks in (sql_refresh_time interval); table

[pmacct-discussion] pmacct 0.12.0rc4 released !

2009-12-21 Thread Paolo Lucente
VERSION. 0.12.0rc4 DESCRIPTION. pmacct is a small set of passive network monitoring tools to measure, account, classify, aggregate and export IPv4 and IPv6 traffic; a pluggable and flexible architecture allows to store collected network data into memory tables or SQL (MySQL, SQLite, PostgreSQL)

Re: [pmacct-discussion] sflow renormalize not working correctly (sql_preprocess does)

2009-12-21 Thread Paolo Lucente
Hi Igor, I'm not aware of any issues with the 'sfacctd_renormalize' feature; perfect you already did some debug with sflowtool - that would have been my first suggestion. I would ask you, if possible, to send me privately a brief capture of some sFlow datagrams (pcap format, full packets) so

Re: [pmacct-discussion] Aggregate traffic count with BGP community match

2009-12-11 Thread Paolo Lucente
Hi Andrew, On Fri, Dec 11, 2009 at 01:32:34PM +1300, Andrew Thrift wrote: I have a requirement to count clients traffic passing through our border router by classes, e.g. local (iBGP), national (BGP marked with specific community) and international (anything not covered by

Re: [pmacct-discussion] Layer 7 classification problem.

2009-12-08 Thread Paolo Lucente
, I don't enter the discussion of the precision of the 3rd party regex classifiers. Action i've taken is to add a note in the documentation. Cheers, Paolo On Wed, Dec 02, 2009 at 10:53:04AM +, Paolo Lucente wrote: Hi Buddhike, On Wed, Dec 02, 2009 at 02:36:20PM +0530, Buddhike wrote

Re: [pmacct-discussion] Not save data in DB when exit

2009-12-02 Thread Paolo Lucente
Hi Slava, On Tue, Dec 01, 2009 at 10:01:28PM +0200, Slava Dubrovskiy wrote: Yes, I remove sql_multi_values and it working. But as I understand this this give more performance. Was suspecting so correct. Is it possible use it? And it working good. Problem only when I stop it. It has to be

Re: [pmacct-discussion] Layer 7 classification problem.

2009-12-02 Thread Paolo Lucente
Hi Buddhike, On Wed, Dec 02, 2009 at 02:36:20PM +0530, Buddhike wrote: http://www.mail-archive.com/pmacct-discussion@pmacct.net/msg01397.html Any chance you can give it a try and post some feedback, ie. whether it yeah I've tried that with pmacct-0.12.0rc3 and still the result is same.

Re: [pmacct-discussion] Not save data in DB when exit

2009-11-30 Thread Paolo Lucente
Hi Slava, On Mon, Nov 30, 2009 at 03:03:41PM +0200, Slava Dubrovskiy wrote: I see such errors: [ ... ] DEBUG ( t2/mysql ): 3 VALUES statements sent to the MySQL server. ERROR ( t2/mysql ): Duplicate entry '0-5-2009-11-28 02:00:00' for key 1 [ ... ] DEBUG ( t1/mysql ): 400 VALUES

Re: [pmacct-discussion] peer-AS calculation problem

2009-11-25 Thread Paolo Lucente
Hi Zenon, On Wed, Nov 25, 2009 at 12:59:04PM +0200, Zenon Mousmoulas wrote: I am not sure if this affects nfacctd or, perhaps, if it overrides this information by looking up the next-hop (and perhaps also the dst peer AS) in the BGP RIB? If i'm not mistaken you are not using the

[pmacct-discussion] pmacct-contribs 20091125 released

2009-11-25 Thread Paolo Lucente
VERSION. 20091125 DESCRIPTION. pmacct is a set of network tools to gather, filter and tag IP traffic; it is able to store collected data either into a DB or a memory table. We see any monitoring, billing or accounting environment as a stack where data are picked from the network, get processed

Re: [pmacct-discussion] Not save data in DB when exit

2009-11-25 Thread Paolo Lucente
Hi Slava, On Wed, Nov 25, 2009 at 09:04:24PM +0200, Slava Dubrovskiy wrote: Seems when I make kill INT PID_OF_CORE_PROCESS it down, but plugins do not write to database. I see delay before off for plugins, but not see that they change command line to DB writer. And not see data for period.

Re: [pmacct-discussion] A strange thing with one traffic direction

2009-11-24 Thread Paolo Lucente
Hi Peter, On Mon, Nov 23, 2009 at 10:45:38PM +0100, Peter Franzel wrote: I thing RX Traffic is brilliant, but why is there such a big difference between TX bytes?! Is there something I am going wrong or where is the fault? I would essentially suggest to go in a couple of directions; first

Re: [pmacct-discussion] Layer 7 classification problem.

2009-11-23 Thread Paolo Lucente
Hi Mike, On Mon, Nov 23, 2009 at 02:00:04PM +0300, Mike Lykov wrote: By the way, L7-filter have two types of filter: The first speed shown for a pattern in the tables below is the speed when used in the kernel (with the old V8 regular expression library). The second is the speed when used

Re: [pmacct-discussion] MySQL options

2009-11-23 Thread Paolo Lucente
Hi Joel, On Sat, Nov 21, 2009 at 12:43:19PM +1100, Joel Roberts wrote: I need to setup traffic accounting (in and out) for each IP address, and then export that data to an EXTERNAL mysql database on a separate machine accessible via IP address. How do I go about setting up pmacct to do this?

Re: [pmacct-discussion] Layer 7 classification problem.

2009-11-20 Thread Paolo Lucente
Hi, On Fri, Nov 20, 2009 at 05:06:25PM +0530, Buddhike wrote: I'm testing pmacct on my network, and pmacct runs on a box inbetween my LAN switch and My ADSL router, and i'm using layer 7 classifires for classify traffic. But when testing I observed that the traffic is not correctly displayed

Re: [pmacct-discussion] invalid network mask?

2009-11-19 Thread Paolo Lucente
Hi Charlie, It appears you didn't put the table in the correct format first. Is it the case? If yes, to make that table pmacct-friendly you have to pre-process it as follows: cat bgptable | sed 's/\([0-9a-f:][0-9a-f\.\/:]*\).* \([0-9][0-9]*\)[ 0-9,{}]*$/\2,\1/' | uniq networks.lst At least it

Re: [pmacct-discussion] Problem with aggregate_filter

2009-11-19 Thread Paolo Lucente
Hi, On Thu, Nov 19, 2009 at 04:00:33PM +0530, Buddhike wrote: I've searched on google but didn't find any examples with mac adresses but with src/dst networks. But i didn't seen any records saying that aggrigate_filter can't be used with src mac and dst mac option. anyway I tested it with

Re: [pmacct-discussion] create my own mysql table

2009-11-17 Thread Paolo Lucente
Hi, On Mon, Nov 16, 2009 at 04:45:57PM -0600, fedora fedora wrote: DEBUG ( default/mysql ): INSERT INTO `test_1` (stamp_updated, stamp_inserted, ip_src, ip_dst, as_src, as_dst, src_port, dst_port, tcp_flags, ip_proto, packets, bytes, flows) VALUES (FROM_UNIXTIME(1258410661),

Re: [pmacct-discussion] protocol classification don't detect http

2009-11-17 Thread Paolo Lucente
Hi Mike, On Tue, Nov 17, 2009 at 02:27:06PM +0300, Mike Lykov wrote: I would suggest a couple of checks: * see if HTTP traffic is reaped by some other classifier, but i guess you might have already checked that. if class_id = unknown, i think it's not this case. Yes, correct. But

Re: [pmacct-discussion] protocol classification don't detect http

2009-11-16 Thread Paolo Lucente
Hi Mike, I see all of those signatures actually working by picking some sites randomly with wget. This is with 0.12.0rc3 but honestly speaking there has not been any major work related to the classification part for the past 3-4 years. I would suggest a couple of checks: * see if HTTP traffic is

Re: [pmacct-discussion] create my own mysql table

2009-11-16 Thread Paolo Lucente
Hi, On Mon, Nov 16, 2009 at 11:58:14AM -0600, fedora fedora wrote: I still see all flow records having the same number 4294967295 in my mysql table, and debug does not seem to tell me why this happens. How do you mean? You see that number appearing in the debug? As writing to the database is

Re: [pmacct-discussion] create my own mysql table

2009-11-13 Thread Paolo Lucente
Hi, On Fri, Nov 13, 2009 at 04:21:26PM -0600, fedora fedora wrote: Thanks for the reply, i disable the daemon option and here are the error messages, i cannot put src_ip and src_as together? INFO ( default/mysql ): 131070 bytes are available to address shared memory segment; buffer size is

Re: [pmacct-discussion] create my own mysql table

2009-11-13 Thread Paolo Lucente
Hi, On Fri, Nov 13, 2009 at 05:10:35PM -0600, fedora fedora wrote: sorry, one more question, when preparing the aggregate, does the order of the values matter? right now my aggregate is like the following No, doesn't matter. One more question, how can i get pmacct to show the flow number in

Re: [pmacct-discussion] NAT question

2009-11-11 Thread Paolo Lucente
Hi JF, As Karl said, libpcap looks what's on the wire and pmacct doesn't get further up in the packet layering. You can always do a quick check by verifying what tcpdump sees. While on NAT Linux, and perhaps not related to this specific issue: the uacctd daemon has been introduced in pmacct as

Re: [pmacct-discussion] No Sqlite3 logging

2009-11-09 Thread Paolo Lucente
Hi JF, On Mon, Nov 09, 2009 at 05:09:40AM -0500, JF Cliche wrote: debug: true syslog: kern plugin_buffer_size: 10240 plugin_pipe_size: 10240 interface: egiga0 daemonize: false promisc: false sql_cache_entries: 2 plugins: sqlite3[in] sql_max_writers: 2 pidfile: /vol1/pmacct.pid

Re: [pmacct-discussion] No Sqlite3 logging

2009-11-09 Thread Paolo Lucente
Hi JF, On Mon, Nov 09, 2009 at 10:26:35AM -0500, JF Cliche wrote: In any case, I cleaned- up my config file and made sure I filter nothing (see config below). I rechecked pmacctd using the memory plugin and data is being gathered. Then I relaunched with the sqlite3 plugin. 'pmacct -s' still

Re: [pmacct-discussion] tracking traffic flowing through a specific AS-path

2009-10-29 Thread Paolo Lucente
Hi Zenon, On Thu, Oct 29, 2009 at 10:09:16AM +0200, Zenon Mousmoulas wrote: BGP RIB lookups. However, for traffic flowing from sources behind that path (inbound for us), how would one go about doing just that? [ ... ] This is quite similar in concept to the mechanics of calculating the

Re: [pmacct-discussion] Long sflow BGP configuration

2009-10-28 Thread Paolo Lucente
Dear Maxence, On Wed, Oct 28, 2009 at 10:56:20AM +0100, Maxence Rousseau wrote: I didn't understood all the documentation... Do I need to use pmacctd ? It seems not necessary because sfacctd is able to launch BGP thread and log to mysql. What's the need of the pmacctd process in my case ?

Re: [pmacct-discussion] Long sflow BGP configuration

2009-10-28 Thread Paolo Lucente
Hi Maxence, On Wed, Oct 28, 2009 at 03:38:59PM +, Maxence Rousseau wrote: The information seems perfect, sample : INFO ( default/core/BGP ): [Id: ROUTER-ID] u Prefix: '208.74.16.0/21' Path: '174 3549 32952' Comms: '174:21000 174:22013' EComms: '' LP: '100' MED: '82011' Nexthop:

[pmacct-discussion] pmacct 0.12.0rc3 released !

2009-10-28 Thread Paolo Lucente
VERSION. 0.12.0rc3 DESCRIPTION. pmacct is a small set of passive network monitoring tools to measure, account, classify, aggregate and export IPv4 and IPv6 traffic; a pluggable and flexible architecture allows to store collected network data into memory tables or SQL (MySQL, SQLite, PostgreSQL)

Re: [pmacct-discussion] peer-AS calculation (plus netflow v9 sampling) questions

2009-10-27 Thread Paolo Lucente
Hi Zenon, On Tue, Oct 27, 2009 at 01:16:06PM +0200, Zenon Mousmoulas wrote: 1. Does bgp_peer_src_as_map apply both to src and dst AS or only the first? In any case, I don't understand how bgp_nexthop can be used for calculating the peer src AS, since that field applies to the destination

Re: [pmacct-discussion] dumping stored netflow into nfacctd

2009-10-26 Thread Paolo Lucente
Hi Charlie, The 'nfacctd_time_new' directive is by default set to false, meaning original timestamps would be honoured (and aggregates inserted in the correct time-bin). Would also suggest to ensure keeping the 'sql_dont_try_update' to false (which is the default) - as you are going to do

Re: [pmacct-discussion] Productivity Pre-Tagging [was] Traffic count only for certain networks

2009-10-26 Thread Paolo Lucente
Hi Slava, Although tagging can play in your case a key (negative) role under sustained loads, i wouldn't know if it is the prime contributor to such hang ups. The log below tells that either the router itself is unable to export all the NetFlow data or such data gets lost before making it to

Re: [pmacct-discussion] BGP-related fields do not show up in the memory table

2009-10-23 Thread Paolo Lucente
Hi Zenon, Thanks very much for your feedback first of all; please follow my replies in-line. On Fri, Oct 23, 2009 at 02:23:34AM +0300, Zenon Mousmoulas wrote: I am exporting netflow v9 (non-aggregated, for the time being) from a Cisco router (12000/PRP with 12.0S) to nfacctd (0.12.0rc2). I

Re: [pmacct-discussion] BGP-related fields do not show up in the memory table

2009-10-23 Thread Paolo Lucente
Hi Zenon, One additional point to my previous reply. On Fri, Oct 23, 2009 at 02:23:34AM +0300, Zenon Mousmoulas wrote: I am reluctant to use 'nfacctd_as_new: bgp' RIB lookups since we probably have this information already (exporter is setup for origin-as). Very true. And it depends on your

Re: [pmacct-discussion] Using nfacctd as netflow probe with high traffic levels

2009-10-11 Thread Paolo Lucente
Hi Chris, Was wondering if it could be a counter-rollover issue. Did you configure pmacct to compile with the --enable-64bit knob? Apart from the above - given you are aggregating on src_host,dst_host in pmacct, to save some precious space on your Sup720 NetFlow TCAM, i would recommend to use

Re: [pmacct-discussion] MySQL connection issues

2009-10-05 Thread Paolo Lucente
Hi Jeremy, On Mon, Oct 05, 2009 at 01:14:46PM +1000, Jeremy Lee wrote: I'm now discovering that there's often a huge time lag before the data goes into the database, ranging from a few minutes to up to an hour. I've got debug going, 20 seconds between SQL refreshes, and I get several:

Re: [pmacct-discussion] Traffic count only for certain networks

2009-09-23 Thread Paolo Lucente
Hi Slava, On Tue, Sep 22, 2009 at 11:42:37PM +0300, Slava Dubrovskiy wrote: I wish to count only traffic from/to local IP. I.e. that in base were only my local IP. For this purpose I specify my local IP in networks_file and I aggregate by src_host, dst_host. I don't wish to store not

Re: [pmacct-discussion] reloading config accuracy

2009-09-21 Thread Paolo Lucente
Hi Tony, On Sun, Sep 20, 2009 at 06:03:18PM -0700, Tony wrote: I haven't upgraded yet, I will be doing that now, but I wanted to give you some feedback on what I'm seeing in the old version and we can see if it persists to the new version. [ ... ] 10306644 10306462182

Re: [pmacct-discussion] Format of storage IP addresses in database.

2009-09-13 Thread Paolo Lucente
Hi Slava, On Wed, Sep 09, 2009 at 09:38:03PM +0300, Slava Dubrovskiy wrote: Why for store IP it is used CHAR type? It's better use INT UNSIGNED and use INET_ATON() and INET_NTOA() for convert. http://dev.mysql.com/doc/refman/5.0/en/miscellaneous-functions.html#function_inet-aton This is

Re: [pmacct-discussion] Traffic count only for certain networks

2009-09-13 Thread Paolo Lucente
Hi Slava, On Thu, Sep 10, 2009 at 02:24:22PM +0300, Slava Dubrovskiy wrote: [ ... ] It is necessary to count the traffic with filtering on certain networks. For this purpose I need write ALL this networks to aggregate_filter. But this list of networks is big and it periodically varies. I

Re: [pmacct-discussion] not build (was pmacct 0.12.0rc2 released !)

2009-09-09 Thread Paolo Lucente
, 2009 at 10:41:39PM +0300, Slava Dubrovskiy wrote: 09.09.2009 20:00, Paolo Lucente ??: VERSION. 0.12.0rc2 it not build. [ ... ] rc1 is building without problems. Do you have any advices? -- WBR, Dubrovskiy Vyacheslav

Re: [pmacct-discussion] reloading config accuracy

2009-09-06 Thread Paolo Lucente
Hi Tony, On Sat, Sep 05, 2009 at 09:01:01PM -0700, Tony wrote: I have tested the above suggested configuration and it is working. There is data going into the SQL table now! I am going to let it run in parallel with the unadjusted data (which is going into another table) and then compare

Re: [pmacct-discussion] nfacctd_renormalize and sql_preprocess: usrf=10 not working

2009-09-04 Thread Paolo Lucente
Hi Slava, On Fri, Sep 04, 2009 at 10:10:46PM +0300, Slava Dubrovskiy wrote: I trying use nfacctd_renormalize option and use sql_preprocess: usrf=10 [ ... ] And after as I add sql_preprocess: usrf=10 it stop write to database. If I comment sql_preprocess: usrf=10 - working, but not

Re: [pmacct-discussion] timestamp rounding bug

2009-08-20 Thread Paolo Lucente
current state of this question. Alex On 04/19/2009 01:00:48 PM, Paolo Lucente wrote: Hi Alex, DST is not supported. Timezones are. The idea behind this was that a backend application (like pmacct is) should ideally work only with UTC (even if timezones are supported) and then front-ends

Re: [pmacct-discussion] pmacctd commands to match cisco ip flow-cache timeout

2009-08-20 Thread Paolo Lucente
Hi Stig, That is correct. Align all other available timeouts (tcp, udp and icmp other than the general one which you already mentioned) to the inactive value. Active timeout goes definitely mapped to maxlife instead. Cheers, Paolo On Wed, Aug 19, 2009 at 07:59:45PM -0700, Stig Thormodsrud

Re: [pmacct-discussion] segv with memory,sfprobe plugins

2009-08-18 Thread Paolo Lucente
Hi Stig, thanks very much for having reported the issue. This is now solved in the CVS. I managed to reproduce it. It was lying in the fact that initialization of the sfprobe plugin was explicitely disabling the IP fragment handler in pmacctd; this was causing the IMT plugin, configured with L4

Re: [pmacct-discussion] Bug in pmacct-0.12.0rc1?

2009-08-06 Thread Paolo Lucente
? In such a case, the solution is to go with 64-bit counters. Let me know. Cheers, Paolo PS: very verbose outputs are very much welcome but go in private email (perhaps compressed) :-) On Thu, Aug 06, 2009 at 07:33:57PM +0200, Johannes Formann wrote: Paolo Lucente pa...@pmacct.net wrote: Hi Paolo

Re: [pmacct-discussion] Bug in pmacct-0.12.0rc1?

2009-08-05 Thread Paolo Lucente
Hi Johannes, The only idea coming to the mind is that two pmacctd instances (each with two plugins) are fired up in parallel and trying to write to the same MySQL tables. If it's not something in this sense, i would ask you to turn on debugging on one of the two plugins (debug[inbound]: true

Re: [pmacct-discussion] timestamp rounding bug

2009-08-05 Thread Paolo Lucente
On Tue, Aug 04, 2009 at 09:18:22PM -0500, Karl O. Pinc wrote: On 08/04/2009 04:35:31 AM, Chris Wilson wrote: Is any real-world system set to UTC? I'm certainly not going to run my firewall (where I run pmacct currently) on UTC. All my logs would be screwed up and much harder to

Re: [pmacct-discussion] multiple interfaces uni-directional flows

2009-08-04 Thread Paolo Lucente
Hi Stig, Very briefly to confirm: a) you are correct, libpcap captures both inbound and outbound traffic and b) the workaround you have put in place not only makes sense but is also by far the most efficient way to filter traffic out of pmacctd. Cheers, Paolo On Tue, Aug 04, 2009 at 10:39:00AM

Re: [pmacct-discussion] dynamic sql tables not created

2009-08-03 Thread Paolo Lucente
Hi Christian, I'm not sure how a change to the frontend can influence that way the backend. Two things to check: 'sql_history' is in use (as it generates the timestamp used at a later stage to work out the name of the dynamic table) and sql_table_schema file is readable. Moreover, something

<    3   4   5   6   7   8   9   10   >