Re: [pmacct-discussion] pmpgplay - what does it do?

2010-07-23 Thread Paolo Lucente
Hi Chris, The 'failsafe' mechanism (ie. writing to a backup database or to log files) kicks in upon receipt of an error code from the RDBMS API. So what you see in the log file should never be already in the database. Your specific configuration is tricky because you write to the RDBMS every 60 s

Re: [pmacct-discussion] How many ips can pmacctd monitorize

2010-07-22 Thread Paolo Lucente
Hi, Try performing a locked access to the memory table. This can be done by appending a '-l' option to the command, ie. "pmacct -s -l -p ...". Let me know. Cheers, Paolo On Thu, Jul 22, 2010 at 10:35:18AM +, Jose Joaquin Anton Herrerias wrote: > I was Reading CONFIG-KEYS and internals docume

Re: [pmacct-discussion] Netflow and 'ip' key in pretag.map

2010-07-22 Thread Paolo Lucente
Hi Damian, Are you running BGP? Would it be feasible for you to past BGP feed(s) into pmacct (granted you run a recent 0.12 release or can upgrade to that)? Idea is you can attach BGP standard communities to IP prefixes as they are advertised or re-distributed into your network. Because comunities

Re: [pmacct-discussion] Netflow and 'ip' key in pretag.map

2010-07-20 Thread Paolo Lucente
Hi Damian, To confirm wildcarding of the agent is not supported. You can multiplex values onto a tag by employing a combination of 'jeq' and 'stack' keys. Perhaps having more contextual information on what you want to achieve would help. Cheers, Paolo On Tue, Jul 20, 2010 at 01:11:22PM +1200,

Re: [pmacct-discussion] How many ips can pmacctd monitorize

2010-07-19 Thread Paolo Lucente
Hi, I'd say limit on IP addresses that can be accounted of is only imposed by resources available (memory) and, when using memory tables (which is your case), the imt_mem_pools_number setting (see CONFIG-KEYS for more details). I see a couple of issues with your config: * quite large plugin_buff

Re: [pmacct-discussion] save raw flows to file

2010-07-08 Thread Paolo Lucente
Hi Nitzan, I'm sure you already know pmacct doesn't support logging to raw files (FAQS document, Q5). As you can read, by choice. I believe pmacct can anyway offer something similar to what you have in mind: via configuration you have full control over spatial aggregation (so you just select prim

Re: [pmacct-discussion] pmacct/pmacctd dumping core on NetBSD 5.

2010-06-30 Thread Paolo Lucente
Hi Ed, Please re-compile the package with --enable-debug so that the gdb backtrace is a bit more rich of information. If output gets too long, consider sending it privately and then we will wrap up the issue on list. Which architecture you run NetBSD on? Which pmacct version are you trying to buil

Re: [pmacct-discussion] Traffic on Linux Bridge

2010-06-22 Thread Paolo Lucente
Hi Tony, Validate the traffic you see off of that inferface with tcpdump; once you reckon you have it, pmacct can let you accomplish your goal: aggregate inbound/outbound traffic per VM IP address and flexibly store data into a PostgreSQL database. Cheers, Paolo On Tue, Jun 22, 2010 at 11:09:34A

Re: [pmacct-discussion] Error for building pmacct 0.12 from source

2010-06-22 Thread Paolo Lucente
Hi Rodolfo, In addition to what Brent correctly said you might want to double check whether you have development kit for MySQL installed at all (libraries and headers). A chance can be you have got installed only, say, MySQL client, server and tools. Cheers, Paolo On Tue, Jun 22, 2010 at 12:01:0

Re: [pmacct-discussion] OS for deploy and usage scenario

2010-06-18 Thread Paolo Lucente
Hi Rodolfo, Will leave good points on which OS or Linux distribution to prefer to the others. Capturing traffic off of a span port is task for 'pmacctd'. It's not clear to me whether the router is the Linux box or alternatively what mirrors traffic (router, switch, optical tap, etc.). Be aware i

Re: [pmacct-discussion] data only in one direction

2010-06-18 Thread Paolo Lucente
Hi Morgan, INTERNALS document is part of the pmacct distribution tarball that you download (ie. pmacct-0.12.2.tar.g). Look into docs/. Cheers, Paolo On Fri, Jun 18, 2010 at 08:35:53AM +0200, Morgan Sellier wrote: > Hi, > > Thanks for your help ! > I will test it today but in the documentation

Re: [pmacct-discussion] Help as the sum of certain ips

2010-06-18 Thread Paolo Lucente
Hi Rafael, Advice is to check out Q8 from FAQS, EXAMPLES document chapter III, 'sql/README.pgsql' for initial SQL table setup. It should be enough to work out initial configs. Cheers, Paolo On Thu, Jun 17, 2010 at 05:14:12PM -0300, Rafael Stein wrote: > Hello to all, > > I'm starting to use th

Re: [pmacct-discussion] data only in one direction

2010-06-17 Thread Paolo Lucente
Hi Morgan, The memory plugin comes with some default settings to avoid taking over all system memory. Chance might be some aggregates might not fit in the default table. You can verify if this is the case by either enabling debug (very verbose!) or adding a "imt_mem_pools_number: 0" line to your c

Re: [pmacct-discussion] inet Error

2010-06-11 Thread Paolo Lucente
Hi Jonathan, It's a bit hard to say with this information. Can you post your config? Are you using any of the default SQL schemas? Which options did you compile the package? Cheers, Paolo On Fri, Jun 11, 2010 at 01:11:23PM +1000, Jonathan Gleeson wrote: > Hi Guys, > > Anyone got any idea on thi

Re: [pmacct-discussion] Wrong byte count?

2010-06-05 Thread Paolo Lucente
Hi, There is clearly something wrong with that; configuration looks OK. But 0.11.6 is a bit old release and it's tricky for me to make a guess on a pre-compiled package. Can you please download latest available release from the pmacct website, compile it yourself according to your needs and verif

Re: [pmacct-discussion] nfprobe won't send out data until i hit ctrl-c?

2010-06-05 Thread Paolo Lucente
Hi, Configuration is lean and looks OK apart a typo on the nfprobe_receiver line, but should be irrelevand and only arisen while anonymizing it, as otherwise you would have got an error back. Indeed, behaviour is not expected and i can't reproduce it. Can you elaborate on: which operating system

[pmacct-discussion] pmacct 0.12.2 released !

2010-05-27 Thread Paolo Lucente
VERSION. 0.12.2 DESCRIPTION. pmacct is a small set of passive network monitoring tools to account, classify, aggregate and export IPv4 and IPv6 traffic; a pluggable architecture allows to store collected network data into memory tables or SQL (MySQL, SQLite, PostgreSQL) databases and export them

Re: [pmacct-discussion] nfacctd log message explanation sought

2010-05-20 Thread Paolo Lucente
Hi Rod, As classification of the log messages suggests, the only to worry about is the last one, which is a warning. For all the INFO/DEBUG messages you better off starting from the docs/INTERNAL document, part of the standard distribution tarball. Should you have any further questions, will be

Re: [pmacct-discussion] SQL duplicate key

2010-05-20 Thread Paolo Lucente
Hi Sergio, Please have a look to the following thread and see if it helps: http://www.mail-archive.com/pmacct-discussion@pmacct.net/msg01545.html It might be the case as i guess you are all set for 5 minutes time-bins ('sql_history: 5m', 'sql_refresh_time: 300') but i see very "unaligned" stamp_

Re: [pmacct-discussion] PostgreSQL large database

2010-05-14 Thread Paolo Lucente
Hi Sergio, On Fri, May 14, 2010 at 10:58:00AM -0300, Sergio Charpinel Jr. wrote: > I couldnt get any useful information from this command. > I get no erros in postgresql, nfacctd and pmacctd log files. I would expect you to see a "Maximum number of SQL writer processes reached" message in the lo

Re: [pmacct-discussion] MySQL index performance

2010-05-13 Thread Paolo Lucente
Hi Chris, > I think it would make no semantic difference, but would increase MySQL > performance with these table types, if the primary key listed > stamp_inserted first instead of last. The change you propose, as you say, would not be impacting - but would you have any testing handy which conf

Re: [pmacct-discussion] PostgreSQL large database

2010-05-13 Thread Paolo Lucente
Hi Sergio, It looks those processes are locked out of the table they want to write to. In MySQL you can check this kind of stuff with a "SHOW PROCESSLIST"; the PostgreSQL equivalent should be "SELECT * FROM pg_stat_activity". Its output might very well shed some light. Just btw, the number of pma

Re: [pmacct-discussion] pmacct sensor on lacp

2010-05-06 Thread Paolo Lucente
Hi Yuriy, Should be fixed now. Cheers, Paolo On Thu, May 06, 2010 at 11:59:52AM +0300, Yavetskiy Yuriy wrote: > Hello. > > Error while updating from cvs: > cvs checkout: failed to create lock directory for > `/home/repo-0.12/pmacct/src/tee_plugin' > (/home/repo-0.12/pmacct/src/tee_plugin/#c

Re: [pmacct-discussion] pmacct sensor on lacp

2010-05-05 Thread Paolo Lucente
Hi Yuriy, That segmentation fault deep in the libc remembers me we did see something on those lines while troubleshooting a different issue on your collector box: http://www.mail-archive.com/pmacct-discussion@pmacct.net/msg01475.html At that stage, the only thing that did help against these was

Re: [pmacct-discussion] Expiring Orphan fragment

2010-05-04 Thread Paolo Lucente
Hi, To wrap-up this thread - unless new details emerge: On Tue, May 04, 2010 at 11:53:59AM +, Paolo Lucente wrote: > > * one capturing some torrent traffic, so that i can replay it in a > testbed and see if i can reproduce and validate the behaviour. Orphan fragments are rea

Re: [pmacct-discussion] Expiring Orphan fragment

2010-05-04 Thread Paolo Lucente
Hi Sergio, On Tue, May 04, 2010 at 08:44:55AM -0300, Sergio Charpinel Jr. wrote: > Yes, it is working, thanks. Good! > And I analyzed some flows related to expiring orphan, and most of them > seems to be related to torrents, but I'm not sure. > > [ ... ] > > WARN: expecting flow '817086981' b

Re: [pmacct-discussion] pmacct segmentation fault when compiled with threads on FreeBSD

2010-05-04 Thread Paolo Lucente
Hi Jason, Thanks for the feedback. Unfortunately that backtrace doesn't help. Can you please re-compile the package appending also "--enable-debug"? It helps giving a more verbose and clueful backtrace in gdb. I'd hence propose to continue this privately and wrap-up here. Cheers, Paolo On Mo

Re: [pmacct-discussion] Expiring Orphan fragment

2010-05-03 Thread Paolo Lucente
Hi Sergio, Good point. I've just marked PGRES_TUPLES_OK as valid return code within the PG_create_dyn_table() function - and committed the change to the CVS. Can you please confirm it works OK for you? Cheers, Paolo On Mon, May 03, 2010 at 10:10:18AM -0300, Sergio Charpinel Jr. wrote: > Paolo

Re: [pmacct-discussion] Expiring Orphan fragment

2010-05-01 Thread Paolo Lucente
Hi Sergio, Thanks indeed for sharing the function. It logs because the PostgreSQL PQexec() function is returning pmacct something else than PGRES_COMMAND_OK. One suggestion might be to play with the exit/return code of the PL/PGSQL function. The error string ultimately is empty (and that is pass

Re: [pmacct-discussion] Expiring Orphan fragment

2010-04-30 Thread Paolo Lucente
Hi Sergio, On Thu, Apr 29, 2010 at 11:23:35AM -0300, Sergio Charpinel Jr. wrote: > Does anyone know what this mean? I getting a lot of them in pmacctd.log > Expiring orphan fragment: ip_src=210.197.202.84 ip_dst=200.137.66.1 proto=17 > id=8885 It means some IP fragments have been staying too lon

Re: [pmacct-discussion] Low performance query

2010-04-28 Thread Paolo Lucente
but increasing history would do the trick? > > Thanks for answering. > > Cheers. > > 2010/4/27 Paolo Lucente > > > Hi Sergio, > > > > I don't know FloX very well - hence would be good information to know > > which specific SQL queries are per

Re: [pmacct-discussion] Low performance query

2010-04-27 Thread Paolo Lucente
Hi Sergio, I don't know FloX very well - hence would be good information to know which specific SQL queries are performing bad. Maybe there is room to improve indexing. Is it also your goal to store every micro-flow into the SQL database? Any chance a more compact aggregation method would fit th

Re: [pmacct-discussion] sflow in_iface out_iface information wrong

2010-04-27 Thread Paolo Lucente
Hi Bernd, If i get it correctly, you should be referring to the big 1073741823 and 2147483648 values in your SQL table. According to the sFlow (v5) specifications, these values are perfectly valid: 1073741823 => 0x3FFF: "[ ... ] this is used in describing traffic which is not bridged, routed,

Re: [pmacct-discussion] Duplicate entry problem.

2010-04-16 Thread Paolo Lucente
Hi Sergey, Duplicates are a clear consequence of the "urgent" DB writer in conjunction with disabling UPDATE queries (sql_dont_try_update) and the configured (default, i guess) primary key. Urgent writers are elicited by shortage of entries available in the SQL cache. Having memory available, my

Re: [pmacct-discussion] ifindex and netmask records

2010-04-09 Thread Paolo Lucente
e filled from the BGP protocol - just as before. Hope this is of interest. Cheers, Paolo On Fri, Apr 02, 2010 at 10:19:31AM +, Paolo Lucente wrote: > Hi Richard, > > On Fri, Apr 02, 2010 at 03:12:23AM -0500, Richard A Steenbergen wrote: > > > * Record (and aggregate o

[pmacct-discussion] pmacct 0.12.1 released !

2010-04-07 Thread Paolo Lucente
VERSION. 0.12.1 DESCRIPTION. pmacct is a small set of passive network monitoring tools to measure, account, classify, aggregate and export IPv4 and IPv6 traffic; a pluggable and flexible architecture allows to store collected network data into memory tables or SQL (MySQL, SQLite, PostgreSQL) data

Re: [pmacct-discussion] ifindex and netmask records

2010-04-02 Thread Paolo Lucente
Hi Richard, On Fri, Apr 02, 2010 at 03:12:23AM -0500, Richard A Steenbergen wrote: > * Record (and aggregate on) the address of the router that exported a > flow via netflow/sflow. Basically I just want to know which router > exported the flow to me, using either the agent address if available

Re: [pmacct-discussion] Creating dynamic mysql tables

2010-03-29 Thread Paolo Lucente
Hi Thomas, Can you please elaborate where you get such error? It looks the variables are not expanded but i'm not sure if you are making use of the 'sql_table_schema' configuration directive or not. Cheers, Paolo On Sun, Mar 28, 2010 at 04:29:00PM +1100, Thomas wrote: > Hi, > > I'm lost. I trie

Re: [pmacct-discussion] uacctd documentation

2010-03-26 Thread Paolo Lucente
Hi Ross, On Fri, Mar 26, 2010 at 02:46:51PM -0400, Ross Vandegrift wrote: > I'm curious if I can acheive better performance for generating sflow > data. My plan was to use the iptables statistics module to do the > sampling, to ensure that only sampled packets were being sent to > userspace. Si

Re: [pmacct-discussion] uacctd documentation

2010-03-26 Thread Paolo Lucente
Hi Ross, On Thu, Mar 25, 2010 at 09:57:05AM -0400, Ross Vandegrift wrote: > Thanks for the examples! I'm having trouble with the iptables piece > of the puzzle though. I suspect this is because I'm mirroring traffic > to this server and the L2 destination doesn't match any address > present on

Re: [pmacct-discussion] BWStat with improved queries and indexes

2010-03-21 Thread Paolo Lucente
Hi Matthew, I guess this can be of general interest, so please go ahead. The post can be a good resource expecially because it's being publicly archived. If there are files to attach or you feel it can get extremely long, consider publishing the content by some other means (ie. your blog or webs

Re: [pmacct-discussion] aggregate-filter min packet count and bgp sql table questions

2010-03-21 Thread Paolo Lucente
x27;ll be logging in from and > I can update my hosts.allow for you and setup an acct. > > Thanks! > -Brent > > On Mar 13, 2010, at 3:26 AM, Paolo Lucente wrote: > >> Hi Brent, >> >> Thanks for reporting the issue. I know of at least one >> other peop

Re: [pmacct-discussion] aggregate-filter min packet count and bgp sql table questions

2010-03-13 Thread Paolo Lucente
INFO: Connection refused while trying to connect to '/tmp/ > sfacctd_prefixes.pipe' > > In my logs I get: > > INFO: connection lost to 'prefixes-memory'; closing connection. > > > Thanks, > -Brent > > > On Mar 12, 2010, at 4:04 AM, Paolo Lucente wr

Re: [pmacct-discussion] aggregate-filter min packet count and bgp sql table questions

2010-03-12 Thread Paolo Lucente
Hi Brent, Good to see progress. The entries stay "forever", there is not an aging-out mechanism. Reason being you are supposed to do it yourself, at regular intervals, to build a time reference for the counters. For example a simplistic scenario is a cronjob entry, set up every 5 minutes, that

Re: [pmacct-discussion] aggregate-filter min packet count and bgp sql table questions

2010-03-11 Thread Paolo Lucente
Hi Brent, On Thu, Mar 11, 2010 at 10:01:26AM -0800, Brent Van Dussen wrote: > I wanted to ask about these messages we're getting in the logs now that > we're using mem tables: > > WARN ( prefixes/memory ): Unable to allocate more memory pools, clear > stats manually! > WARN ( as_path/memory ):

Re: [pmacct-discussion] aggregate-filter min packet count and bgp sql table questions

2010-03-11 Thread Paolo Lucente
gt; Thanks for getting this set up Paolo! >> >> We'll get the latest CVS version loaded and tested this week to >> provide feedback. >> >> Cheers, >> -Brent >> >> On Mar 7, 2010, at 1:34 AM, Paolo Lucente wrote: >> >>> Hi Brent, All,

Re: [pmacct-discussion] aggregate-filter min packet count and bgp sql table questions

2010-03-07 Thread Paolo Lucente
Hi Brent, All, On Sat, Feb 20, 2010 at 01:05:20AM +, Paolo Lucente wrote: > > Would it also be possible to have the dst_net appended with mask length > > and a slightly larger DB field to accomodate it? 255.255.255.255/25 > > would be a CHAR(18) instead of CHAR(15) but

Re: [pmacct-discussion] dynamic IP

2010-03-03 Thread Paolo Lucente
Hi Jeff, You can opt to switch to a pre_tag_filter and tag packets basing on a filter. What matches the filter gets a certain tag and such tag is allowed through. The beauty of this method is that the pre_tag_map can be reloaded at runtime - by switching refresh_maps to true. Still, as Karl sugg

Re: [pmacct-discussion] Log output from pmacctd Ive never seen before

2010-03-03 Thread Paolo Lucente
Hi Jeff, How often you get this message? Every time you see it, you miss a packet. The way to read that line is: libpcap passed only the first 37 bytes of the datagram to pmacct; L4 appears to start at the 38th octet; and you might have specified src_port or dst_port in your aggregation method.

Re: [pmacct-discussion] aggregate-filter min packet count and bgp sql table questions

2010-02-19 Thread Paolo Lucente
Hi Brent, On Fri, Feb 19, 2010 at 10:51:21AM -0800, Brent Van Dussen wrote: > I was curious if there was a way to have sfacctd only insert into the > database if a certain number of packets and/or bytes threshold is > reached. It seems you are looking for the sql_preprocess directive - and m

Re: [pmacct-discussion] Pmacct data inconsistencies between tables.

2010-02-19 Thread Paolo Lucente
ds > > -- > Daniel Levy > > Aptivate | http://www.aptivate.org/ | +44 (0)1223 760887 > The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES > > Aptivate is a not-for-profit company registered in England and Wales > with company number 04980791. > >

Re: [pmacct-discussion] unable to compile latest pmacct with pfring enabled libpcap ...

2010-02-18 Thread Paolo Lucente
> Note that we are using pf_ring and it works perfectly with ntop ... > Could you try to build it using libpcap from pf_ring svn ? > > svn co https://svn.ntop.org/svn/ntop/trunk/PF_RING/ > > > > 2010/2/18 Paolo Lucente > > > Hi Sebastien, > > >

Re: [pmacct-discussion] unable to compile latest pmacct with pfring enabled libpcap ...

2010-02-18 Thread Paolo Lucente
Hi Sebastien, It's not clear to me if this was working for you before (some earlier pmacct release) and it doesn't instead with the latest. I've just tried myself to compile pmacct 0.12.0 against a libpcap 1.0.0 (vanilla) and it works fine. I'm also fairly sure this worked up to 0.12.0rc4 for othe

[pmacct-discussion] pmacct 0.12.0 released !

2010-02-16 Thread Paolo Lucente
VERSION. 0.12.0 DESCRIPTION. pmacct is a small set of passive network monitoring tools to measure, account, classify, aggregate and export IPv4 and IPv6 traffic; a pluggable and flexible architecture allows to store collected network data into memory tables or SQL (MySQL, SQLite, PostgreSQL) data

Re: [pmacct-discussion] Pmacct data inconsistencies between tables.

2010-02-16 Thread Paolo Lucente
wo tables. Would this give you the information > you're looking for? > > -- > Daniel Levy > > Aptivate | http://www.aptivate.org/ | +44 (0)1223 760887 > The Humanitarian Centre, Fenner's, Gresham Road, Cambridge CB1 2ES > > Aptivate is a not-for-profit compa

Re: [pmacct-discussion] pmacct count only 5% of SYN packets.

2010-02-12 Thread Paolo Lucente
Hi Yuriy, Which version of pmacct you are using? Indeed the syntax for those SQL queries is wrong - but i've never seen that happening so i'm a bit puzzled. Are these issues related to a specific plugin or you can see such weird behaviour across all of them? Finally, can you post privately some of

Re: [pmacct-discussion] Pmacct data inconsistencies between tables.

2010-02-12 Thread Paolo Lucente
Hi Daniel, Unfortunately the configuration doesn't make evident where the issue can be. The 'sql_dont_try_update' very well protects against duplicate tuples - so i'm rather inclined to exclude that reason. Which version are you using? How you did discover the issue - ie. did you upgrade recentl

Re: [pmacct-discussion] Missing information in mysql table

2010-02-09 Thread Paolo Lucente
Hi Jonas, On Tue, Feb 09, 2010 at 04:55:52PM +0100, Jonas Nylund wrote: > > [ ... ] > > mysql> select * from acct_v5_06 limit 10; > +--+--+-+-+--+-+-+--+--+--+-+-+---+---+-

Re: [pmacct-discussion] pmacct count only 5% of SYN packets.

2010-02-09 Thread Paolo Lucente
Hi Yuriy, You have also other means to get a count of the TCP/SYN packets out of pmacct. I would suggest one for troubleshooing purposes with the goal to check where the issue lies: * keep the 'tcpflags' primitive out of the 'aggregate' directive * add a 'pcap_filter' directive to the config; it

Re: [pmacct-discussion] pre_tag_map issue + questions

2010-02-09 Thread Paolo Lucente
Hi Zenon, On Mon, Feb 08, 2010 at 02:43:49PM +0200, Zenon Mousmoulas wrote: >> records. If it's in there, then i'd like to give it a look myself: i >> would ask you to produce a trace and send it to me privately so that >> i can have a look. We can then summarize findings here. > > OK. I will sen

Re: [pmacct-discussion] The accounting of flows requires SQL table v4?

2010-02-08 Thread Paolo Lucente
ied adding this directive (and tried it one more time > just before writing this). Unfortunately it made no difference... > > Z. > > On 08 ?? 2010, at 11:51 , Paolo Lucente wrote: > >> Hi Zenon, >> >> Good to see you around again. Please add to your c

Re: [pmacct-discussion] pre_tag_map issue + questions

2010-02-08 Thread Paolo Lucente
Hi Zenon, On Mon, Feb 08, 2010 at 10:37:54AM +0200, Zenon Mousmoulas wrote: > I have a netflow v9 feed to nfacctd from a juniper router (JUNOS > 9.6R2.11), using a service pic. According to a packet capture, records > include ingress and egress interface and they seem to be properly > defin

Re: [pmacct-discussion] The accounting of flows requires SQL table v4?

2010-02-08 Thread Paolo Lucente
Hi Zenon, Good to see you around again. Please add to your config: sql_table_version[sqltest]: 4 Reason being most of the primitives are still connected to the SQL table versioning concept (in essence: pmacct expects you to make explicit which SQL schema you are running). This is in the process

Re: [pmacct-discussion] Not working pmacct -N [matching data['; ' ... ]]

2010-02-04 Thread Paolo Lucente
Hi Slava, Can you confirm which version you are running into this issue? Can you also post your config - just in case? Labbing this scenario up, i see it working fine for me. One thing i can suggest is: append a "-l" to your query to ensure a locked access to the memory table; if it's a concurre

Re: [pmacct-discussion] Enterasys nfacctd expecting flow error

2010-01-15 Thread Paolo Lucente
8GB of memory. CPU is seeing very little use at all times. > > > > - Original Message > From: Paolo Lucente > To: pmacct-discussion@pmacct.net > Sent: Wed, January 13, 2010 3:33:36 PM > Subject: Re: [pmacct-discussion] Enterasys nfacctd expecting flow error &g

Re: [pmacct-discussion] Enterasys nfacctd expecting flow error

2010-01-13 Thread Paolo Lucente
Hi Marc, Such messages tell it has been detected some issues with NetFlow datagram sequence numbers. This can be caused by packet loss between an agent and the collector, mistakes in the sequencing encoding among the others. Besides the warning messages, which can be turned off, NetFlow datagrams

Re: [pmacct-discussion] export netflow v5 for 10G interface

2010-01-09 Thread Paolo Lucente
Hi Anatoliy, I would recommend a good current CPU; dual-CPU being also beneficial for this job. As of memory, i don't have any specific recommendation as it might vary depending on the traffic footprint: but 1GB should be more than enough. Attention point is operating system choice and underlyin

Re: [pmacct-discussion] pmacct can't create mysql table, but I can with the same schema

2009-12-22 Thread Paolo Lucente
Hi Jeff, If i'm getting this correctly, you see everything working except for the table creation error you get back. If this is the case, i wouldn't say you are doing something wrong; pmacct tries to create the table every time the SQL cache scanner kicks in (sql_refresh_time interval); table na

Re: [pmacct-discussion] sflow renormalize not working correctly (sql_preprocess does)

2009-12-21 Thread Paolo Lucente
Hi Igor, I'm not aware of any issues with the 'sfacctd_renormalize' feature; perfect you already did some debug with sflowtool - that would have been my first suggestion. I would ask you, if possible, to send me privately a brief capture of some sFlow datagrams (pcap format, full packets) so that

[pmacct-discussion] pmacct 0.12.0rc4 released !

2009-12-21 Thread Paolo Lucente
VERSION. 0.12.0rc4 DESCRIPTION. pmacct is a small set of passive network monitoring tools to measure, account, classify, aggregate and export IPv4 and IPv6 traffic; a pluggable and flexible architecture allows to store collected network data into memory tables or SQL (MySQL, SQLite, PostgreSQL) d

Re: [pmacct-discussion] Aggregate traffic count with BGP community match

2009-12-11 Thread Paolo Lucente
Hi Andrew, On Fri, Dec 11, 2009 at 01:32:34PM +1300, Andrew Thrift wrote: > I have a requirement to count clients traffic passing through our border > router by classes, e.g. local (iBGP), national (BGP marked with specific > community) and international (anything not covered by local/nationa

Re: [pmacct-discussion] Layer 7 classification problem.

2009-12-08 Thread Paolo Lucente
, I don't enter the discussion of the precision of the 3rd party regex classifiers. Action i've taken is to add a note in the documentation. Cheers, Paolo On Wed, Dec 02, 2009 at 10:53:04AM +0000, Paolo Lucente wrote: > Hi Buddhike, > > On Wed, Dec 02, 2009 at 02:36:20PM +0

Re: [pmacct-discussion] buffer overflow detected ***: nfacctd: Core Process [default] terminated

2009-12-04 Thread Paolo Lucente
Hi Slava, I essentially see you reporting two different issues: a) the debug message saying "unknown template"; which you should see disappearing after a while, ie. as soon as the router exports the template to pmacct. Before that happens, pmacct doesn't know how to parse the NetFlow v9

Re: [pmacct-discussion] Layer 7 classification problem.

2009-12-02 Thread Paolo Lucente
Hi Buddhike, On Wed, Dec 02, 2009 at 02:36:20PM +0530, Buddhike wrote: > > http://www.mail-archive.com/pmacct-discussion@pmacct.net/msg01397.html > > > Any chance you can give it a try and post some feedback, ie. whether it > > yeah I've tried that with pmacct-0.12.0rc3 and still the result is

Re: [pmacct-discussion] Not save data in DB when exit

2009-12-02 Thread Paolo Lucente
Hi Slava, On Tue, Dec 01, 2009 at 10:01:28PM +0200, Slava Dubrovskiy wrote: > Yes, I remove sql_multi_values and it working. But as I understand this > this give more performance. Was suspecting so & correct. > Is it possible use it? And it working good. Problem only when I stop it. It has to

Re: [pmacct-discussion] Not save data in DB when exit

2009-11-30 Thread Paolo Lucente
Hi Slava, On Mon, Nov 30, 2009 at 03:03:41PM +0200, Slava Dubrovskiy wrote: > I see such errors: > > [ ... ] > > DEBUG ( t2/mysql ): 3 VALUES statements sent to the MySQL server. > ERROR ( t2/mysql ): Duplicate entry '0-5-2009-11-28 02:00:00' for key 1 > > [ ... ] > > DEBUG ( t1/mysql ): 400 VAL

Re: [pmacct-discussion] Netflow v5 / BGP export as Netflow v8 aggregated by as

2009-11-29 Thread Paolo Lucente
Hi Andre, On Sun, Nov 29, 2009 at 06:17:02PM +0100, Andre Keller wrote: > I'm very new to pmacct. I attended the presentation Paolo held the last > swinog meeting and go interessted Glad to have somebody from SwiNOG on this discussion list. > So I tried to get this working with pmacct step-by-s

Re: [pmacct-discussion] peer-AS calculation problem

2009-11-28 Thread Paolo Lucente
On Thu, Nov 26, 2009 at 09:36:35AM +0200, Zenon Mousmoulas wrote: > You are right. So far the only reason for v9 was exporting BGP next-hop. > I would wish that IPv6 accounting was supported, but it isn't, on this > platform. > In that respect, v9 turns out to be just a more expensive way to exp

Re: [pmacct-discussion] Layer 7 classification problem.

2009-11-27 Thread Paolo Lucente
Hi, On Fri, Nov 27, 2009 at 11:27:49AM +, buddhi...@gmail.com wrote: > yeah I went through that link through also but couldn't get the problems > solved yet. As part of that thread, I posted a patch but never got back from Mike: http://www.mail-archive.com/pmacct-discussion@pmacct.net/msg

Re: [pmacct-discussion] Not save data in DB when exit

2009-11-27 Thread Paolo Lucente
Hi Slava, On Thu, Nov 26, 2009 at 02:42:22PM +0200, Slava Dubrovskiy wrote: > I use "killall INT nfacctd" and "killall -s INT nfacctd" and by script > > [ ... ] > > Not work. But previos version (rc2) working good. Don't manage to reproduce the issue with rc3 on Linux and Solaris; i see it work

Re: [pmacct-discussion] Not save data in DB when exit

2009-11-25 Thread Paolo Lucente
Hi Slava, On Wed, Nov 25, 2009 at 09:04:24PM +0200, Slava Dubrovskiy wrote: > Seems when I make kill INT PID_OF_CORE_PROCESS it down, but plugins do > not write to database. I see delay before off for plugins, but not see > that they change command line to "DB writer". And not see data for period

Re: [pmacct-discussion] peer-AS calculation problem

2009-11-25 Thread Paolo Lucente
Hi Zenon, On Thu, Nov 26, 2009 at 01:51:44AM +0200, Zenon Mousmoulas wrote: > I was under the impression that 'nfacctd_as_new: bgp' would cause > nfacctd to lookup ASNs even though the origin ASN is already exported in > netflow datagrams; this is something I was trying to avoid. Consider the

[pmacct-discussion] pmacct-contribs 20091125 released

2009-11-25 Thread Paolo Lucente
VERSION. 20091125 DESCRIPTION. pmacct is a set of network tools to gather, filter and tag IP traffic; it is able to store collected data either into a DB or a memory table. We see any monitoring, billing or accounting environment as a stack where data are picked from the network, get processed in

Re: [pmacct-discussion] peer-AS calculation problem

2009-11-25 Thread Paolo Lucente
Hi Zenon, On Wed, Nov 25, 2009 at 12:59:04PM +0200, Zenon Mousmoulas wrote: > I am not sure if this affects nfacctd or, perhaps, if it overrides this > information by looking up the next-hop (and perhaps also the dst peer AS) > in the BGP RIB? If i'm not mistaken you are not using the 'nfacctd

Re: [pmacct-discussion] A strange thing with one traffic direction

2009-11-24 Thread Paolo Lucente
Hi Peter, On Mon, Nov 23, 2009 at 10:45:38PM +0100, Peter Franzel wrote: > I thing RX Traffic is brilliant, but why is there such a big difference > between TX bytes?! Is there something I am going wrong or where is the fault? I would essentially suggest to go in a couple of directions; first

Re: [pmacct-discussion] MySQL options

2009-11-23 Thread Paolo Lucente
Hi Joel, On Sat, Nov 21, 2009 at 12:43:19PM +1100, Joel Roberts wrote: > I need to setup traffic accounting (in and out) for each IP address, and > then export that data to an EXTERNAL mysql database on a separate machine > accessible via IP address. How do I go about setting up pmacct to do this

Re: [pmacct-discussion] Layer 7 classification problem.

2009-11-23 Thread Paolo Lucente
Hi Mike, On Mon, Nov 23, 2009 at 02:00:04PM +0300, Mike Lykov wrote: > By the way, L7-filter have two types of filter: > "The first speed shown for a pattern in the tables below is the speed when > used in the kernel (with the old V8 regular expression library). The second > is the speed when u

Re: [pmacct-discussion] Layer 7 classification problem.

2009-11-20 Thread Paolo Lucente
Hi, On Fri, Nov 20, 2009 at 05:06:25PM +0530, Buddhike wrote: > I'm testing pmacct on my network, and pmacct runs on a box inbetween my LAN > switch and My ADSL router, and i'm using layer 7 classifires for classify > traffic. But when testing I observed that the traffic is not correctly > displa

Re: [pmacct-discussion] MySQL options

2009-11-20 Thread Paolo Lucente
Hi Joel, On Fri, Nov 20, 2009 at 05:24:29PM +1100, Joel Roberts wrote: > > I'm trying to install pmacct for the first time on XenServer. I have > installed mysql and can confirm the library files can be found: > > [r...@localhost pmacct-0.12.0rc3]# find / -name libmysql* > /usr/lib/mysql/libmysq

Re: [pmacct-discussion] Problem with aggregate_filter

2009-11-19 Thread Paolo Lucente
Hi, On Thu, Nov 19, 2009 at 04:00:33PM +0530, Buddhike wrote: > I've searched on google but didn't find any examples with mac adresses but > with src/dst networks. But i didn't seen any records saying that > aggrigate_filter can't be used with src mac and dst mac option. anyway I > tested it with

Re: [pmacct-discussion] invalid network mask?

2009-11-19 Thread Paolo Lucente
Hi Charlie, It appears you didn't put the table in the correct format first. Is it the case? If yes, to make that table pmacct-friendly you have to pre-process it as follows: cat bgptable | sed 's/\([0-9a-f:][0-9a-f\.\/:]*\).* \([0-9][0-9]*\)[ 0-9,{}]*$/\2,\1/' | uniq > networks.lst At least it

Re: [pmacct-discussion] create my own mysql table

2009-11-17 Thread Paolo Lucente
Hi, On Tue, Nov 17, 2009 at 02:42:15PM -0600, fedora fedora wrote: > I guess the thing I am confused about is netflow records are already > aggregated, generally only when a connection finishes or times out will a > router sends out the connection 'summary' to the netflow collector box. So > besi

Re: [pmacct-discussion] protocol classification don't detect http

2009-11-17 Thread Paolo Lucente
Hi Mike, On Tue, Nov 17, 2009 at 02:27:06PM +0300, Mike Lykov wrote: > > I would suggest a couple of checks: > > * see if HTTP traffic is reaped by some other classifier, but i guess > > you might have already checked that. > > if class_id = unknown, i think it's not this case. Yes, correct.

Re: [pmacct-discussion] create my own mysql table

2009-11-17 Thread Paolo Lucente
Hi, On Mon, Nov 16, 2009 at 04:45:57PM -0600, fedora fedora wrote: > DEBUG ( default/mysql ): INSERT INTO `test_1` (stamp_updated, > stamp_inserted, ip_src, ip_dst, as_src, as_dst, src_port, dst_port, > tcp_flags, ip_proto, packets, bytes, flows) VALUES > (FROM_UNIXTIME(1258410661), FROM_UNIXTIME

Re: [pmacct-discussion] create my own mysql table

2009-11-16 Thread Paolo Lucente
Hi, On Mon, Nov 16, 2009 at 11:58:14AM -0600, fedora fedora wrote: > I still see all flow records having the same number "4294967295" in my mysql > table, and debug does not seem to tell me why this happens. How do you mean? You see that number appearing in the debug? As writing to the database i

Re: [pmacct-discussion] protocol classification don't detect http

2009-11-16 Thread Paolo Lucente
Hi Mike, I see all of those signatures actually working by picking some sites randomly with wget. This is with 0.12.0rc3 but honestly speaking there has not been any major work related to the classification part for the past 3-4 years. I would suggest a couple of checks: * see if HTTP traffic is

Re: [pmacct-discussion] create my own mysql table

2009-11-13 Thread Paolo Lucente
Hi, On Fri, Nov 13, 2009 at 05:10:35PM -0600, fedora fedora wrote: > sorry, one more question, when preparing the aggregate, does the order of > the values matter? right now my aggregate is like the following No, doesn't matter. > > One more question, how can i get pmacct to show the flow number

Re: [pmacct-discussion] create my own mysql table

2009-11-13 Thread Paolo Lucente
Hi, On Fri, Nov 13, 2009 at 04:21:26PM -0600, fedora fedora wrote: > Thanks for the reply, i disable the daemon option and here are the error > messages, i cannot put src_ip and src_as together? > > INFO ( default/mysql ): 131070 bytes are available to address shared memory > segment; buffer size

Re: [pmacct-discussion] create my own mysql table

2009-11-13 Thread Paolo Lucente
Hi, It seems the daemon would like to complain about something but you send it to background (daemonize: true). Perhaps comment the daemonize line out and set debug to true (debug: true) and see what's the story. Put it like that, it could be a number of things. Cheers, Paolo On Fri, Nov 13, 200

Re: [pmacct-discussion] NAT question

2009-11-11 Thread Paolo Lucente
Hi JF, As Karl said, libpcap looks what's on the wire and pmacct doesn't get further up in the packet layering. You can always do a quick check by verifying what tcpdump sees. While on NAT & Linux, and perhaps not related to this specific issue: the "uacctd" daemon has been introduced in pmacct

<    5   6   7   8   9   10   11   12   13   14   >