Re: [pmacct-discussion] Ideas/pointers/hints wanted for using pmacct
On Thu 14 Jun 2007, Ruben Laban wrote: Hi Peter, On Thursday 14 June 2007, Peter Nixon wrote: On Tue 12 Jun 2007, Ruben Laban wrote: On Tuesday 12 June 2007, Ruben Laban wrote: I am still researching further optimizations. PF_RING sounds promising but since it requires a kernel rebuild I'd very much like to avoid that road. Being able to run stock SuSE kernels surely has my preference. Using libpcap-mmap seems to be way to go. I'll have to investigate how I can install this version of libpcap alongside the 'normal' libpcap and have only pmacctd use it (again to stay as close as possible to a clean SuSE install). Hi Ruben I maintain the pmacct packages for SUSE (and some other distros) and run pmacct myself in production on SLES10. I would be happy to assist you in building (if possible) libpcap-mmap packages for SUSE. Do you have any experience with using 2 versions of libpcap alongside eachother on the same machine by any chance? For me that would be a highly prefered situation. Ideally I would only have pmacctd use libpcap-mmap and other apps like tcpdump should just the shipped libpcap version. Then again, now I think of it, I'll most likely be performing tcpdump on high bandwidth interfaces as well, and thus would benefit from the use of libpcap-mmap in that case. I already did take a brief look at libpcap-mmap in order to see if it would be suitable for us. Though in its source package there seems to be very little references in the documentation to the actual mmap part of it. Though, either way, I am interested in giving libpcap-mmap a try. The spec file shipped with it sure does need some tweaking in order to make a proper x86_64 package from it (like hardcoded paths to /usr/lib/). If you have a more 'universal' and SLES(9) oriented spec file handy, I'd be more than glad to receive it. Hi Ruben If you can send me a copy of whatever you have right now, I will have a look at it tonight and stick it up on the opensuse build servers. Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] IP billing solution for datacenter
On Tue 08 May 2007, satish patel wrote: Dear All I have recently join this list this is my firest question for you i am working in datacenter so i need ip base accouting and billing for customer means i will assign client server on my datacenter switch and collect ip traffic from that port and genrate bill so can u suggest me which solftware i use for this task IPFlow or netflow i am confuse in this all software so please guid me ... Hi Satish Welcome to the pmacct list. As I mentioned to you on the freeradius list pmacct should be a much closer fit for the type of problem you are trying to solve. I suggest you read the documentation first, then ask specific questions. Pmacct and Postgresql/Mysql can do what you want without the need for further software.. Regards -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Classification
On Thu 16 Nov 2006 18:13, Sven Anderson wrote: Hi all, Paolo Lucente, 16.11.2006 02:03: Now, back to our syncronous approach: what was killing it was the excessive slowness the database and the concurrent arrival of packets at very high rates. Keeping the two entities (network and DB) asyncronous, segregated, gave a big relief and better performances (under normal conditions). Things that were requiring to be written down at the time were: a way to establish a kind of normal database behaviour in order to promptly react to excessive slowness, how the process can understand when to give up (pretty variable) and, of course, how to react, ie. what to do if data continue to accumulate. of course separation of metering and DB access is a good thing. But I think what Chris is suggesting is to have the write-to-DB jobs queued in one single writer thread rather than creating a new thread for each job. I also have the feeling this might be more effective, but it's just a feeling, I have no clue how much new resources an additional thread really needs. Yes. This is also something I have been suggesting for some time. Basically on a single CPU, single DISK database server ONE or at most TWO database threads will be fastest.. If you have more CPUs and DISKs then it MAY make sense to increase the number of DB threads depending on which DB system you use AND which locking scheme... Of course, if the data is coming too fast for the DB, it is to fast, no matter how the writer is organized. But if you use more resources for other stuff (like creating new threats) you might reach this limit earlier. This is a given... -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] nfacctd warnings
On Wed 15 Nov 2006 23:27, Peter Nixon wrote: Hi Guys I am currently getting the following ALOT in my logs. nfacctd[10115]: WARN: expecting flow '38601682' but received '38601711' collector=(null):2100 agent=x.x.x.x:0 Is it one of my ciscos playing up or a problem with nfacctd?? Interestingly when I nfacctd -L y.y.y.y on the primary ip on the interface instead of an alias ip (and change the ciscos to send to that ip obviously) collector no longer shows as null: WARN: expecting flow '35253' but received '35282' collector=y.y.y.y:2100 agent=x.x.x.x:0 It appears to be unrelated to the actual problem though as the error still occurs.. I have also rebooted the ciscos so it doesnt appear to be that.. Any ideas? I am running pmacct-0.11.1-3.1 Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Classification
On Mon 13 Nov 2006 08:57, Chris Wilson wrote: Hi Peter, Paolo and all, On Sat, 11 Nov 2006, Peter Nixon wrote: This is something I have discussed with Paolo on a number of occasions also. I see no practical reason why pmacct cannot store all flows with src and dest port and ip for a low speed link (10Mbit) yet 256K of traffic manages to overwhelm an opteron DB server. Ouch, I didn't know that the performance was that bad, thanks for the warning. Do you mean 256 kbits or kbytes? For what it's worth I have pmacct monitoring an 8mbit/384kbit link which is often full, on a Celeron 366, and it normally runs OK, but it has brought down that box on one occasion due to the number of database threads spawned, and the resulting system load. I mean 256kbps ( 30KB/sec). As I said this is when running a config like the following: aggregate[all]: proto,src_host,dst_host,src_port,dst_port Now, in my case the src side is a single class B so its not quite as bad as it looks either. This is an ambitious setting I know, but pmacct should be the one blowing up, NOT the database. (At least there should be a config option to force the load onto pmacct instead of the database) Actually to be honest I am using postgresql, and I have not actually seen it die, but it does get sooo many sockets open (500+) that things run so slowly that pmacct starts to give up on the db handles (And postgres obviously starts doing deadlock detection which makes it even slower).. it pmacct hit it with a single thread it would be committing is sub 100ms range instead of 10 sec range and the problem would basically disapear.. This is purely because of the WAY that pmacct access the database, not the amount of data (At least from the testing I have done), but it would require a change to the way DB access is handled to fix. Basically any query over threadlimit (which should default to 2 or 3) should be queued instead of dropped...This obviously requires someone write (or borrow) a decent queuing system for db access.. I don't think it's as hard as all that. The OS will happily queue UDP and pcap packets for us while we write to the database. So I'd just drop the separate thread entirely, and write to the database in the main thread. Under normal conditions this will not cause any problems at all, and it will degrade gracefully under load, unlike the current situation. OK. Sounds like its worth a try. I have since moved to a pretag config as the only way to keep my system reliable and get the data I want, but if someone writes the code I will test it with the old config. Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Classification
On Wed 08 Nov 2006 09:00, Chris Wilson wrote: --snip-- I'm still concerned about the performance of the MySQL plugin with threading, so I'm considering providing an option to disable the extra threads, and run updates synchronously. Interesting. What about having also a switch to have numbers-only tables, that is IP addresses, timestamps, class_id, mac addresses and protocol are all stored as integers? I don't see how that would help. It's basically just changing the constant multiplier cost. The problem I'm having is that when the database or the box is busy, pmacct starts spawning more and more threads that end up sleeping on the database. This eats resources and can lead to catastrophic failure (it has done it to me at least once). I would rather delay writing to the database by having it done synchronously, to limit the damage that it can do to the rest of the box. This is something I have discussed with Paolo on a number of occasions also. I see no practical reason why pmacct cannot store all flows with src and dest port and ip for a low speed link (10Mbit) yet 256K of traffic manages to overwhelm an opteron DB server. This is purely because of the WAY that pmacct access the database, not the amount of data (At least from the testing I have done), but it would require a change to the way DB access is handled to fix. Basically any query over threadlimit (which should default to 2 or 3) should be queued instead of dropped...This obviously requires someone write (or borrow) a decent queuing system for db access.. Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Large number of threads
On Wed 18 Oct 2006 21:50, Chris Wilson wrote: Hi Paolo, On Wed, 18 Oct 2006, Paolo Lucente wrote: If you see, such processes are not taking CPU, they are just laying out there. This is because they are all on their LOCK, waiting for the green light. LOCK serializes things: selects one of those processes, allows it to do its job and terminate, then selects another one, etc. I think you're right that they weren't taking CPU, but they do take memory. And creating increasing numbers of them is unlikely to help the box come back to life after such an incident. You should avoid such a queue to come up. It might be related to the low specs box. But it might also be that you are not aggregating things that much (are you?). Take a look to the discussion happened on the list just earlier this month about database, performances, etc. Then, using MySQL, you can also take a look to the following configuration directives in CONFIG-KEYS: sql_dont_try_update and sql_multi_values. They usually help. It seems to be partly related to this. The problem is that I do actually want to log a lot of data, and I want to see how much I can get away with on this old box. Preferably without killing it, because it's also our firewall and LDAP server. And it seems to be doing fine under normal conditions. But when the box does get overloaded, pmacct doesn't degrade gracefully. I have also discussed the same problem with Paolo. Its a problem that needs solving but needs work. -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Wellcome message
On Fri 21 Jul 2006 09:33, Jaime Nebrera wrote: Hi all, This is my first email to the list as I just discovered what seems a wonderful tool. Welcome to the party! I have been reading the site deepelly but as I'm not a tech guy (more of a management guy) well, have not looked into testing yet. From what I understand, pmacct can work as a network sniffer and store its data both in RAM tables or SQL. But I dont see if it can work as a more standard NetFlow probe and send the data to an external system to store and process it. This functionality was added but Paolo yesterday (Both sflow and netflow). You need to use pmacct 0.11.0rc1 (or higher). Secondly, I see its compatible with all NetFlow and sFlow versions. Thats impressive as the tool we are currently using (flow tools) only supports NFv5. Does this mean it can work as a collector (server) for NetFlow enabled devices? Does this with the same software or you have a different daemon for this? Yes. Same software, different daemon (called nfacctd). If you use my SUSE packages they install startup scripts and example config files for you all ready to go :-) One of the things that has really impressed me is that you already have some state machine and pattern matching code to detect some confiltive apps like FTP or even better, SIP, P2P, etc. Are this features available only in the probe software or also in the server software? These should work in both, although I dont use them myself. And last, is the probe software small enough to be run in a small device? (not exactly an embeddded device but a not hugelly powerfull platform). Of course, the better the hardware, the more traffic you can analyse but still interesting to do so with smaller resources :) We are currently using nprobe from ntop with very good success but lacks some of the features pmacct seems to have. As long as you don't run an SQL server on your small device it should work fine :-) Well, this is it. If I have the right feeling we could end up using this tool very soon and we will devote some people to help develop the project. We have some experience in this and can give some ideas that we have used for a flowtools based platform. Also, some students do their diploma jobs with us, so we could just put some of them to work in improving this. Veryt thankful in advance. GOOD JOB !!! 99% of the work is done by someone we like to call SuperPaolo :-) Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc pgpaoISadl9eu.pgp Description: PGP signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] pmacct + peer to peer traffic
On Tue 18 Jul 2006 16:40, Gregory Machin wrote: Hi. Can I use pmacct to monitor peer to peer traffic that is passing trough my firewall .. ? It depends on what is your firewall is? If it runs linux then yes. If not, but it supports sflow or netflow, then yes, otherwise no.. Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc pgpWhhEdPrMZP.pgp Description: PGP signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Removing Un-needed database fields
On Wed 12 Jul 2006 01:36, Jon Hall wrote: Hi, Is there a way to remove the un-needed fields in a mysql database? I am not using at least half of them and wanted to remove that added storage/complexity from the equation. If I simply delete the fields from the schemas, my inserts and updates from nfacctd error. You may delete rows from the schema as long as you don't use those fields in an aggregate clause and if you include sql_optimize_clauses: true in your config file. Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc pgpcpTbYkjf2L.pgp Description: PGP signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Removing Un-needed database fields
[resend: corrected wrong info] On Wed 12 Jul 2006 01:36, Jon Hall wrote: Hi, Is there a way to remove the un-needed fields in a mysql database? I am not using at least half of them and wanted to remove that added storage/complexity from the equation. If I simply delete the fields from the schemas, my inserts and updates from nfacctd error. You may delete columns from the schema as long as you don't use those fields in an aggregate clause and if you include sql_optimize_clauses: true in your config file. Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc pgpomYoXdcr2p.pgp Description: PGP signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] sfacctd/nfacctd security
On Wed 05 Jul 2006 01:20, Paolo Lucente wrote: Hi Peter, nfacctd_allow_file and sfacctd_allow_file config directives are aimed precisely to this. The idea behind them is the same as for hosts.allow. They are both listed in CONFIG-KEYS. Ahh. Thanks. I missed them when I scanned the documentation. Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc pgpGZpG7s5Ka3.pgp Description: PGP signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] pmacctd - netflow/sflow export
On Wed 05 Jul 2006 10:13, Ivan A. Beveridge wrote: Hi all, On 05/07/2006 00:07, Paolo Lucente wrote: while the idea of integrating a kind of sFlow/NetFlow probe has been already considered (i remember some thoughts recently exchanged with Sven Anderson about this), i'm somewhat not fully convinced. In a first instance it will take time as it's absolutely not trivial; this has been correctly noted by Ivan and somewhat answers to the question from Peter. But my point is the following: out there we already have a range of NetFlow probes, ie. softflowd, nprobe, fprobe to name a few; don't actually know if exists any for sFlow (out of curiosity, Ivan, can you confirm this ?). There is also choise for replicators, ie. UDP samplicator, flow-fanout - which is part of flow-tools. To the best of my knowledge the only open-source software that outputs sflow data is ntop (I'm not sure whether the slightly commercialised nprobe also does). The only piece of software that I know that converts sflow - netflow is inmon's sflowtool. sflowtool will also do fanout (both sflow - sflow and sflow - netflow). My requirement is to take in sflow data, filter it on MAC address and output multiple streams based on the filters (1 stream showing in and out for a particular MAC). This is similar to taking a netflow stream, splitting into per-port netflow and then exporting these netflow streams (except more complicated). Basically what I need must act as a (sflow) collector, filter [I think pmacct will do both of these] and then act as an agent/probe to convert that filtered data back to sflow/netflow ... and I can't think that any tool combination will do this. Yes. That was the reason for suggesting a netflow/sflow backend plugin for pmacct I *know* my requirement is non-trivial, and I suspect I'm in a rather unique position in wanting/needing this (with the possible exception of other IXes) ... having customers that want to parse their own sflow/netflow data from our network kit. Non-trivial yes.. However it is something I think I will need to do also which is the reason I asked about it :-) I am also having some technical problems with nprobe that don't exist with pmacct... Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc pgpLkpMP04OnT.pgp Description: PGP signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] RMON
Hi Guys I want to collect data from a group of switches that only support RMON and not sFlow. Does anyone know of any decent open source RMON packages that can be integrated with pmacct? Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc pgpkTSvWZNL0m.pgp Description: PGP signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] default sql_cache_entries
What is the default?? The documentation doesn't say.. KEY:[ sql_cache_entries | print_cache_entries ] DESC: SQL plugins and print plugin sport a Plugin Memory Cache (PMC) to accumulate counters for each aggregate; it has been thought to limit the interaction with the backend. Data is pushed (to stdout or SQL database) each '[sql| print]_refresh_time'; this directive sets the number of PMC buckets. The default value is suitable for most common scenarios, however when facing with large-scale networks, it's higly recommended to carefully tune this parameter to improve performances. Use a prime number of buckets. -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc pgpFfToWuwNwV.pgp Description: PGP signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] SQL Performance
On Mon 05 Jun 2006 17:39, Wim Kerkhoff wrote: Peter Nixon wrote: As you can see the machine is not waiting on disk, but rather on the locks. Before I go about trying to optimise this any further, is there something basic I have wrong that can speed this up? The configuration looks fine. It seems like there's no indexes or something, causing the UPDATEs to take forever. Yep. I am optimizing the indexes as we speak. I now have the following indexes on both tables, and it has made a significant difference to system load. acct_combo btree (stamp_inserted, vlan, ip_src, ip_dst, port_src, port_dst) acct_datetime btree (stamp_inserted) acct_dst btree (port_dst, ip_dst) acct_src btree (port_src, ip_src) I have also deleted all unnecessary fields from the schemas (Mac Address etc). It could also be that there are a lot of counters. Calculate 500 hosts, 2 directions, 3 protocols (ICMP/TCP/UDP), server ports (10 ish), client ports (hundreds), and you could easily be trying to update hundreds of thousands of records every 10 minutes. Sure. I am aware of the scope of the data. I was just wondering if there were any obvious nfacctd options I should have turned on that I didn't. As I am monitoring a GPRS network and 99% of my clients are HP Ipaqs the port range is fairly limited by the type of applications that can run on an Ipaq :-) Turn on debug, redirect the output of nfacctd to a file, and watch to see how many updates are actually being done. Thanks -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc pgp8oJ8pM5Oyt.pgp Description: PGP signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] PostgreSQL performance
On Wed 31 May 2006 19:23, Sven Anderson wrote: Hi all, I want to share with you an interesting experience I just had. I did the following SELECT: SELECT ip_dst,SUM(bytes),SUM(packets),SUM(flows) FROM tcom_v5_20060530 WHERE stamp_inserted='2006-05-30 00:00:00' AND stamp_inserted'2006-05-30 02:00:00' AND port_src='53' AND port_dst='53' AND ip_src='xxx.xxx.xxx.xxx' GROUP BY ip_dst ORDER BY SUM(bytes) DESC LIMIT '10'; which seemed to never end (30 minutes). I had only two indexes on this table: tcom_v5_20060530_idx btree (ip_src, ip_dst, stamp_inserted) tcom_v5_20060530_stamp_idx btree (stamp_inserted) After dropping the tcom_v5_20060530_idx, the query was answered in 20 seconds. When is the last time you did a VACUUM ANALYZE? -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc pgpUrzZXNexl7.pgp Description: PGP signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Comparing nfacctd and pmacctd
Hi Nikola I already have a configuration almost identical to yours. As I mentioned below, I am happily getting data from the external interface also however the flows are all hidden by the single nat overload which means I have no way to associate them with the traffic on the internal interface. Does anyone have a way to resolve this? I figure that there must be a way to get around this problem by using a loopback interface but as yet I haven't figured out the correct configuration. Cheers Peter On Tue 23 May 2006 10:38, Nickola Kolev wrote: Hello, Peter, In order to see the traffic in both directions, you have to enable cache-flow on both interfaces - incoming and outgoing for your network. I'm using a Cisco to gather billing and traffic accounting statistics with netflow, but I'm not using NAT. Firstly, you have to enable it: ip flow-cache timeout active 2 This enables a 2 minute active timeout for flows. Then, on each of your interfaces, f.e. : interface GigabitEthernet0/1 ip route-cache flow interface GigabitEthernet0/2 ip route-cache flow And finally to send the netflow data to a nfacctd, or any other NetFlow accounting software: ip flow-export version 5 origin-as ip flow-export destination 192.168.1.2 Hope this helps. On Mon, 22 May 2006 23:35:08 +0300 Peter Nixon [EMAIL PROTECTED] wrote: Hi List As a relative newbie to netflow can someone confirm for me whether or not netflow records from a single interface of a cisco router contain information about packets in BOTH directions or only one? I am attempting to replace a linux box acting as a router running pmacctd with a cisco router running netflow sending records to nfacctd. The tricky bit is that I am running NAT on the external interface of the router with a private IP block behind it and I need to see data on inbound AND outbound traffic. With pmacctd on a linux box I can see data in both directions on the internal interface(s) but I don't appear to be getting it with the cisco. If in enable ip route-cache flow on the external interface I see all the flows related to the external NAT IP which is useless as I need to match it to the hosts behind. I have also tried to setup a looback interface, with netflow enabled on it, and route all traffic via it, but I dont seem to be receiving any flow records from it. Can anyone help? -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc pgpPTveD4h382.pgp Description: PGP signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] Comparing nfacctd and pmacctd
Hi List As a relative newbie to netflow can someone confirm for me whether or not netflow records from a single interface of a cisco router contain information about packets in BOTH directions or only one? I am attempting to replace a linux box acting as a router running pmacctd with a cisco router running netflow sending records to nfacctd. The tricky bit is that I am running NAT on the external interface of the router with a private IP block behind it and I need to see data on inbound AND outbound traffic. With pmacctd on a linux box I can see data in both directions on the internal interface(s) but I don't appear to be getting it with the cisco. If in enable ip route-cache flow on the external interface I see all the flows related to the external NAT IP which is useless as I need to match it to the hosts behind. I have also tried to setup a looback interface, with netflow enabled on it, and route all traffic via it, but I dont seem to be receiving any flow records from it. Can anyone help? -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc pgpbVQylhZUAt.pgp Description: PGP signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] Putting inbound and outbound packets into same table
Hi List I am currently using the following config: aggregate[db_in]: proto,src_host,dst_host,src_port,dst_port aggregate[db_out]: proto,src_host,dst_host,src_port,dst_port aggregate_filter[db_in]: dst net 10.11.0.0/16 aggregate_filter[db_out]: src net 10.11.0.0/16 plugins: pgsql[db_in], pgsql[db_out] sql_table[db_in]: acctin sql_table[db_out]: acctout This gives me 2 tables which is can then do an SQL JOIN (where src_host=dst_host and dst_host=src_host and src_port=dst_port and dst_port=src_port) to get complete data on each actual connection. What I would like to be able to do is to reverse way the data is inserted in the table for one of the 2 aggregate lines, and then put the data in the SAME sql table. The config could look something like: aggregate[db_in]: proto,src_host,dst_host,src_port,dst_port aggregate[db_out]: proto,src_host_reverse,dst_host_reverse,src_port_reverse,dst_port_reverse aggregate_filter[db_in]: dst net 10.11.0.0/16 aggregate_filter[db_out]: src net 10.11.0.0/16 plugins: pgsql[db_in], pgsql[db_out] sql_table[db_in]: acct sql_table[db_out]: acct Therefor db_out would put the put dst_host data in the src_host column in the db, dst_port data in the src_port etc. For my purposes the number of bytes for in and out could be summed (which would fit with this system just fine but it could also be usefull to have in_bytes and out_bytes in separate columns... Does anyone have any comments on this? Is there an easy way to do this currently? Regards -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc pgpBv5tksjySg.pgp Description: PGP signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Putting inbound and outbound packets into same table
On Fri 19 May 2006 14:09, Paolo Lucente wrote: Hi Peter, good point (again). pmacct actually lacks of such thing. But it could be good idea to implement it. The method will need some extra cares in order to make things work smoothly: a) writes need to be interleaved in order to avoid one plugin to lock out the other while racing for the same SQL table; b) with no tags, UPDATEs would become unavoidable. Here, the almost forgotten post_tag directive might be used to tag things coming from different plugins, thus avoiding online UPDATEs. Rather than creating a whole new set of reversed primitives wouldn't it be better to create a new config directive ? In the end, the reverse concept would apply only to direction-specific primitives (ie. TCP/UDP ports and MAC/IP addresses) and one should not really want to reverse them singularly (as this would produce spurious tuples). Look at the following example: === ... aggregate[db_in]: tag,proto,src_host,dst_host,src_port,dst_port aggregate[db_out]: tag,proto,src_host,dst_host,src_port,dst_port aggregate_filter[db_in]: dst net 10.11.0.0/16 aggregate_filter[db_out]: src net 10.11.0.0/16 aggregate_reverse[db_out]: true plugins: pgsql[db_in], pgsql[db_out] sql_table[db_in]: acct sql_table[db_out]: acct post_tag[db_in]: 1 post_tag[db_out]: 2 Thanks for the quick reply. In-fact this was my alternate suggestion that I was too lazy to write. I favoured the other option as it provides a little more flexibility that someone somewhere might want to use. However, after thinking about it again, I believe you are correct that no-one would actually want to reverse the IPs, but not the ports etc so I think your option is the sanest way to do it. The only question that remains, is how to handle bytes? Ideally I think the schema should be changed/extended to have an bytesin and bytesout column.. What do you think? -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc pgpjwD9kXM9FL.pgp Description: PGP signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Putting inbound and outbound packets into same table
On Fri 19 May 2006 15:18, Paolo Lucente wrote: Hi Peter, On Fri, May 19, 2006 at 02:38:37PM +0300, Peter Nixon wrote: The only question that remains, is how to handle bytes? Ideally I think the schema should be changed/extended to have an bytesin and bytesout column.. What do you think? I think that while it would be the best ever solution (with no doubts), it will also break many things. And backward compatibility is great priority. The actually viable but intuitively less economical solution is to mantain the 2 tuples segregated, by assigning them different tags. This should at least ease the job of getting out in/out/total traffic and definitely avoid the ugly JOINs. So do you mean having the existing bytes column stay which in the case where you have a reversed module it would be the sum of both in and out, as well as adding a bytesin and bytesout?? Or did I not understand correctly what you meant? -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc pgp8Ji57Tbely.pgp Description: PGP signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] pmacct 0.10.2 released !
On Wed 17 May 2006 14:33, Paolo Lucente wrote: VERSION. 0.10.2 DESCRIPTION. pmacct is a small set of passive network monitoring tools to measure, account, classify and aggregate IPv4 and IPv6 traffic; a pluggable and flexible architecture allows to store the collected traffic data into memory tables or SQL (MySQL, SQLite, PostgreSQL) databases. pmacct supports fully customizable historical data breakdown, flow sampling, filtering and tagging, recovery actions, and triggers. Libpcap, sFlow v2/v4/v5 and NetFlow v1/v5/v7/v8/v9 are supported, both unicast and multicast. Also, a client program makes it easy to export data to tools like RRDtool, GNUPlot, Net-SNMP, MRTG, and Cacti. HOMEPAGE. http://www.pmacct.net/ DOWNLOAD. http://www.pmacct.net/pmacct-0.10.2.tar.gz Those who run SUSE Linux can download 0.10.2 prebuilt rpms (for 10.0 and 10.1 in both i586 and x86_64 format) from: ftp://ftp.suntel.com.tr/pub/pmacct/suse/ Thanks goes to OpenSUSE for providing access to their Build service. Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc pgpZtD4RsO2hH.pgp Description: PGP signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] beginning
On Fri 12 May 2006 18:17, Paolo Lucente wrote: Hey Peter, On Thu, May 11, 2006 at 09:13:09AM +0300, Peter Nixon wrote: I would love to see an SNMP agent however so that the in memory tables could be queried remotely via SNMP. This would allow trivial integration with any number of SNMP graphing tools :-) That's a good point. Seems like the smooth way to this is Net-SNMP. They have an API and support AgentX protocol (ie. a master agent, snmpd, communicates with somewhat registered sub-agents - providing particular SNMP trees - via such protocol). That would be the state of the art solution (and requires a fair amount of time); This would be the full solution... what is your opinion about a 2-stages approach either embedding pmacct client queries in the SNMP tree itself (ie. SNMP part of Q9, FAQS doc) or something scriptic (ie. to emulate in/out interface counters) ? Yep. This is already possible as net-snmp has the capability to call external scripts and return the results. I personally use this to trigger an sql query to the backend Postgresql database (not pmacct directly) but the effect is the same. Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc pgpB9BfYBqlNe.pgp Description: PGP signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] beginning
--snip-- The scriptic way is, of course, available even going the SQL way aswell. As today pmacct is not featuring a RRD plugin, so there isn't support to write directly into RRD files. This bases on my very personal assumption (and if anyone reading has different ideas and/or proposals ... please let your comments to follow-up) that the collector should not care about creating/rotating files/directory trees (as this is likely to be prone to consistent delays and endless debates about the best-ever file hierarchy) and leave the work to external scripts/works. -snip- I am in agreement with this. I would love to see an SNMP agent however so that the in memory tables could be queried remotely via SNMP. This would allow trivial integration with any number of SNMP graphing tools :-) -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc pgpAtiIPdfszn.pgp Description: PGP signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] pmacct problems on x86_64?
On Wed 19 Apr 2006 16:52, Paolo Lucente wrote: Hi Peter, i was wondering whether it's something related with VLANs and the aggregate_filter directives: traffic seen on tunnel0 is tagged (and doesn't match the filter -- pcap filter need to match the vlan layer, ie. vlan and src net ...) while the one on the eth0 isn't (thus, matching it). Do you see this possible ? And what does it happen if you remove both aggregate_filter directives on the x86_64 box ? Hi Paolo It does appear to be related to filters not working correctly when applied to a tunnel interface. The tunnels we use are GRE and pmacctd correctly summarises the data and writes it to Postgres if no filters are used, but no data makes it to the database if a filter is applied which should generate a subset of the same data. We are not using VLANs at present. The traffic I am trying to account for comes into an ethernet interface encapsulated in GRE, gets NATed and leaves again via the same ethernet interface, so I really need to account for the traffic before it gets NATed on the tunnel interface otherwise I can't see the source hosts. -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc pgpWeBuzA8QUZ.pgp Description: PGP signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
