Re: [pmacct-discussion] beginning
Hi,So I worked a little with pmacct an had a few results.here is the config file I wrote. debug: false daemonize: false nfacctd_port: nfacctd_allow_file: /home/cedric/pmacct-0.10.1/maconfig/routeurs.lst ports_file: /home/cedric/pmacct-0.10.1/maconfig/ports.lst plugin_buffer_size: 2048 plugin_pipe_size: 2048000 !networks_file: /home/cedric/pmacct-0.10.1/maconfig/reseaux.lst plugins: memory[in], memory[out] aggregate[in]: src_host,dst_host aggregate[out]: src_host,dst_host aggregate_filter[in]: dst net x.x.0.0/16 aggregate_filter[out]: src net x.x.0.0/16 imt_path[in]: /tmp/in.pipe imt_path[out]: /tmp/out.pipe I use it with this command : nfacctd -f nfacct.conf It works fine but a few sec after running it, these messages appears : WARN ( in/memory ): Unable to allocate more memory pools, clear stats manually! WARN ( out/memory ): Unable to allocate more memory pools, clear stats manually! using this command : pmacct -s -p /tmp/out.pipe , I can see that nfacctd collects more and more data in spite of this error. SRC_IP DST_IP PACKETS BYTES x.x.x.x 66.249.85.104 84 16970 x.x.x.x 67.43.10.51 23 2166 x.x.x.x 193.62.157.66 8 596 Entries count increase and pass 32900. So I have datas (yeh!) :) . I would like now create a graph showing the incoming and outgoing trafic of my network. I first tried to wiew with pmacct the network use. pmacct -N x.x.0.0 -c src_net -p /tmp/out.pipe gives me 0. But it works fine for a computer : pmacct -N x.x.131.29 -c dst_host -p /tmp/in.pipe returns 944470 bytes So is there an easy way to see stats about my network (and subnetworks). lisening your advices, I installed cacti (not so easy) and created a graph using pedro's method. It seams to work fine. Thanks pedro :) My problem is now that i need informations about many sub-networks and machines (dns, dhcp ...). Is there a way to script graphs creation (from a conf file or oter) ? Thanks ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] PostgreSQL performance
Hi Wim,
Wim Kerkhoff, 04.05.2006 06:09:
Disabling fsync() does improve performance. I did this once for a Pmacct
database, but was bitten horribly after the server was rebooted a couple
of times without being shutdown properly (power/UPS failure etc). So
don't do that.
You mean: all the data was gone, not only the recent one?
- enable optimal indexes based on queries being used by pmacctd to
update the tables
I disabled updates at all. But anyway, with the default standard primary
key index over all the update fields there was no chance. Now I have
already over 18 million rows with these indexes:
Indexes:
acct_v5_idx btree (ip_src, ip_dst, stamp_inserted)
stamp_inserted_idx btree (stamp_inserted)
and all the queries are fast enough as long as a time window is selected,
which is not too big. But even adding a hash-table for ip_src was creating
trouble. So my problem is definitely correlated with too much indexes.
- remove unused indexes
I would add or not so important indexes. ;-)
- run all SELECT UPDATE under explain analyze ... to see if they
are actually hitting the indexes
Good hint.
BTW.: What is the best indexing to make a select with WHERE
stamp_inserted=2006-04-21 20:17:55 AND stamp_inserted2006-04-21
21:17:55 fast? I guess a btree index, but maybe there's something better?
This question is still open. No DBMS gurus here? ;-)
Are you actually doing queries like this, or is this just an example?
Yes, I'm doing grouped sums in certain time slots, where I use queries
like this:
SELECT src_ip,sum(bytes) FROM acct_v5 GROUP BY src_ip ORDER BY sum(bytes)
DESC WHERE stamp_inserted=2006-04-21 20:17:55 AND
stamp_inserted2006-04-21 21:17:55;
What you could do is an an index based on a group clause. Eg, group by
date_trunc('hour', stamp_inserted). Then it will be very fast to find
the records from the hour. Then a sequential scan for records in those
selected hours should be pretty quick.
Will it also intelligently use this index when doing WHERE
stamp_inserted=2006-04-21 20:17:55 AND stamp_inserted2006-04-21
21:17:55, so that it looks only in the two touched hours?
I have counter tables with 7+ million records in Pg, and no complaints
at all from the people hitting web apps that are hitting them.
Therefore I created an acct_monthly,
acct_daily, and acct_hourly tables. acct_monthly is blazing fast because
it has barely any records in it. acct_daily has much less records then
acct_hourly.
Sounds like the different resolution time lines that are used to be
created with RRDTool.
Cheers,
Sven
--
Sven Anderson
Institute for Informatics - http://www.ifi.informatik.uni-goettingen.de
Georg-August-Universitaet Goettingen
Lotzestr. 16-18, 37083 Goettingen, Germany
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] dynamic table names
Hi Jamie, Jamie Wilkinson, 03.05.2006 02:52: It's outside the transaction, so it doesn't blow up horridly. It is annoying to see 3600 messages in your logs, sure. well it's 1440 per day for me. That's not so nice, I will solve it with a cronjob. Cheers, Sven -- Sven Anderson Institute for Informatics - http://www.ifi.informatik.uni-goettingen.de Georg-August-Universitaet Goettingen Lotzestr. 16-18, 37083 Goettingen, Germany ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] beginning
Hey Cedric, On Fri, May 12, 2006 at 12:19:44PM +0200, Cédric Delaunay wrote: aggregate[in]: src_host,dst_host aggregate[out]: src_host,dst_host I would correct the above few lines in: aggregate[in]: dst_host aggregate[out]: src_host This is very likely to stop the error message you are receiving (which means the memory table has reached its upper bound - play with imt_mem_pools_size, imt_mem_pools_number configuration directives to enlarge/tune it, if required). Furthermore, it's a more safe setting, as you should not be interested by the remote peers your local boxes connect to, isn't it ? For some more bits about this argument, divert your eyes to Q5/Q6 points of FAQS. I first tried to wiew with pmacct the network use. pmacct -N x.x.0.0 -c src_net -p /tmp/out.pipe gives me 0. But it works fine for a computer : pmacct -N x.x.131.29 -c dst_host -p /tmp/in.pipe returns 944470 bytes If you wish to have stats for your subnet, then you need to instruct a couple more memory plugins: the client tool is unable to perform any sub-aggregation of data. Perhaps you wish to adjust your configuration as follows: === nfacctd.conf === ... ! plugins: memory[in_host], memory[out_host], memory[in_net], memory[out_net] ! aggregate[in_host]: dst_host aggregate_filter[in_host]: dst net x.x.0.0/16 imt_path[in_host]: /tmp/in_host.pipe ! aggregate[out_host]: src_host aggregate_filter[out_host]: src net x.x.0.0/16 imt_path[out_host]: /tmp/out_host.pipe ! aggregate[in_net]: dst_net aggregate_filter[in_net]: dst net x.x.0.0/16 imt_path[in_net]: /tmp/in_net.pipe networks_file[in_net]: /path/to/networks.lst ! aggregate[out_net]: src_net aggregate_filter[out_net]: src net x.x.0.0/16 imt_path[out_net]: /tmp/out_net.pipe networks_file[out_net]: /path/to/networks.lst ! ... === === networks.lst === x.y.z.a/24 x.y.z.b/24 x.y.z.c/24 x.y.z.d/23 ... === This way, in_host/out_host will hold per-host traffic data; in_net/out_net will hold per-subnet traffic data. Pay few extra bytes for an overral resource savy solution (often sub/re-aggregations are really crunchy). My problem is now that i need informations about many sub-networks and machines (dns, dhcp ...). Is there a way to script graphs creation (from a conf file or oter) ? AFAIK, there isn't anything pre-cooked. Don't know whether on the Cacti side there is anything generic to load configuration in batches. It *should* be easy stuff in the case Cacti saves profiles the SQL/XML way (which could be the case). Perhaps having some useful pointer about this, i may help .. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] PostgreSQL performance
Hi Paolo, Paolo Lucente, 01.05.2006 17:20: it seems to be known, that a default MySQL performs a lot better than a default PostgreSQL. One difference is, that PostgreSQL is doing an COMMIT Not to open a possible flame, but i fully agree with this view. However, we massively deal with network objects (IP and MAC addresses, network subnets) which are handled properly only by PostgreSQL. Do you see chances that this can re-open the challenge on this scenario ? what do you mean with properly? Basically these types are just numbers. And these are handled properly by mysql. Sure, doesn't look so nice in a raw table. The problem in the MySQL scheme for me are the strings, especially in ip_proto. For me it looks like PostgreSQL is cool thing if you have complex table relations. But we have plain flat single tables. Maybe even MySQL is an overhead here. ;-) Did you closer look at flowd (http://www.mindrot.org/flowd.html)? It has it's own binary file format for flow data. Maybe this is the right direction? My dream is anyway a backend-independent API for accessing the data, where you don't have to know, where and in which format the data is stored. ;-) Cheers, Sven -- Sven Anderson Institute for Informatics - http://www.ifi.informatik.uni-goettingen.de Georg-August-Universitaet Goettingen Lotzestr. 16-18, 37083 Goettingen, Germany ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] beginning
Hey Peter, On Thu, May 11, 2006 at 09:13:09AM +0300, Peter Nixon wrote: I would love to see an SNMP agent however so that the in memory tables could be queried remotely via SNMP. This would allow trivial integration with any number of SNMP graphing tools :-) That's a good point. Seems like the smooth way to this is Net-SNMP. They have an API and support AgentX protocol (ie. a master agent, snmpd, communicates with somewhat registered sub-agents - providing particular SNMP trees - via such protocol). That would be the state of the art solution (and requires a fair amount of time); what is your opinion about a 2-stages approach either embedding pmacct client queries in the SNMP tree itself (ie. SNMP part of Q9, FAQS doc) or something scriptic (ie. to emulate in/out interface counters) ? Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] beginning
On Fri 12 May 2006 18:17, Paolo Lucente wrote: Hey Peter, On Thu, May 11, 2006 at 09:13:09AM +0300, Peter Nixon wrote: I would love to see an SNMP agent however so that the in memory tables could be queried remotely via SNMP. This would allow trivial integration with any number of SNMP graphing tools :-) That's a good point. Seems like the smooth way to this is Net-SNMP. They have an API and support AgentX protocol (ie. a master agent, snmpd, communicates with somewhat registered sub-agents - providing particular SNMP trees - via such protocol). That would be the state of the art solution (and requires a fair amount of time); This would be the full solution... what is your opinion about a 2-stages approach either embedding pmacct client queries in the SNMP tree itself (ie. SNMP part of Q9, FAQS doc) or something scriptic (ie. to emulate in/out interface counters) ? Yep. This is already possible as net-snmp has the capability to call external scripts and return the results. I personally use this to trigger an sql query to the backend Postgresql database (not pmacct directly) but the effect is the same. Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc pgpB9BfYBqlNe.pgp Description: PGP signature ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
