Hi Guys, sorry to join this - interesting, despite Peter's exagerations :-) - thread a bit late, i'm having some terribly busy days. I want just to put a comment to the following lines:
On Mon, Nov 13, 2006 at 09:57:09AM +0300, Chris Wilson wrote: > I don't think it's as hard as all that. The OS will happily queue UDP and > pcap packets for us while we write to the database. So I'd just drop the > separate thread entirely, and write to the database in the main thread. > Under normal conditions this will not cause any problems at all, and it > will degrade gracefully under load, unlike the current situation. > > Paolo, please could you help us to do something about this? It appears to > be a real problem with pmacct that affects several users. In the very early days, pmacct had such a solution: just a main SQL process that was doing everything, receiving packets from the Core Process, getting in touch with the database and some extra stuff. The overral result was poor. In first instance, we should agree on the fact that the rate at which data are captured from the network greatly outperforms the rate at which even the best RDBMS engine is able to handle them (and this is why i insist on a fact that should be common understanding: the less you aggregate, the less you can insert in your database; trepassing a certain limit, you simply start loosing data). Indeed, this doesn't apply to our home (DSL/Cable) connections. Now, back to our syncronous approach: what was killing it was the excessive slowness the database and the concurrent arrival of packets at very high rates. Keeping the two entities (network and DB) asyncronous, segregated, gave a big relief and better performances (under normal conditions). Things that were requiring to be written down at the time were: a way to establish a kind of normal database behaviour in order to promptly react to excessive slowness, how the process can understand when to give up (pretty variable) and, of course, how to react, ie. what to do if data continue to accumulate. An idea could be to let pmacct become polite by imposing a (configurable) maximum number of concurrent writers to the database (set by default to a widely accepted number, say, 10) - and stop relying on the system for this. Any extra writer will be dropped and data will be lost. We might also think to a second value, ie. when to start accepting new writers again (say, 2-3), which in some cases could allow for a quicker recovery. Cheers, Paolo _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
