Hi Chris and all, On Thu, Nov 16, 2006 at 08:49:17AM +0300, Chris Wilson wrote:
> What about the OS queues for packets? Were they not effective? Was pmacct > doing a database write for each packet? (I can see how that would kill it > very quickly). My basic idea is to rely on the OS the less possible to relief pressure on it. Avoiding involvement of kernel as much as possible gives better chances to scale. Ie., this is why we are using mmap() - instead of SVr4's shm - to build circular buffers (= queues) between Core Process and each Plugin. And, no, the sql_refresh_time mechanism was in place but everything was happening in the very same process - which was sharing itself between getting new packets from the Core Process and dealing with the database. > I would be much happier if there was a single thread that would flush data > to the database at (up to) the maximum rate that the database could > support. I don't think there is any benefit to multiple threads with > MySQL; you will not actually be able to insert any more rows into the same > table that way, at least with MyISAM tables. > > A single thread doesn't seem like it would be too hard to implement, > perhaps as a configurable option. The thread would just start when pmacctd > starts, sleep until the refresh time expires, and then flush all dirty > records from memory to the database, then sleep again if necessary, i.e. > unless the inserts took more than the sql_refresh_time. > > This way, we could get the maximum performance from the database (at least > MySQL) without interfering with packet capture at all. If the database > can't keep up with the sql_refresh_time, you simply get fewer > updates/inserts, no data loss except temporal resolution. > > What do you think about that idea? Now your idea is much clear to my eyes. And i'm fully positive on its regards, agreeing with comments dropped before me by both Sven and Peter. Next weekend i'll gather some time to build up a SVN repository - astonishment and surprise - for pmacct where i will include the freshest code snapshot. I hope this is appreciated and will ease the process of contributing code to the project. Chris, do you have any plans to write down some code regarding your idea? If that's the case, i can help you a bit in planning/integrating it. At a very rough analysis it would require a basic thread implementation (no thread pools as thread's creation should be one off operation, but just mutexes to access the cache) and dropping the fork(), perhaps replacing it with something similar to a semaphore (ie. when next sql_refresh_time expires, the main thread will let the other thread to go). Thoughts? Cheers, Paolo _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
