Hi Jeremy, On Mon, Oct 05, 2009 at 01:14:46PM +1000, Jeremy Lee wrote:
> I'm now discovering that there's often a huge time lag before the data > goes into the database, ranging from a few minutes to up to an hour. I've > got debug going, 20 seconds between SQL refreshes, and I get several: Quoting only introduction to the issue for the sake of brevity; you got the reason why you get nothing for minutes, then suddenly it's all there: buffering. Perhaps try with incremental steps if you not have already done that - instead of jumping from 1024 to 10240. Get the trade-off which better suits your scenario. Overall, what peak Mbps is this installation about? Any pps figure? What i'm trying to figure is why by using buffers of 1K you loose data. Any chance there is a concurrent process leaking full CPU cycles for a substantial amount of time which doesn't allow the daemon to cope with peaks of traffic? On the persistency of the database connection; i'm open to discussion and comments on this. I also see it would apply just fine to you. But let me say some forewords: * pmacct comes from a persistent connection implementation (many years ago); this was dropped because too fragile when adopted as a general purpose solution. Hence migrating to a more stateless approach. This was for a mix of reasons, mainly: a) some conditions hard to detect: server was shut down not properly, firewall, NAT or load-balancers in the middle timing out the session or restarting, etc. b) communications with the database server always passing through 3rd party APIs; this easily translates in not having full control on things. * Adding a clean option in this sense might require quite some work to make it generally applicable, ie. not speaking about a quick fix but something which has to be ported (and tested working fine) across the multiple database software supported by pmacct. Cheers, Paolo _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
