Many more measurements are needed before declaring a conclusive cause, but on the surface it seems that your bottleneck is not rsyslog or the sending server but the database itself. Comments below.
On Thu, Jan 15, 2009 at 10:28, Daniel Anson <[email protected]> wrote: > I have been dealing with this problem for a few days now and perhaps I > will be able to solicit some advice or help. Here is the issue. I have > an rsyslog relay writing to a remote database server and caching to > disk. The write to the database uses a MySQL stored procedure that can > write about 4000 records per second. The rsyslog.conf parts are set up Is that 4000 TPS burst or sustained speed? > If I turn off the database, in this case I turned it off for almost a > day, it backlogs nearly a 1 GB worth of information. The problem is Roughly how many records? > that it takes nearly 6 hours to catch back up from this. While catching > up, it only uses about 1% of the proc. Bandwidth is not an issue as the What's the processor and disk load look like on your MySQL server? > fibre link is only about 50% saturated. Is there a way to force Presuming 50% is your bps, what was your PPS? Depending on how large your average event/transaction are, you may never see 100% due to small packets. > not. Is there a way to force rsyslogd to use a pool of MySQL > connections or intiate a new connection each time a record is written? Ranier confirmed my suspicion that rsyslog executes a single transaction per event, which is (as he also notes) sub-optimal for performance. Batching really should be about the same logic as the MARK functionality: every N foo, output "bar". Multiple actions per transaction (batching) is a classic query tuning technique and can be approached many ways, but you probably need to verify your database I/O is indeed the bottleneck. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
