Mhhh... with the current design, it submits messages individually to the database. I think what you experience is simply the turn-around from the database call (no other idea what it could be). It doesn't use more CPU because the database layer seems not to return any faster.
There has been some discussion on batching multiple statements together, but this is non-trivial. I lost funding and things like this need a corporate sponsor now (they are not of importance for the non-commercial user field...). You could try to run the action on its own queue and with multiple workers. That could (could!) improve performance. But it is just a guess. Do you have any chance to see how long the query takes inside the SQL engine? Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Daniel Anson > Sent: Thursday, January 15, 2009 6:29 PM > To: [email protected] > Subject: [rsyslog] Baclogged files to disk are pretty slow > > I have been dealing with this problem for a few days now and perhaps I > will be able to solicit some advice or help. Here is the issue. I > have > an rsyslog relay writing to a remote database server and caching to > disk. The write to the database uses a MySQL stored procedure that can > write about 4000 records per second. The rsyslog.conf parts are set up > like so: > > $ModLoad immark > $ModLoadd imudp > $UDPServerAddress 172.16.12.138 > $UDPServerRun 514 > $ModLoad imtcp > $ModLoad imuxsock > $ModLoad imklog > $ModLoad ommysql.so > > $template template1,"CALL > SAT2_RSYSLOG_EVENT_INSERT('%timestamp:::date-mysql%', > '%timegenerated:::date-mysql%', '%syslogfacility%', '%syslogpriority%', > '%hostname%', '%syslogtag%', '%msg%')", sql > > $WorkDirectory /rsyslog/work > $ActionQueueType LinkedList # use asynchronous processing > $ActionQueueFileName dbq # set file name, also enables disk mode > $ActionResumeRetryCount -1 # infinite retries on insert failure > > *.* >172.16.2.238,rsyslog,syslogwriter,topsecret;template1 > > If I turn off the database, in this case I turned it off for almost a > day, it backlogs nearly a 1 GB worth of information. The problem is > that it takes nearly 6 hours to catch back up from this. While > catching > up, it only uses about 1% of the proc. Bandwidth is not an issue as > the > fibre link is only about 50% saturated. Is there a way to force > rsyslogd to consume more of the proc and move faster. I have placed a > -20 nice value on the process in hopes that would help but it really > has > not. Is there a way to force rsyslogd to use a pool of MySQL > connections or intiate a new connection each time a record is written? > > > Daniel M. Anson > Linux Systems Engineer > Rackspace Managed Hosting > [email protected] > > > > > Confidentiality Notice: This e-mail message (including any attached or > embedded documents) is intended for the exclusive and confidential use > of the > individual or entity to which this message is addressed, and unless > otherwise > expressly indicated, is confidential and privileged information of > Rackspace. > Any dissemination, distribution or copying of the enclosed material is > prohibited. > If you receive this transmission in error, please notify us immediately > by e-mail > at [email protected], and delete the original message. > Your cooperation is appreciated. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
