mhhh... This is obviously not intended behavior. There are some rate-limiting features that can deliberately slow down a queue, but I guess you have not configured these. So there either is a bug that activates them at some point during processing (e.g. an invalid memory access could do that) or there is some other bug that causes the dequeue to happen very slowly. In any case, it is hard to guess.
Given the volume of the messages, a debug log probably does not help. We could gain a little insight in at least the queue sizes that rsyslog sees via imdiag plus the (very rudamentary) v5 debug front-end (it doesn't require the engine to be v5!). I would need to spend at least a little work on the front-end to enable remote access, but that's not really a problem. One other thing is that I am still holding a v4-beta with additional fixes as I didn't want to put these in the middle of some other debugging. However, these fixes address potential memory problems, so the most appropriate course of action would be to give that version at least a try. It needs to be released in the next days in any case. I have uploaded that (pre-4.5.6) version so that you can give it a try if you like: http://www.rsyslog.com/download/rsyslog/pre/rsyslog-4.5.6.tar.gz I think it would good if you could try it at least once, because I think any other troubleshooting will boil down to looking at the fixes this version contains. Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Aaron Wiebe > Sent: Monday, November 02, 2009 11:52 PM > To: rsyslog-users > Subject: [rsyslog] Queuing bug (4.5.5) > > Greetings all, > > I appear to have run into an issue with 4.5.5 where queues are not > being flushed in a timely manner. In this specific case, I have data > from last wednesday that is being written to disk in small chunks > today since last wednesday. Unfortunately I cannot be too specific in > details, but here is what I can see: > > Using omfile+gzip, there appears to have been a decent burst in > traffic sometime last wednesday. The rsyslog instance has grown to > 1.8GB of memory and stayed there for a while. I have both main > message and action queues defined using fixedarray, and I see no > on-disk queues (appears to be entirely in memory). > > I've got templates writing out to filenames using an hourly timestamp > (filenames like: [token]-[host]-YYYYMMDD-HH.txt.gz) In some of those > files, all of them less than 5k in size, there are between 5 and 15 > lines of content, all of them from last wednesday, and within a few > seconds of each other. It's almost like there was a significant queue > built up, the hour rolled over, and only the first block of lines were > pulled from the queue. Then the hour rolled over again, and another > block of lines were pulled from the queue. Then the next hour, then > another 5-15 lines. > > Is it possible that one of the queues still has a good chunk of data > built up, and is flushing it out very slowly? It hasn't been > consistantly at the top of the hour either, and not every hour. But > the log lines themselves are sequentially written out, and usually the > lines are within a few seconds of each other. > > For example: > > syslog-myhost-20091102-18.txt.gz: 3 lines, 2 with TS Oct 21 18:46:34 > and one 18:46:35 > syslog-myhost-20091102-19.txt.gz: 17 lines, 3 Oct 21 18:46:35, 14 > lines Oct 21 18:46:36 > syslog-myhost-20091102-20.txt.gz: 12 lines, 8 Oct 21 18:46:36, 4 > lines Oct 21 18:46:37 > > Thoughts? > -Aaron > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
