> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of [email protected] > Sent: Tuesday, April 21, 2009 4:39 AM > To: rsyslog-users > Subject: Re: [rsyslog] multi-message handling and databases > > On Mon, 20 Apr 2009, [email protected] wrote: > > > On Mon, 20 Apr 2009, Rainer Gerhards wrote: > > > >>> -----Original Message----- > >>> From: [email protected] [mailto:rsyslog- > >>> [email protected]] On Behalf Of [email protected] > >>> > >>> On Mon, 20 Apr 2009, Rainer Gerhards wrote: > >>> > >>>>> > >>>>> X=max_messages > >>>>> > >>>>> if (messages in queue) > >>>>> mark that it is going to process the next X messages > >>>>> grab the messages > >>>>> format them for output > >>>>> attempt to deliver the messages > >>>>> if (message delived sucessfully) > >>>>> mark messages in the queue as delivered > >>>>> X=max_messages (reset X in case it was reduced due to > delivery > >>>>> errors) > >>>>> else (delivering this batch failed, reset and try to deliver > the > >>>>> first half) > >>>> > >>>> I think, in our previous discussion (mailing list archive), we > >>> concluded that > >>>> there is no value in re-trying with half of the batch. > >>> > >>> very possibly, I'm not remembering it. > >>> > >>> not doing so will simplify the code considerably, but the > advantages of > >>> retrying with half the batch are: > >>> > >>> 1. you deliver as much as you can > >>> > >>> 2. when you finally get stuck, you can pinpoint directly what > message > >>> you > >>> were stuck on (in case you have a failure based on the data, say > quotes > >>> in > >>> something that then gets formatted into a database, or slashes in > >>> something that becomes a filename component) > >>> > >>> your call > >> > >> I need to refer you back to our previous discussion. Unfortunately, > it was > >> private. I dug the link out and sent it via private mail. Sorry all > others, > >> please stand by a little moment. If I have not read it wrong, it > boiled down > >> to we have no non-transactional sources that were problematic and we > had not > >> identified cases where it would be useful to retry with fewer > elements. > >> > >> I'd provide a more complete description, but that would probably > take me > >> another 2...4 hours, and I hope to get around (yes, it was a > reeeaaaly long > >> discussion). David, if you like to quote anything from me, feel free > to do > >> so. > > > > I'll dig through this today and tonight and review this > > > > to be clear, I'm mostly concerned about the debugging/troubleshooting > > issues (which one of these 1000 messages made the database > complain..). > > but I guess this can be addressed by stopping rsyslog and restarting > it > > with a smaller batch size until you track it down. it should be rare > > enough to make that tolerable. > > looking back over the thread, I think a quick summary is that you > believe > that most output modules would not be able to do anything useful if > they > get an error in the middle of a set of messages.
Yes > I'm not sure that I > agree, but I think it's a fairly easy thing to add later if you are > wrong. > so if you think it's a significant win to drop that for now, go ahead Not yet sure - I think we need to do quite some more doc before actually coding anything ;) > > > my initial proposal was for the output module to return the number of > records sucessfully written, so that those would not be retried. that > definantly isn't possible with a string-based interface, so we then > moved > to 'retry with half' before the thread wound down. I see the picture, but I am not yet sure if this turns out to be a problem. It depends much on the not-yet-specified ways of error recovery. Having said this, I think it is time to do a couple of documents. Most importantly, I think we need to list the failure/error cases and see how we can/need to handle them. This, I think, can lead us to the road of actual implementation work. But you've probably noticed there is much to be written, so let me relax a bit and think about where to start ;) Rainer > > David Lang > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
