David, I start with some quick pointers. I think it makes sense to move the results of this discussion into a document - or alternatively move it to the wiki, if you (or others) find this useful. I have to admit that I am a bit skeptic about the wiki, I guess mail is better for discussion here. But I wanted to mention this option.
Now on to the meat: > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of [email protected] > Sent: Saturday, April 18, 2009 12:29 AM > To: rsyslog-users > Subject: [rsyslog] multi-message handling and databases > > the company that I work for has decided to sponser multi-message queue > output capability, they have chosen to remain anonomous (I am posting > from > my personal account) > > there are two parts to this. > > 1. the interaction between the output module and the queue > > 2. the configuration of the output module for it's interaction with the > database > > for the first part (how the output module interacts with the queue), > the > criteria are that > > 1. it needs to be able to maintain guarenteed delivery (even in the > face > of crashes, assuming rsyslog is configured appropriately) > > 2. at low-volume times it must not wait for 'enough' messages to > accumulate, messages should be processed with as little latency as > possible > > > > to meet these criteria, what is being proposed is the following > > a configuration option to define the max number of messages to be > processed at once. > > the output module goes through the following loop This sentence covers much of the complexity of this change ;) The "problem" is that is it the other way around. It is not the output module that asks the queue engine for data, it is the queue engine that pushes data to the output module. While this sounds like a simple change of positions, it has greater implications. ... especially if you think about the data flow. At this point, it may make sense to review the data flow. I have described it here: http://www.rsyslog.com/Article350.phtml Even if you don't listen to the presentation, the diagram is useful. In it, you see there are n queues, with n being 1 + number of actions. The "1"-queue is the main message queue. So each message moves first into the main queue, is dequeued there (in the push-way described above), run through the filter engine and then placed into the relevant action queues. So the new interface does not necessarily need to modify the main queue (but there is much benefit in doing so). But it must change the way action queues deliver messages. That, in turn, means that the new batch mode can only work if the action is configured to use any actual queueing mode (not the default "DIRECT" mode, where incoming messages are directly handed over to the action processing without any actual in-memory buffering). So the approach is probably to enhance the queue object (which drives both the main and action queues) to support dequeueing of multiple messages at once (what, as a side-effect, will also greatly reduce looking conflicts). Under normal operations, this is relatively straightforward. It gets messy when there is failure in the actions and it gets very complex if we think about the various shutdown scenarios (not to mention disk assisted queues actually running in DA mode). I have begin to look at these issues (part of today's and over-the-weekend thinking ;)), but this will probably need some more time to finally solve - plus some discussion, I guess... > > X=max_messages > > if (messages in queue) > mark that it is going to process the next X messages > grab the messages > format them for output > attempt to deliver the messages > if (message delived sucessfully) > mark messages in the queue as delivered > X=max_messages (reset X in case it was reduced due to delivery > errors) > else (delivering this batch failed, reset and try to deliver the > first half) I think, in our previous discussion (mailing list archive), we concluded that there is no value in re-trying with half of the batch. > unmark the messages that it tried to deliver (putting them back > into the status where no delivery has been attempted) > X=int(# messages attempted / 2) > if (X=0) > unable to deliver a single message, do existing message error > process > > > > this approach is more complex than a simple 'wait for X messages, then > insert them all', but it has some significant advantages > > 1. no waiting for 'enough' things to happen before something gets > written > > 2. if you have one bad message, it will transmit all the good messages > before the bad one, then error out only on the bad one before picking > up > with the ones after the bad one. This needs to be specified. Again, I think our prior conclusion was that this would not make much sense. After all, if e.g. a SQL statement is invalid in the template, how should it recover? If the sql statement is correct, why should it eternally fail? Or should we drop a message if it fails after n attempts (OK, we can do that already ;)). Hard to do for non-transactional outputs. > > 3. nothing is marked as delivered before delivery is confirmed. > > > > an example of how this would work > > max_messages=15 > > messages arrive 1/sec > > it takes 2+(# messages/2) seconds to process each message (in reality > the > time to insert things into a database is more like 10 + (# messages / > 100) > or even more drastic) > > with the traditional rsyslog output, this would require multiple output > threads to keep up (processing a single message takes 1.5 seconds with > messages arriving 1/sec) > > with the new approach and a cold start you would see > > message arrives (Q=1) at T=0 > om starts processing message a T=0 (expected to take 2.5) > message arrives (Q=2) at T=1 > message arrives (Q=3) at T=2 > om finishes processing message (Q=2) at T=2.5 > om starts processing 2 messages at T=2.5 (expected to take 3) > message arrives (Q=4) at T=3 > message arrives (Q=5) at T=4 > message arrives (Q=6) at T=5 > om finishes processing 2 messages (Q=4) at T=5.5 > om starts processing 4 messages at T=5.5 (expected to take 4) > message arrives (Q=5) at T=6 > message arrives (Q=6) at T=7 > message arrives (Q=7) at T=8 > message arrives (Q=8) at T=9 > om finishes processing 4 messages (Q=4) at T=9.5 > om starts processing 4 messages at T=9.5 (expected to take 4) > > the system is now in a steady state > > message arrives (Q=5) at T=10 > message arrives (Q=6) at T=11 > message arrives (Q=7) at T=12 > message arrives (Q=8) at T=13 > om finishes processing 4 messages (Q=4) at T=13.5 > om starts processing 4 messages at T=13.5 (expected to take 4) > > if a burst of 10 extra messages arrived at time 13.5 this last item > would > become > > 11 messages arrive at (Q=14) at T=13.5 > om starts processing 14 messages at T=13.5 (expected to take 9) > message arrives (Q=15) at T=14 > message arrives (Q=16) at T=15 > message arrives (Q=17) at T=16 > message arrives (Q=18) at T=17 > message arrives (Q=19) at T=18 > message arrives (Q=20) at T=19 > message arrives (Q=21) at T=20 > message arrives (Q=22) at T=21 > message arrives (Q=23) at T=22 > om finishes processing 14 messages (Q=9) at T=22.5 > om starts processing 9 messages at T=22.5 (expected to take 6.5) > > > > thoughts? > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
