2015-01-17 0:43 GMT+01:00 David Lang <[email protected]>:
> On Fri, 16 Jan 2015, Todd Mortensen wrote:
>
> I am watching this thread closely as I have a use case that message order
>> is important, that is using rsyslog to ship a mysql slow log read in via
>> imfile. I have worked around the issue with a DA queue sending messages
>> out of order by using a disk queue.
>>
>> $ModLoad omrelp
>> $ActionQueueType Disk
>> *.* :omrelp:remoteserver:514;RSYSLOG_ForwardFormat
>>
>>
>> What I have recently tried to do is setup a ruleset bound to the imfile
>> input and then use that ruleset to send the logs to the remote destinate,
>> I tried to use a queue of direct here hoping that if the remote server is
>> down, that the imfile would just stop reading in the file.
>>
>> My tests show that rsyslog still is queueing messages with this config
>> though.
>>
>
> input modules gather messages and add them to the main queue. worker
> thread then pull messages from this main queue and deliver them to the
> action queues, or directly to the actions if the action queue is set to
> direct (the default if you don't specify otherwise), delivers them to the
> actions
>
> I don't know if you can set the main queue type to direct or not, you
> could set it's size down to something insanely small, but the performance
> would tank.
It's possible to set it to direct, and that actually makes sense if you
want to have as strict ordering as possible. This is because to achieve
strictest ordering, you need to turn of parallel processing as much as
possible (this is restricted to the local instance, of course). With
"direct" mode, you actually turn off the queue system and as such a lot of
parallel processing. This can also make sense for inputs like tcp, where
the queue then doesn't really disappear but is moved to the network and the
sender. Running a dedicated ruleset with *pure* disk or memory (but not DA)
queue with a single worker thread, and being bound to a single input, and
using exclusive outputs, also helps getting as much as serial processing as
possible, with the added benefit a queue can provide. For pull mode inputs
like imfile, it's better to run in direct mode in this case (because why
buffer when you can simply push back pulling?).
The reason imfile still picks up messages when the destination is down is
because it uses input batching, that is it reads a batch of messages before
it submits them for processing. This essentially gains the same performance
benefits as output batching. The default input batch size is 1024, but you
can set it with the "maxSubmitAtOnce" parameter to one. Note that both RELP
and the underlying TCP are windowing protocols, so they buffer some
messages, too.
HTH
Rainer
>
>
> David Lang
>
>
> ruleset(name="infiles") {
>> action(name="relpinfiles" type="omrelp" template="RSYSLOG_ForwardFormat"
>> target="removeserver" port="514" queue.type="Direct" )
>> }
>>
>> input(type="imfile" file="/local/mysql/slow-queries.log" tag="slowlog:"
>> severity="debug" facility="local0" ruleset="infiles")
>>
>>
>>
>>
>> On 01/16/2015 10:25 AM, David Lang wrote:
>>
>>> On Fri, 16 Jan 2015, Dave Caplinger wrote:
>>>
>>> If I understand correctly, if an Rsyslog queue is in DAQ mode sending
>>>> to a output module (because the output is temporarily unavailable, or not
>>>> emptying the output queue quickly enough), then the output will start
>>>> getting messages out of order as the Rsyslog sends some current messages
>>>> from the front of the queue as well as some from the on-disk back of the
>>>> queue. I presume this is an optimization to help get the backlog delivered
>>>> and try to get out of DAQ mode ASAP. It would be handy (for me at least) if
>>>> we could optionally turn that off for an output queue in order to deliver
>>>> the queued messages in-order even if there is an additional disk write
>>>> penalty to pay (for longer).
>>>>
>>>
>>> One issue is that disk queues are very slow compared to memory queues,
>>> so it's possible that if you force all messages to be written to the queue
>>> while you are also pulling messages from the queue that this will slow you
>>> down so much that you will never catch up. I think there is room for
>>> improvement here, but that would be pretty major surgery.
>>>
>>> I'll also point out that even without disk assisted queues, you can get
>>> messages out of order for several reasons.
>>>
>>> 1. UDP packets can pass each other 'on the wire' in a sufficiently
>>> complex network.
>>>
>>> 2. since rsyslog processes messages in batches, when you have multiple
>>> threads working, thread 1 can grab messages 1-100 and a millisecond later
>>> thread 2 can grab messages 101-200 from the queue, message 101 will be sent
>>> long before message 100 (possibly before message 2 gets processed,
>>> depending on the ruleset)
>>>
>>> 3. If you have redundant relay systems, one may get delayed (it may go
>>> down before relaying all it's messages and send them when it comes back up
>>> for example)
>>>
>>> Even before rsyslog implemented batches and had the potential to send
>>> messages out of order, there were still conditions that could cause
>>> out-of-order delivery. When I took the Simple Event Correlator class we
>>> were taught to not do if A followed by B followed by C then X and instead
>>> do if A set flagA, if B set flagB, if C set flagC, if flagA,flagB,flagC
>>> then X.
>>>
>>> David Lang
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
