On Fri, 16 Jan 2015, Todd Mortensen wrote:
I am watching this thread closely as I have a use case that message order is
important, that is using rsyslog to ship a mysql slow log read in via
imfile. I have worked around the issue with a DA queue sending messages out
of order by using a disk queue.
$ModLoad omrelp
$ActionQueueType Disk
*.* :omrelp:remoteserver:514;RSYSLOG_ForwardFormat
What I have recently tried to do is setup a ruleset bound to the imfile input
and then use that ruleset to send the logs to the remote destinate, I tried
to use a queue of direct here hoping that if the remote server is down, that
the imfile would just stop reading in the file.
My tests show that rsyslog still is queueing messages with this config
though.
input modules gather messages and add them to the main queue. worker thread then
pull messages from this main queue and deliver them to the action queues, or
directly to the actions if the action queue is set to direct (the default if
you don't specify otherwise), delivers them to the actions
I don't know if you can set the main queue type to direct or not, you could set
it's size down to something insanely small, but the performance would tank.
David Lang
ruleset(name="infiles") {
action(name="relpinfiles" type="omrelp" template="RSYSLOG_ForwardFormat"
target="removeserver" port="514" queue.type="Direct" )
}
input(type="imfile" file="/local/mysql/slow-queries.log" tag="slowlog:"
severity="debug" facility="local0" ruleset="infiles")
On 01/16/2015 10:25 AM, David Lang wrote:
On Fri, 16 Jan 2015, Dave Caplinger wrote:
If I understand correctly, if an Rsyslog queue is in DAQ mode sending to a
output module (because the output is temporarily unavailable, or not
emptying the output queue quickly enough), then the output will start
getting messages out of order as the Rsyslog sends some current messages
from the front of the queue as well as some from the on-disk back of the
queue. I presume this is an optimization to help get the backlog
delivered and try to get out of DAQ mode ASAP. It would be handy (for me
at least) if we could optionally turn that off for an output queue in
order to deliver the queued messages in-order even if there is an
additional disk write penalty to pay (for longer).
One issue is that disk queues are very slow compared to memory queues, so
it's possible that if you force all messages to be written to the queue
while you are also pulling messages from the queue that this will slow you
down so much that you will never catch up. I think there is room for
improvement here, but that would be pretty major surgery.
I'll also point out that even without disk assisted queues, you can get
messages out of order for several reasons.
1. UDP packets can pass each other 'on the wire' in a sufficiently complex
network.
2. since rsyslog processes messages in batches, when you have multiple
threads working, thread 1 can grab messages 1-100 and a millisecond later
thread 2 can grab messages 101-200 from the queue, message 101 will be sent
long before message 100 (possibly before message 2 gets processed,
depending on the ruleset)
3. If you have redundant relay systems, one may get delayed (it may go down
before relaying all it's messages and send them when it comes back up for
example)
Even before rsyslog implemented batches and had the potential to send
messages out of order, there were still conditions that could cause
out-of-order delivery. When I took the Simple Event Correlator class we
were taught to not do if A followed by B followed by C then X and instead
do if A set flagA, if B set flagB, if C set flagC, if flagA,flagB,flagC
then X.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
