No problem, hope this helps. Recapping the previous message, plus some additional config snippets:
First define the message format template:
template(name="QradarForwardMsgFormat" type="string"
string="<%pri%>%timestamp% %fromhost-ip% %syslogtag%%msg%")
Then, I have a ruleset that uses the template to omfwd to QRadar. This is
where the queue is defined:
ruleset(name="ForwardToQRadar") {
action(
name="forwardToQRadar" # useful when you're looking at
statistics/troubleshooting impstats output
type="omfwd"
Template="QradarForwardMsgFormat"
Target="<destination hostname or IP address"
Port="514"
Protocol="tcp"
queue.type = "LinkedList"
queue.spoolDirectory = "/var/log/queue" # start writing in
messages to disk under this directory
queue.filename = "forwardToQRadar_q" # this base
filename that will be created under spoolDirectory
queue.size = "300000000" # Sized for my environment.
This will let me buffer about 6 hours' worth of data
queue.highwatermark = "500000" # num messages in memory
queue before it starts writing to disk queue
queue.lowwatermark = "400000" # until memory queue is back
to 400k, at which point the memory queue is used again
queue.maxdiskspace="200g" # the queue will be full when
it occupies 200GB of space on disk. Should not be reached. queue.size=300mil *
512 bytes/msg = ~153GB
queue.saveonshutdown="on" # save queue contents to disk
at shutdown
# simple rate limiting - real max EPS to QRadar for this queue
is 1,000,000 / dequeueslowdown * dequeuebatchsize
queue.dequeueslowdown = "275" # wait 275 microseconds
between dequeuing each batch of messages. if batch size = 2 and this is 275,
effective rate is 7272 msg/sec (or slightly less)
queue.dequeuebatchsize = "2" # clear queue in max of 2 msg
batches. 1,000,000 microseconds per second / 275 = 3636 * 2 batch size = 7272
EPS
action.resumeRetryCount="-1" # retry forever
action.reportSuspension="on" # report errors to log
action.reportSuspensionContinuation="on"
)
}
Some things to keep in mind with the queue are:
- This queue only takes place where you use 'call ForwardToQRadar' as your
action for a filter, such as:
if ($msg contains 'Teardown' )
then {
call ForwardToQRadar
}
- If you have another queue set up, that will have a completely separate set
of dequeueing parameters. Keep that in mind as you're choosing your values so
you don't overload your sending system.
- The queue also serves as a way to rate limit the data being forwarded, and
the rate limiting will take place even if data isn't building up in the queue.
I think the way the documentation describes this is that ALL incoming logs end
up passing through the queue, and therefore will only be forwarded as fast as
your dequeueing settings allow. Keep that in mind as you are choosing the
settings. If the data's bursty the queue will smooth out the delivery (and
also start delivering out of order, see the next point), but if you start
sustaining log rates higher than your dequeue rate, your queue will keep
growing in size and never empty.
- Messages could/will be sent out of order, even during normal operation, but
also guaranteed while the buffer is emptying. Incoming messages to the queue
will be sent out before the older messages are pulled off the top of the
buffer. For us this isn't a problem, because I'd rather have the data
available and out of order, than not have the data at all. Keep this in mind
if you're using the data in SIEM though, as alerts might get generated when
they normally wouldn't have, or vice versa.
Hopefully that's clear enough. I'm sure one of the people who are much more
familiar and better versed in rsyslog will chime in if I've inadvertently lied
about anything. This has been working for me for about 3 months now and has
helped me through a few QRadar upgrades with (as far as I can tell) no message
loss.
Dan Woodruff
-----Original Message-----
From: Dave Cottlehuber [mailto:[email protected]]
Sent: Wednesday, December 28, 2016 11:22 AM
To: [email protected]; Woodruff, Dan <[email protected]>
Subject: Re: [rsyslog] collect and forward w/o change
On Fri, 23 Dec 2016, at 19:16, Woodruff, Dan wrote:
> Then later on if you want to get real fancy, you can set up a queue on
> the ForwardToQRadar ruleset so when QRadar is down for patching logs
> will be buffered and forwarded once QRadar is back up. I just got this
> worked a few weeks ago and can share the full ruleset config if you're
> interested.
>
> Hope that helps,
> Dan
An example of buffer/forward would be super interesting - please share Dan.
A+
Dave
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
