On Wed, 14 Dec 2016, David Lang wrote:
this will slow rsyslog down by a factor of >1000x. I ran tests on
this using a very high end PCI based SSD and had a 2000x - 8000x
slowdown, and this was with v5, the added improvements in performance
since wukk cause an even bigger slowdown as they would reduce the
amount of time rsyslog spends processing the message outside of the
queue manipulation, I would expect the slowdown to be significantly
above 10,000x nowdays.
you also have to set batch size = 1
The problem is that in a crash you can't count on anything in memory
making it to disk, so you have to write each message to disk, do a
fsync on the data, possibly do a fsync on the directory [1], and only
then respond with the ack
doing anything else exposes you to a window where the message could be
lost.
But the performance of filesystems is such that doing a fsync for
every write to the disk is crippling, even with SSDs [2]. You also
need to make sure that the SSDs don't buffer the write at all (many
do), and that you are writing to a RAID set (so that the failure of s
single SSD doesn't loose the data), which then means you need to make
sure your RAID controller either doesn't do any buffering, or has a
battery backed cache.
If you are willing to loose messages in the case of a complete system
crash/power outage/hardware failure, then things get much simpler, you
don't need to use a disk type queue, just use a disk assisted queue
(type FixedArray or LinkedList with a filename) and saveonshutdown and
you get very good performance (except for the known performace issues
with retrieving data from a disk queue)
LinkedList and saveonshutdown will be.
About the "creating temporary queues" if ruleset doesn't have a queue
defined, I'll wait rgerhards for a deep explanation, full of examples,
colors and music all around.
would imrelp->file->mmjson+mmnorm->elastic be a better approach?
[1] it's actually even worse than this with filesystems newer than ext2
you write the data to the file and do a fsync on the data.
if this changes the size of the file in disk blocks, you then need to
do a fsync on the directory the file is in so that the size change is
written
each of these writes are written to the filesystem journal as a
temporary thing, and then the filesystem needs to write a copy of the
data to the final location and (once it knows the data is safe there,
possibly involving another fsync for both the data and metadata), do
another write to the journal to know that the data was safely saved in
the final location
so processing one log message can require 6 writes, all of which can
require a fsync
[2] without SSDs you are limited to <rpm/60 fsyncs/sec due to the
simple physics of a write requireing the disk to rotate once, and
since you aren't infinantly fast, you end up a fair bit below that.
ext3 had a bug where a fsync required writing all pending data to the
disk, including data that was written after the fsync started. People
documented a single fsync taking >30 seconds.
These are enough reasons to definitively totally absolutely use fsync as
much as we can
:P
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
