On Wed, 14 Dec 2016, David Lang wrote:
this will slow rsyslog down by a factor of >1000x. I ran tests on this using
a very high end PCI based SSD and had a 2000x - 8000x slowdown, and this was
with v5, the added improvements in performance since wukk cause an even
bigger slowdown as they would reduce the amount of time rsyslog spends
processing the message outside of the queue manipulation, I would expect
the slowdown to be significantly above 10,000x nowdays.
you also have to set batch size = 1
The problem is that in a crash you can't count on anything in memory making it
to disk, so you have to write each message to disk, do a fsync on the data,
possibly do a fsync on the directory [1], and only then respond with the ack
doing anything else exposes you to a window where the message could be lost.
But the performance of filesystems is such that doing a fsync for every write to
the disk is crippling, even with SSDs [2]. You also need to make sure that the
SSDs don't buffer the write at all (many do), and that you are writing to a RAID
set (so that the failure of s single SSD doesn't loose the data), which then
means you need to make sure your RAID controller either doesn't do any
buffering, or has a battery backed cache.
If you are willing to loose messages in the case of a complete system
crash/power outage/hardware failure, then things get much simpler, you don't
need to use a disk type queue, just use a disk assisted queue (type FixedArray
or LinkedList with a filename) and saveonshutdown and you get very good
performance (except for the known performace issues with retrieving data from
a disk queue)
You really need to think hard about what messages you Absolutly Must Not Loose
under any conditions. I expect that you will find that there are actually very
few logs that have this requirement, and they probably all come from a custom
application. In this case you are probably better off having the application
write the logs directly to an ACID compliant database (which is slow) rather
than feeding the logs to rsyslog.
If you do need to feed the logs to rsyslog, then you want to have a separate
instance of rsyslog (or at the very least a separate input/ruleset/queue) for
these super critical messages as opposed to all the other messages that you are
going to process.
David Lang
[1] it's actually even worse than this with filesystems newer than ext2
you write the data to the file and do a fsync on the data.
if this changes the size of the file in disk blocks, you then need to do a fsync
on the directory the file is in so that the size change is written
each of these writes are written to the filesystem journal as a temporary thing,
and then the filesystem needs to write a copy of the data to the final location
and (once it knows the data is safe there, possibly involving another fsync for
both the data and metadata), do another write to the journal to know that the
data was safely saved in the final location
so processing one log message can require 6 writes, all of which can require a
fsync
[2] without SSDs you are limited to <rpm/60 fsyncs/sec due to the simple physics
of a write requireing the disk to rotate once, and since you aren't infinantly
fast, you end up a fair bit below that.
ext3 had a bug where a fsync required writing all pending data to the disk,
including data that was written after the fsync started. People documented a
single fsync taking >30 seconds.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
