2016-12-14 15:07 GMT+01:00 mostolog--- via rsyslog <[email protected]>: > >> On Wed, 14 Dec 2016, David Lang wrote: >> >>> this will slow rsyslog down by a factor of >1000x. I ran tests on this >>> using a very high end PCI based SSD and had a 2000x - 8000x slowdown, and >>> this was with v5, the added improvements in performance since wukk cause an >>> even bigger slowdown as they would reduce the amount of time rsyslog spends >>> processing the message outside of the queue manipulation, I would expect the >>> slowdown to be significantly above 10,000x nowdays. >> >> you also have to set batch size = 1 >> >> The problem is that in a crash you can't count on anything in memory >> making it to disk, so you have to write each message to disk, do a fsync on >> the data, possibly do a fsync on the directory [1], and only then respond >> with the ack >> >> doing anything else exposes you to a window where the message could be >> lost. >> >> But the performance of filesystems is such that doing a fsync for every >> write to the disk is crippling, even with SSDs [2]. You also need to make >> sure that the SSDs don't buffer the write at all (many do), and that you are >> writing to a RAID set (so that the failure of s single SSD doesn't loose the >> data), which then means you need to make sure your RAID controller either >> doesn't do any buffering, or has a battery backed cache. >> >> If you are willing to loose messages in the case of a complete system >> crash/power outage/hardware failure, then things get much simpler, you don't >> need to use a disk type queue, just use a disk assisted queue (type >> FixedArray or LinkedList with a filename) and saveonshutdown and you get >> very good performance (except for the known performace issues with >> retrieving data from a disk queue) > > LinkedList and saveonshutdown will be. > > About the "creating temporary queues" if ruleset doesn't have a queue > defined, I'll wait rgerhards for a deep explanation, full of examples, > colors and music all around.
I don't understand "temporary queues", but I must also admit that I cannot dig deeper into the ultra-reliable case this year. There is ample of existing conversation on the web, especially the mailingl list archive. > > would imrelp->file->mmjson+mmnorm->elastic be a better approach? > >> [1] it's actually even worse than this with filesystems newer than ext2 >> >> you write the data to the file and do a fsync on the data. >> if this changes the size of the file in disk blocks, you then need to do a >> fsync on the directory the file is in so that the size change is written >> each of these writes are written to the filesystem journal as a temporary >> thing, and then the filesystem needs to write a copy of the data to the >> final location and (once it knows the data is safe there, possibly involving >> another fsync for both the data and metadata), do another write to the >> journal to know that the data was safely saved in the final location >> >> so processing one log message can require 6 writes, all of which can >> require a fsync >> >> [2] without SSDs you are limited to <rpm/60 fsyncs/sec due to the simple >> physics of a write requireing the disk to rotate once, and since you aren't >> infinantly fast, you end up a fair bit below that. >> >> ext3 had a bug where a fsync required writing all pending data to the >> disk, including data that was written after the fsync started. People >> documented a single fsync taking >30 seconds. > > These are enough reasons to definitively totally absolutely use fsync as > much as we can > :P It is safer to walk from Madrid to Barcelona than to drive a car, yet most people accept the additional risk for the benefit involved ;-) Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
