On Jan 16, 2015, at 2:51 PM, David Lang <[email protected]> wrote: > > On Fri, 16 Jan 2015, Dave Caplinger wrote: > >> ... filesystem buffer can help speed reading data previously written to >> disk if your outage was short enough to not get "too far" behind, because >> the >> data is still actually in RAM so you don't actually have to pay physical >> IOPS >> to touch the disk to retrieve it. > > the filesystem actions are the super expensive parts, even if things are > cached > to ram. There are also fsyncs that take place to make the data safe, and they > force disk IOPS
I agree the write path is certainly expensive (and more so by frequent fsyncs), but when you come back 'n' minutes later to read it (and it's still in the filesystem buffer), I only meant that it's much quicker than having to actually seek and read from disk again. So you're not paying the penalty twice in this case. >> ... time windows ... > > something to think about here, what do you use as a time reference (both for > 'now' and for the log message you are processing), do you use the current > time > on the system doing the processing, or the timestamps in the messages. A combination of receive time at the collector closest to the source (which we can control the clocks on) along with current time at the system doing the processing. Lies the source device told about it's time are kept as-is but not believed... > Using the system time can cause some false positive alerts when logs are > catching up (as you have events that happened over a wide timeframe delivered > over a short timeframe), but you don't have to deal (much) with time going > backwards Very true; adding reliable timestamps as close as possible to the source is our mitigation. > Using the timestamp in the log message gets interesting as you deal with > machines local times drifting, being in different timezones, or just plain > being > wrong. And as you say, how do you know when an event is really 'too old' and > you > can stop tracking it. (what if a redundant box goes down over a long weekend, > do > you really want to keep the correlations open for days in case it has > 'interesting' combinations of events that it will finish delivering when it's > fixed??) See? Lies! :-) -- Dave Caplinger, Director of Architecture | Ph: (402) 361-3063 | Solutionary — An NTT Group Security Company _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
