Peter, David,
Thank you for your input. I appreciate that there is always packet loss,
and unordered delivery/detection.
I am considering redis, so that I could process the numerical sequence
(outside of rsyslog) in arrears - that is to say up to a time period in
the past (initially never getting closer to real time than 1 minute in
the past). This would assume:
a) All packets had been processed by then (to be empirically tested)
b) Missing sequence _could_ be an indication of an undetected
problem.
c) All collectors (rsyslog) are on the same sideband management
subnet.
Redis could also de-duplicate locally log events received by both
collectors by the use of said serial number. This would work fine for
Cisco routers. ASA well, that is just the log content... .My intention is to create log rate metrics for different types of event (eg ASA connection build/break rates etc.) and filtering before sending forward to the central log collection point. David, thank you for pointing out I don't need mmsequence (read the documentation now and understand). However, I would still like to see omhiredis in the repository. Simon On Mon, 2016-02-22 at 14:23 -0500, Peter Portante wrote: > On Mon, Feb 22, 2016 at 1:57 PM, David Lang <[email protected]> wrote: > > > Hi, > >> I would like to check to see if I have missed any syslog reports from my > >> Cisco kit. I have a log in the form of: > >> > >> 2016-02-08T08:47:57.747201+00:00 router.office.eseye.net 19321286: > >> 192.168.107.1: 17326462: Feb 8 2016 08:47:56.746 BST: % > >> SEC-6-IPACCESSLOGP: blah blah blah > >> > >> I'm, not currently looking to check the delay from when the log was > >> generated, to when rsyslog processed it. This may change when I'm > >> monitoring rsyslog to see if it's having a hard time etc. > >> I do have a sequence number (19321286 above), and on the raw feed, I > >> would like to make sure this is incremented by 1 (one) each time. > >> > > > > There's not a good way to do this because there are a good number of > > conditions that can cause logs to end up processed out of order. Anything > > that uses multiple threads to process logs is going to have this sort of > > problem. I believe that includes redis. > > > > I think that at the source log file reader you would have to ensure the > file offset of the log line is read and included as part of the metadata of > the log file, along with a UUID of that log file instance in order to > properly restore the order later (assuming eventually all logs sent make it > to the central repository). > > > > > > Log delivery over a local network if pretty darn reliable, but there are > > cases where there are known failures that will cause logs to get lost. > > > > If you use UDP to deliver the logs, network congestion or the destination > > server being overloaded can cause you to loose logs. > > > > If you use TCP to deliver the logs, any logs in flight when a connection > > is broken and needs to be re-established will be lost. > > > > on a local network with a good HA pair of receivers, my opinion is that > > UDP is going to end up being more reliable, but the difference is small and > > only kicks in when other things are going wrong. > > > > My suspicion is I should use redis, but I would love someone to say "A > >> better solution is to use ...". I also want to rate check debug entries, as > >> just sometimes I forget to turn them off (blush). Again, my suspicion is I > >> should use the count module. And again, is this a sensible starting point? > >> > > > > you could, but you can also use global variables ($\blah variables). the > > count module was created at a time when the global variables weren't > > working. > > > > as far as the debug logs go, since you sometimes want them on and other > > times don't, rather than doing a rate check in rsyslog as you go, why not > > put them (or a copy of them) into a separate file and then have a nightly > > report than tells you how many you have (or alerts you if you have 'too > > many')? > > > > And, if that wasn't enough to ask, are there any plans to release these > >> two modules on the v8-stable/epel-6 repository? I don't mind compiling > >> etc. It's just nice to have yum track changes rather than me.... > >> > > > > you mention the count module, what other module are you looking for? > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT.
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
