Rainer, David Lang's suggestion to use sec (thank you David), is where I'll start. My thought was redis would be a simple de-duplication across two rsyslog receivers (rdis in host to host replication). I would like to investigate sec more thoroughly first, before potentially wasting your time.
Thank you both for your pointers in the right direction. Simon On Tue, 2016-02-23 at 09:24 +0100, Rainer Gerhards wrote: > 2016-02-23 9:19 GMT+01:00 Simon Earthrowl (Eseye) <[email protected]>: > > Peter, David, > > Thank you for your input. I appreciate that there is always packet loss, > > and unordered delivery/detection. > > I am considering redis, so that I could process the numerical sequence > > (outside of rsyslog) in arrears - that is to say up to a time period in > > the past (initially never getting closer to real time than 1 minute in > > the past). This would assume: > > a) All packets had been processed by then (to be empirically tested) > > b) Missing sequence _could_ be an indication of an undetected > > problem. > > c) All collectors (rsyslog) are on the same sideband management > > subnet. > > Redis could also de-duplicate locally log events received by both > > collectors by the use of said serial number. This would work fine for > > Cisco routers. ASA well, that is just the log content... . > > > > My intention is to create log rate metrics for different types of event > > (eg ASA connection build/break rates etc.) and filtering before sending > > forward to the central log collection point. > > > > David, thank you for pointing out I don't need mmsequence (read the > > documentation now and understand). However, I would still like to see > > omhiredis in the repository. > > Which packages are required to build it? > > Rainer > > > > Simon > > > > On Mon, 2016-02-22 at 14:23 -0500, Peter Portante wrote: > >> On Mon, Feb 22, 2016 at 1:57 PM, David Lang <[email protected]> wrote: > >> > >> > Hi, > >> >> I would like to check to see if I have missed any syslog reports from my > >> >> Cisco kit. I have a log in the form of: > >> >> > >> >> 2016-02-08T08:47:57.747201+00:00 router.office.eseye.net 19321286: > >> >> 192.168.107.1: 17326462: Feb 8 2016 08:47:56.746 BST: % > >> >> SEC-6-IPACCESSLOGP: blah blah blah > >> >> > >> >> I'm, not currently looking to check the delay from when the log was > >> >> generated, to when rsyslog processed it. This may change when I'm > >> >> monitoring rsyslog to see if it's having a hard time etc. > >> >> I do have a sequence number (19321286 above), and on the raw feed, I > >> >> would like to make sure this is incremented by 1 (one) each time. > >> >> > >> > > >> > There's not a good way to do this because there are a good number of > >> > conditions that can cause logs to end up processed out of order. Anything > >> > that uses multiple threads to process logs is going to have this sort of > >> > problem. I believe that includes redis. > >> > > >> > >> I think that at the source log file reader you would have to ensure the > >> file offset of the log line is read and included as part of the metadata of > >> the log file, along with a UUID of that log file instance in order to > >> properly restore the order later (assuming eventually all logs sent make it > >> to the central repository). > >> > >> > >> > > >> > Log delivery over a local network if pretty darn reliable, but there are > >> > cases where there are known failures that will cause logs to get lost. > >> > > >> > If you use UDP to deliver the logs, network congestion or the destination > >> > server being overloaded can cause you to loose logs. > >> > > >> > If you use TCP to deliver the logs, any logs in flight when a connection > >> > is broken and needs to be re-established will be lost. > >> > > >> > on a local network with a good HA pair of receivers, my opinion is that > >> > UDP is going to end up being more reliable, but the difference is small > >> > and > >> > only kicks in when other things are going wrong. > >> > > >> > My suspicion is I should use redis, but I would love someone to say "A > >> >> better solution is to use ...". I also want to rate check debug > >> >> entries, as > >> >> just sometimes I forget to turn them off (blush). Again, my suspicion > >> >> is I > >> >> should use the count module. And again, is this a sensible starting > >> >> point? > >> >> > >> > > >> > you could, but you can also use global variables ($\blah variables). the > >> > count module was created at a time when the global variables weren't > >> > working. > >> > > >> > as far as the debug logs go, since you sometimes want them on and other > >> > times don't, rather than doing a rate check in rsyslog as you go, why not > >> > put them (or a copy of them) into a separate file and then have a nightly > >> > report than tells you how many you have (or alerts you if you have 'too > >> > many')? > >> > > >> > And, if that wasn't enough to ask, are there any plans to release these > >> >> two modules on the v8-stable/epel-6 repository? I don't mind compiling > >> >> etc. It's just nice to have yum track changes rather than me.... > >> >> > >> > > >> > you mention the count module, what other module are you looking for? > >> > > >> > David Lang > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com/professional-services/ > >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> > DON'T LIKE THAT. > >> > > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > > LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT.
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
