2016-02-23 9:19 GMT+01:00 Simon Earthrowl (Eseye) <[email protected]>: > Peter, David, > Thank you for your input. I appreciate that there is always packet loss, > and unordered delivery/detection. > I am considering redis, so that I could process the numerical sequence > (outside of rsyslog) in arrears - that is to say up to a time period in > the past (initially never getting closer to real time than 1 minute in > the past). This would assume: > a) All packets had been processed by then (to be empirically tested) > b) Missing sequence _could_ be an indication of an undetected > problem. > c) All collectors (rsyslog) are on the same sideband management > subnet. > Redis could also de-duplicate locally log events received by both > collectors by the use of said serial number. This would work fine for > Cisco routers. ASA well, that is just the log content... . > > My intention is to create log rate metrics for different types of event > (eg ASA connection build/break rates etc.) and filtering before sending > forward to the central log collection point. > > David, thank you for pointing out I don't need mmsequence (read the > documentation now and understand). However, I would still like to see > omhiredis in the repository.
Which packages are required to build it? Rainer > > Simon > > On Mon, 2016-02-22 at 14:23 -0500, Peter Portante wrote: >> On Mon, Feb 22, 2016 at 1:57 PM, David Lang <[email protected]> wrote: >> >> > Hi, >> >> I would like to check to see if I have missed any syslog reports from my >> >> Cisco kit. I have a log in the form of: >> >> >> >> 2016-02-08T08:47:57.747201+00:00 router.office.eseye.net 19321286: >> >> 192.168.107.1: 17326462: Feb 8 2016 08:47:56.746 BST: % >> >> SEC-6-IPACCESSLOGP: blah blah blah >> >> >> >> I'm, not currently looking to check the delay from when the log was >> >> generated, to when rsyslog processed it. This may change when I'm >> >> monitoring rsyslog to see if it's having a hard time etc. >> >> I do have a sequence number (19321286 above), and on the raw feed, I >> >> would like to make sure this is incremented by 1 (one) each time. >> >> >> > >> > There's not a good way to do this because there are a good number of >> > conditions that can cause logs to end up processed out of order. Anything >> > that uses multiple threads to process logs is going to have this sort of >> > problem. I believe that includes redis. >> > >> >> I think that at the source log file reader you would have to ensure the >> file offset of the log line is read and included as part of the metadata of >> the log file, along with a UUID of that log file instance in order to >> properly restore the order later (assuming eventually all logs sent make it >> to the central repository). >> >> >> > >> > Log delivery over a local network if pretty darn reliable, but there are >> > cases where there are known failures that will cause logs to get lost. >> > >> > If you use UDP to deliver the logs, network congestion or the destination >> > server being overloaded can cause you to loose logs. >> > >> > If you use TCP to deliver the logs, any logs in flight when a connection >> > is broken and needs to be re-established will be lost. >> > >> > on a local network with a good HA pair of receivers, my opinion is that >> > UDP is going to end up being more reliable, but the difference is small and >> > only kicks in when other things are going wrong. >> > >> > My suspicion is I should use redis, but I would love someone to say "A >> >> better solution is to use ...". I also want to rate check debug entries, >> >> as >> >> just sometimes I forget to turn them off (blush). Again, my suspicion is I >> >> should use the count module. And again, is this a sensible starting point? >> >> >> > >> > you could, but you can also use global variables ($\blah variables). the >> > count module was created at a time when the global variables weren't >> > working. >> > >> > as far as the debug logs go, since you sometimes want them on and other >> > times don't, rather than doing a rate check in rsyslog as you go, why not >> > put them (or a copy of them) into a separate file and then have a nightly >> > report than tells you how many you have (or alerts you if you have 'too >> > many')? >> > >> > And, if that wasn't enough to ask, are there any plans to release these >> >> two modules on the v8-stable/epel-6 repository? I don't mind compiling >> >> etc. It's just nice to have yum track changes rather than me.... >> >> >> > >> > you mention the count module, what other module are you looking for? >> > >> > David Lang >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > DON'T LIKE THAT. >> > >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
