It appears that v7 handles the load much, much better. I cloned one of our remote servers and upgraded to v7 (very painless using yum). I then had one of our DBAs generate a huge load on a dev postgres server which is generating a ton of logs. When pointing the postgres server at the v7 remote server rsyslog uses around 25% cpu, when I take that same load and point it at a v4 server the cpu instantly pegs at 100%.
I’ll hopefully get the OK to implement v7 in production this week and I’ll report back but things are looking very positive. Thanks everyone! DAN FINN Linux System Administrator Office: 801-746-7580 ext. 5381 Mobile: 801-609-4705 [email protected] Backcountry.com <http://www.backcountry.com/> Competitive Cyclist <http://www.competitivecyclist.com/> RealCyclist.com <http://www.realcyclist.com/> Dogfunk.com <http://www.dogfunk.com/> SteepandCheap.com <http://www.steepandcheap.com/> Chainlove.com <http://www.chainlove.com/> WhiskeyMilitia.com <http://www.whiskeymilitia.com/> On 12/4/13, 11:41 AM, "David Lang" <[email protected]> wrote: >you should be able to. I didn't look really closely, but v7 is backwards >compatible for just about everything, and I don't think you were doing >anything >near the edges of that. > >David Lang > > On Wed, 4 Dec 2013, Dan Finn wrote: > >> Date: Wed, 4 Dec 2013 17:58:36 +0000 >> From: Dan Finn <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: rsyslog-users <[email protected]> >> Subject: Re: [rsyslog] rsyslog frequently queuing to disk when it >>should be >> sending over the network >> >> If I upgrade to v7 on the central servers can I reuse those configs? >> >> >> >> DAN FINN >> Linux System Administrator >> >> Office: 801-746-7580 ext. 5381 >> Mobile: 801-609-4705 >> [email protected] >> >> Backcountry.com <http://www.backcountry.com/> >> Competitive Cyclist <http://www.competitivecyclist.com/> >> RealCyclist.com <http://www.realcyclist.com/> >> Dogfunk.com <http://www.dogfunk.com/> >> SteepandCheap.com <http://www.steepandcheap.com/> >> Chainlove.com <http://www.chainlove.com/> >> WhiskeyMilitia.com <http://www.whiskeymilitia.com/> >> >> >> >> >> >> On 12/4/13, 10:56 AM, "David Lang" <[email protected]> wrote: >> >>> with a quick glance at things >>> >>> you are doing a lot of dynamic filename templates, since you do not >>> change the >>> default dynafile cache size (and I don't know if you can on that >>>ancient >>> a >>> version), rsyslog is spending a LOT of time syncing, closing, and >>>opening >>> files. >>> >>> Also, you are extensively using the if..then style filters, those are >>> much >>> slower than other filters on versions prior to 7.x >>> >>> So it's probably the case that if you upgrade your central servers to a >>> current >>> version, and set a large enough DynaFileCacheSize your performance >>> problems will >>> disappear. >>> >>> David Lang >>> >>> On Wed, 4 Dec 2013, Dan Finn wrote: >>> >>>> OK, now we might be onto something. I can’t determine exactly which >>>> remote machine the client is hitting because it’s going through the F5 >>>> so >>>> what I did is took a look at the stats on the F5 and picked the >>>>busiest >>>> remote server. There is an rsyslog thread on there that is hovering >>>>at >>>> very close to 100%. >>>> >>>> Here’s the config from our destination servers. They all share an >>>> identical config. http://pastebin.com/35K9gw97 >>>> >>>> >>>> >>>> DAN FINN >>>> Linux System Administrator >>>> >>>> Office: 801-746-7580 ext. 5381 >>>> Mobile: 801-609-4705 >>>> [email protected] >>>> >>>> Backcountry.com <http://www.backcountry.com/> >>>> Competitive Cyclist <http://www.competitivecyclist.com/> >>>> RealCyclist.com <http://www.realcyclist.com/> >>>> Dogfunk.com <http://www.dogfunk.com/> >>>> SteepandCheap.com <http://www.steepandcheap.com/> >>>> Chainlove.com <http://www.chainlove.com/> >>>> WhiskeyMilitia.com <http://www.whiskeymilitia.com/> >>>> >>>> >>>> >>>> >>>> >>>> On 12/4/13, 10:41 AM, "David Lang" <[email protected]> wrote: >>>> >>>>> Ok, then the question is how fast is the receiving machine accepting >>>>> messages. >>>>> unless you have an unusually complex template, you should be able to >>>>> send >>>>> messages very fast. >>>>> >>>>> But if the receiving machine is not processing messages fast enough >>>>> there >>>>> will >>>>> be a buildup. but if all it's doing is writing to local files (and >>>>>you >>>>> aren't >>>>> doing a lot of dynamic filename stuff) it's unlikely that it should >>>>>be >>>>> that >>>>> slow. >>>>> >>>>> you could look at what the different threads are doing using top >>>>> (remember to >>>>> hit 'H' to see the threads) and if one or more threads is maxing out >>>>> the >>>>> CPU, >>>>> you can then look at the batching settings. >>>>> >>>>> But I really don't think the sending machine is the bottleneck, if it >>>>> was >>>>> it >>>>> wouldn't be able to write the queue files either. >>>>> >>>>> David Lang >>>>> >>>>> On Wed, 4 Dec 2013, Dan Finn wrote: >>>>> >>>>>> I’m thinking it’s most likely something around #3. :) >>>>>> >>>>>> I don’t think it’s a network or F5 related problem as far as I can >>>>>> tell. >>>>>> For example, right now we have a server that is writing logs to the >>>>>> local >>>>>> spool. I ran tcpdump and I can see rsyslog talking to the >>>>>>destination >>>>>> servers just fine but the spool is slowly growing. According to >>>>>> netstat >>>>>> rsyslog is only making 1 TCP connection to the VIP on the F5 and it >>>>>> seems >>>>>> to be able to pass traffic through that connection. >>>>>> >>>>>> >>>>>> >>>>>> DAN FINN >>>>>> Linux System Administrator >>>>>> >>>>>> Office: 801-746-7580 ext. 5381 >>>>>> Mobile: 801-609-4705 >>>>>> [email protected] >>>>>> >>>>>> Backcountry.com <http://www.backcountry.com/> >>>>>> Competitive Cyclist <http://www.competitivecyclist.com/> >>>>>> RealCyclist.com <http://www.realcyclist.com/> >>>>>> Dogfunk.com <http://www.dogfunk.com/> >>>>>> SteepandCheap.com <http://www.steepandcheap.com/> >>>>>> Chainlove.com <http://www.chainlove.com/> >>>>>> WhiskeyMilitia.com <http://www.whiskeymilitia.com/> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 12/4/13, 9:31 AM, "Dave Caplinger" >>>>>><[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Without impstats output it's hard to say for sure, but since your >>>>>>> config >>>>>>> is so succinct, you are getting a lot of default buffer sizes and >>>>>>> watermark parameters. I see you have $ActionResumeRetryCount set >>>>>>>to >>>>>>> -1 >>>>>>> for infinite retries (which is good). Note though that default >>>>>>>high >>>>>>> and >>>>>>> low water marks are 8,000 and 2,000 messages, respectively. So >>>>>>>once >>>>>>> you >>>>>>> get into disk-assisted mode, you won't leave it until the action >>>>>>> queue >>>>>>> gets all the way down to 2000 messages. The default action queue >>>>>>> size >>>>>>> will be 10,000 messages, and that's really not very much, >>>>>>>especially >>>>>>> in >>>>>>> an environment that has significant spikes in volume. >>>>>>> >>>>>>> The other possibilities that come to mind are: >>>>>>> >>>>>>> 1) that the F5 is correctly sending to an rsyslog server that isn't >>>>>>> listening any more for some reason >>>>>>> >>>>>>> If the receiving side's TCP session gets stuck, or something else >>>>>>> goes >>>>>>> wrong but the F5 doesn't know it, the hashing algorithm will >>>>>>>continue >>>>>>> to >>>>>>> send traffic to the same (dead) destination. TCP default timeouts >>>>>>> are >>>>>>> 2 >>>>>>> minutes; this can seem like an eternity when digging through packet >>>>>>> captures. So on the sending side, perhaps it sends a SYN trying to >>>>>>> open >>>>>>> the session, and then nothing happens for 2 minutes before it tries >>>>>>> all >>>>>>> over again? >>>>>>> >>>>>>> 2) perhaps there's something else in the network breaking the TCP >>>>>>> session, such as a firewall doing NAT >>>>>>> >>>>>>> I've seen cases before where the NAT-ing firewall would time-out >>>>>>> translated IP addresses after a certain period, breaking >>>>>>>long-running >>>>>>> sessions. The Cisco PIX/ASA, for example, has both idle >>>>>>> address-translation timeouts, as well as total duration timeouts. >>>>>>>So >>>>>>> even a currently in-use session can still be affected by something >>>>>>> like >>>>>>> this. >>>>>>> >>>>>>> 3) maybe there is some odd behavior in v4 of rsyslog pertaining to >>>>>>> this >>>>>>> situation that has long since been fixed :-) >>>>>>> >>>>>>> Not pointing fingers; I just don't have a lot of experience with >>>>>>> rsyslog >>>>>>> that old so I'm just speculating. >>>>>>> >>>>>>> -- >>>>>>> Dave Caplinger, Director of Architecture | 402.361.3063 >>>>>>> Solutionary | Relevant . Intelligent . Security >>>>>>> >>>>>>> On Dec 3, 2013, at 6:10 PM, Dan Finn <[email protected]> wrote: >>>>>>> >>>>>>>> I’ve done that and I’ve seen 2 things happen during these periods >>>>>>>> where >>>>>>>> files are being written locally. >>>>>>>> >>>>>>>> 1) Nothing at all was attempted to be sent to the remote >>>>>>>> destination. >>>>>>>> Using telnet I could make a connection just fine but rsyslog >>>>>>>>wasn’t >>>>>>>> even >>>>>>>> attempting to send or talk to the destination server over TCP 514. >>>>>>>> Message queue was growing extremely fast. I can’t explain it but >>>>>>>>on >>>>>>>> the >>>>>>>> 2nd or 3rd restart it started talking to the remote again and >>>>>>>>began >>>>>>>> flushing out the queue. >>>>>>>> 2) lots of traffic is going to the remote over TCP 514. The queue >>>>>>>> is >>>>>>>> slowly growing but growing at a consistent rate. This is the most >>>>>>>> common >>>>>>>> situation, I’ve only seen situation #1 once. I don’t see any >>>>>>>>errors >>>>>>>> or >>>>>>>> retrys or anything like that. >>>>>>>> >>>>>>>> On 12/3/13, 5:01 PM, "David Lang" <[email protected]> wrote: >>>>>>>> >>>>>>>>> On Tue, 3 Dec 2013, Erik Steffl wrote: >>>>>>>>> >>>>>>>>>> we have sort of similar problem, in our case it's Amazon Elastic >>>>>>>>>> Load >>>>>>>>>> Balancer (ELB) that somehow causes the connection go "bad" if >>>>>>>>>> there >>>>>>>>>> is >>>>>>>>>> no >>>>>>>>>> traffic for 5 min (not sure what the exact time is, 1 minute is >>>>>>>>>> ok, >>>>>>>>>> 5 >>>>>>>>>> minutes >>>>>>>>>> is not). >>>>>>>>>> >>>>>>>>>> not sure what going "bad" actually means (still investigating) >>>>>>>>>>but >>>>>>>>>> the >>>>>>>>>> data >>>>>>>>>> is not going through, rsyslog sends data but there is no >>>>>>>>>> response... >>>>>>>>>> it >>>>>>>>>> recovers eventually but not sure what exactly triggers the >>>>>>>>>> recovery >>>>>>>>>> (sending >>>>>>>>>> more messages is what triggers it but how exactly is not clear). >>>>>>>>>> >>>>>>>>>> It's not the same case but maybe you can look into VIP and >>>>>>>>>> connections >>>>>>>>>> and >>>>>>>>>> see what happens there, maybe use strace to see what are the >>>>>>>>>> responses >>>>>>>>>> when >>>>>>>>>> rsyslogd sends data to destination... >>>>>>>>> >>>>>>>>> or use tcpdump to watch the traffic over the network. >>>>>>>>> >>>>>>>>> David Lang >>>>>>>>> >>>>>>>>>> erik >>>>>>>>>> >>>>>>>>>> On 12/03/2013 01:12 PM, Dan Finn wrote: >>>>>>>>>>> I had kind of wondered about that as well but I have a few >>>>>>>>>>> reasons >>>>>>>>>>> that >>>>>>>>>>> make it seem like that is not the case. >>>>>>>>>>> >>>>>>>>>>> The ³central server² is actually a VIP on our F5 load balancer >>>>>>>>>>> with 4 >>>>>>>>>>> rsyslog destination servers behind it. We have about 200 >>>>>>>>>>>servers >>>>>>>>>>> in >>>>>>>>>>> our >>>>>>>>>>> environment and during these busy times the only servers that >>>>>>>>>>> ever >>>>>>>>>>> seem to >>>>>>>>>>> log locally are the postgres servers. The volume of logs being >>>>>>>>>>> written on >>>>>>>>>>> these servers is certainly much higher than anywhere else. My >>>>>>>>>>> theory >>>>>>>>>>> is >>>>>>>>>>> that the rsyslog ³client² is not keeping up with the sheer >>>>>>>>>>>volume >>>>>>>>>>> on >>>>>>>>>>> these >>>>>>>>>>> servers during the busy times but until I can find some >>>>>>>>>>>concrete >>>>>>>>>>> info >>>>>>>>>>> that >>>>>>>>>>> is just a theory. >>>>>>>>>>> >>>>>>>>>>> We are looking gat upgrading to v7 but unfortunately that¹s not >>>>>>>>>>> going >>>>>>>>>>> to >>>>>>>>>>> be a quick fix. I was hoping maybe there was an issue in my >>>>>>>>>>> config >>>>>>>>>>> or >>>>>>>>>>> something that could be tweaked but it sounds like maybe that >>>>>>>>>>>is >>>>>>>>>>> not >>>>>>>>>>> the >>>>>>>>>>> case? >>>>>>>>>>> >>>>>>>>>>> I did capture some debug output while this was happening. >>>>>>>>>>> Unfortunately >>>>>>>>>>> it was pretty large so I don¹t know if I can share the whole >>>>>>>>>>> thing >>>>>>>>>>> but >>>>>>>>>>> is >>>>>>>>>>> there anything in particular I would be looking for in there? >>>>>>>>>>>I >>>>>>>>>>> see >>>>>>>>>>> that >>>>>>>>>>> it says it¹s writing the files locally but I didn¹t see where >>>>>>>>>>>it >>>>>>>>>>> says >>>>>>>>>>> why. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Dan >>>>>>>>>>> >>>>>>>>>>> On 12/3/13, 3:03 AM, "David Lang" <[email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> you are sending the logs via TCP, which means that if the >>>>>>>>>>>>system >>>>>>>>>>>> you >>>>>>>>>>>> are >>>>>>>>>>>> sending >>>>>>>>>>>> logs to gets backed up, logs will queue on the sending system, >>>>>>>>>>>> spilling >>>>>>>>>>>> to disk >>>>>>>>>>>> as needed. >>>>>>>>>>>> >>>>>>>>>>>> the bottleneck is probably on the central server, but we have >>>>>>>>>>>>no >>>>>>>>>>>> info >>>>>>>>>>>> about what >>>>>>>>>>>> it's doing. >>>>>>>>>>>> >>>>>>>>>>>> The go-to tool for diagnosting this sort of problem is the >>>>>>>>>>>> impstats >>>>>>>>>>>> module, but >>>>>>>>>>>> I don't think that existed back in the 4.x days, and tracking >>>>>>>>>>>> down >>>>>>>>>>>> the >>>>>>>>>>>> bottleneck without it is significantly harder. Is there any >>>>>>>>>>>>way >>>>>>>>>>>> you >>>>>>>>>>>> can >>>>>>>>>>>> upgrade >>>>>>>>>>>> to a current version? >>>>>>>>>>>> >>>>>>>>>>>> David Lang >>>>>>>>>>>> >>>>>>>>>>>> On Mon, 2 Dec 2013, Dan Finn wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Date: Mon, 2 Dec 2013 20:53:54 +0000 >>>>>>>>>>>>> From: Dan Finn <[email protected]> >>>>>>>>>>>>> Reply-To: rsyslog-users <[email protected]> >>>>>>>>>>>>> To: "[email protected]" <[email protected]> >>>>>>>>>>>>> Subject: [rsyslog] rsyslog frequently queuing to disk when it >>>>>>>>>>>>> should >>>>>>>>>>>>> be >>>>>>>>>>>>> sending over the network >>>>>>>>>>>>> >>>>>>>>>>>>> Hello, >>>>>>>>>>>>> >>>>>>>>>>>>> I¹m trying to get some insight into an issue that we have >>>>>>>>>>>>>been >>>>>>>>>>>>> seeing >>>>>>>>>>>>> quite a bit. We have some postgres servers that are quite >>>>>>>>>>>>> verbose. >>>>>>>>>>>>> When the servers get busy we have an issue where they queue >>>>>>>>>>>>> their >>>>>>>>>>>>> logs >>>>>>>>>>>>> locally instead of sending over the network however I can¹t >>>>>>>>>>>>> find >>>>>>>>>>>>> any >>>>>>>>>>>>> reason why that would be, at least not from a OS resource >>>>>>>>>>>>> standpoint. >>>>>>>>>>>>> We are running rsyslog4-4.8.0-1.ius.el5. This is my config >>>>>>>>>>>>> from >>>>>>>>>>>>> the >>>>>>>>>>>>> client that was having issues : http://pastebin.com/n3XpRdMm. >>>>>>>>>>>>> >>>>>>>>>>>>> I watched it queue about 10k files under /var/spool/rsyslog >>>>>>>>>>>>> before >>>>>>>>>>>>> I >>>>>>>>>>>>> finally had to manually delete them out because disk was >>>>>>>>>>>>> filling >>>>>>>>>>>>> up. >>>>>>>>>>>>> >>>>>>>>>>>>> What¹s the best way to get some insight into why this might >>>>>>>>>>>>>be >>>>>>>>>>>>> happening? Is there a way I can enable some debug logging >>>>>>>>>>>>>for >>>>>>>>>>>>> the >>>>>>>>>>>>> rsyslog process itself? Any settings in our config that >>>>>>>>>>>>>could >>>>>>>>>>>>> be >>>>>>>>>>>>> tweaked? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Dan >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> DAN FINN >>>>>>>>>>>>> >>>>>>>>>>>>> Linux System Administrator >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [email protected]<mailto:[email protected]> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Backcountry.com<http://www.backcountry.com/> >>>>>>>>>>>>> >>>>>>>>>>>>> Competitive Cyclist<http://www.competitivecyclist.com/> >>>>>>>>>>>>> >>>>>>>>>>>>> RealCyclist.com<http://www.realcyclist.com/> >>>>>>>>>>>>> >>>>>>>>>>>>> Dogfunk.com<http://www.dogfunk.com/> >>>>>>>>>>>>> >>>>>>>>>>>>> SteepandCheap.com<http://www.steepandcheap.com/> >>>>>>>>>>>>> >>>>>>>>>>>>> Chainlove.com<http://www.chainlove.com/> >>>>>>>>>>>>> >>>>>>>>>>>>> WhiskeyMilitia.com<http://www.whiskeymilitia.com/> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >>>>>>>>>>>>> by a >>>>>>>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO >>>>>>>>>>>>> NOT >>>>>>>>>>>>> POST >>>>>>>>>>>>> if you DON'T LIKE THAT. >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> rsyslog mailing list >>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >>>>>>>>>>>by a >>>>>>>>>>> myriad of >>>>>>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>>>>>> you >>>>>>>>>>> DON'T >>>>>>>>>>> LIKE THAT. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> rsyslog mailing list >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by >>>>>>>>>>a >>>>>>>>>> myriad of >>>>>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>>>>> you >>>>>>>>>> DON'T >>>>>>>>>> LIKE THAT. >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>>>>> POST >>>>>>>> if you DON'T LIKE THAT. >>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>> myriad >>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>> you >>>>>>> DON'T LIKE THAT. >>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>>>POST >>>>>> if you DON'T LIKE THAT. >>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >>>> if you DON'T LIKE THAT. >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >>if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
