could you send me this debug log? I don't know if it already contains everything I need, but it's wort a try ;)
Thx, Rainer On Sat, Nov 23, 2013 at 4:01 AM, Erik Steffl <[email protected]> wrote: > Found these suspicious messages: > > erik@yummly-ubuntu-erik-gazelle:~/work/rrr$ grep 'imuxsock: no > ratelimiter for pid [0-9]\+, creating one' rsyslog.1385086245.relevant.txt > 7701.960144409:7fe2084d3700: imuxsock: no ratelimiter for pid 12768, > creating one > 7701.961486715:7fe2084d3700: imuxsock: no ratelimiter for pid 12769, > creating one > 7702.130585584:7fe2084d3700: imuxsock: no ratelimiter for pid 12770, > creating one > 8001.192419206:7fe2084d3700: imuxsock: no ratelimiter for pid 12771, > creating one > 8001.193828735:7fe2084d3700: imuxsock: no ratelimiter for pid 12772, > creating one > 8001.366835569:7fe2084d3700: imuxsock: no ratelimiter for pid 12773, > creating one > 8301.437811000:7fe2084d3700: imuxsock: no ratelimiter for pid 12774, > creating one > 8301.439113462:7fe2084d3700: imuxsock: no ratelimiter for pid 12775, > creating one > 8301.603468314:7fe2084d3700: imuxsock: no ratelimiter for pid 12776, > creating one > 8369.573510086:7fe2084d3700: imuxsock: no ratelimiter for pid 12778, > creating one > 8388.868412591:7fe2084d3700: imuxsock: no ratelimiter for pid 12780, > creating one > erik@yummly-ubuntu-erik-gazelle:~/work/rrr$ TZ=UTC date -d @1385087701 > Fri Nov 22 02:35:01 UTC 2013 > erik@yummly-ubuntu-erik-gazelle:~/work/rrr$ TZ=UTC date -d @1385088001 > Fri Nov 22 02:40:01 UTC 2013 > erik@yummly-ubuntu-erik-gazelle:~/work/rrr$ TZ=UTC date -d @1385088301 > Fri Nov 22 02:45:01 UTC 2013 > erik@yummly-ubuntu-erik-gazelle:~/work/rrr$ TZ=UTC date -d @1385088369 > Fri Nov 22 02:46:09 UTC 2013 > erik@yummly-ubuntu-erik-gazelle:~/work/rrr$ TZ=UTC date -d @1385088388 > Fri Nov 22 02:46:28 UTC 2013 > > It says it created rate limiter mostly at the time when the bursts of > messages were sent (plus few after the traffic started to flow at which > point the traffic was pretty high because of all those messages in queues). > > The docs says that when rate limiting is applied there would be a > message about dropped messages but I don't see any such message (anywhere > in /var/log/syslog, did a brute force search of all the recently changed > files) and I don't see any messages actually missing (all START, 0 - 199, > END messages are written to files). Maybe rate limiting works differently > with RELP? > > All the rate limiting messages seem to apply to socket 3: > > 7701.960125220:7fe2084d3700: Message from UNIX socket: #3 > 7701.960144409:7fe2084d3700: imuxsock: no ratelimiter for pid 12768, > creating one > > Think that refers to /dev/log because of this line in the beginning of > the full debug file: > > 6245.675534652:7fe20c370780: imuxsock: Opened UNIX socket '/dev/log' (fd > 3). > > This makes some sense since that's where the burst of messages is coming > from. > > Another possibly interesting message is: > > 7975.038523942:7fe2064cf700: main Q: doEnqSingleObject: LightDelay mark > reached for light delayable message - blocking a bit. > > which was received approximately once per second during following > interval (this is also when the traffic went down to zero): > > 1385087975 Fri Nov 22 02:39:35 UTC 2013 > 1385088546 Fri Nov 22 02:49:06 UTC 2013 > > Does this shed any light on what's going on? > > erik > > > On 11/22/2013 01:41 PM, Erik Steffl wrote: > >> got the debug log and pstats for the following failure scenario (RELP >> used for all communication), it's a bit simpler than the one I posted >> before: >> >> - 1 client host sends messages to collector-test:5140 using RELP >> (50-100kB/s) >> >> - collector-test stores messages received on port 5140 to files >> >> - collector-test run program that sends burst of 200 messages every 5 >> minutes to /dev/log >> >> - collector-test sends messages received in /dev/log to >> collector-prod using RELP >> >> Failure: shortly after the burst of messages is sent the traffic to >> collector-test goes down to almost zero, stays down until the next >> (sometimes second next) burst of messages at which point the traffic >> resumes (with a spike of messages that were waiting for transfer). >> >> In the meantime the program on the client host is not slowed down so >> I assume rsyslogd is queueing the messages (as expected). After the >> traffic starts again there is a spike which confirms the assumption (the >> queued messages are delivered) >> >> Files: >> >> collector-test rsyslogd debug file (what I think is) relevant part >> 02:35:00 to 02:46:59: >> https://drive.google.com/file/d/0B5zeQB_ZuLw0X2l0N3kyaWk3MG8/edit?usp= >> sharing >> >> >> collector-test rsyslogd debug file full file (4GB): >> https://drive.google.com/file/d/0B5zeQB_ZuLw0ZHhnU0R4cjVINjA/edit?usp= >> sharing >> (csplit can be used to split the file into manageable chunks using >> regular expressions) >> >> collector-test pstats: >> https://drive.google.com/file/d/0B5zeQB_ZuLw0b0lKTkdxSFk1Smc/edit?usp= >> sharing >> >> >> Significant events: >> >> - 1385086245 Fri Nov 22 02:10:45 UTC 2013 started monitoring >> - silence starts at 2:38/39 (caused by 2:35 burst) >> - nothing happened during 2:40 burst (still zero traffic) >> - traffic goes up at 2:45/46 >> >> The timestamps in debug file (first four digits) are all related to >> 1385086245 (if I understand it correctly for line starting with 8404 the >> actual timestamp would be 1385080000 + 8404 = 1385088404). >> >> Interesting points in debug file: >> >> 8404.437411330:7fe2054cd700: Filter: check for property 'msg' (value ' >> @cee:{"count":200,"START":"Fri Nov 22 02:35:02 2013"}') contains '[UFW >> ': FALSE >> >> the above line is the first message of the 200 message burst (well, >> 202, cause there's START, END and 200 messages in between). This is the >> cause of silence, even though silence only occurs at 2:38/39, pretty >> sure about this since if there are no bursts there is no silence. This >> message was logged at 02:46:44, AFTER the traffic started to flow again >> (if I got my timestamps right) >> >> 8487.447139470:7fe2054cd700: Filter: check for property 'msg' (value ' >> @cee:{"count":200,"START":"Fri Nov 22 02:45:01 2013"}') contains '[UFW >> ': FALSE >> >> The line above is the first message that restarted the RELP traffic, >> it was received at 02:48:07 so it's not in the smaller debug file. >> >> 362429 syslog 414 <134>2013-11-21T18:40:08.604921-08:00 >> yummly-ubuntu-erik-gazelle erikTestSix: >> @cee:{"message-json":{"date":"2013-11-21T18:40:08,587887420-0800 >> "},"yummlyLogOrigin":{"supportLevel":"dev","system":" >> erikTestSytem","cluster":"erikGazelle","role":" >> madeUpRole","host":"yummly-ubuntu-erik-gazelle","tag":" >> erikTestSix:","programname":"erikTestSix","priority":"local0.info >> ","timestamp":"2013-11-21T18:40:08.604921-08:00"}} >> >> >> the above line is an example of a message sent from 1 client host to >> collector-test, they are all same except of date and tag. >> >> RELP traffic (counts for 'relp session read' for each second): >> >> 75 7973 >> 67 7974 >> 4 7975 Fri Nov 22 02:39:35 UTC 2013 >> 1 7976 >> 1 8050 >> 1 8125 >> 1 8199 >> 1 8273 >> 1 8333 >> 1 8348 >> 1 8392 Fri Nov 22 02:46:32 UTC 2013 >> 36 8394 >> 43 8395 >> >> This shows that RELP traffic virtually stopped in interval 02:39:35 >> to 02:46:32. >> >> I also tried to run the burst script on the client host (i.e. burst >> of 200 messages was sent from one client host to collector-test) and I >> was not able to reproduce the problem. So I assume it's sending the >> messages that causes the problem (the client host is not receiving any >> messages so the problem is not visible there). >> >> Hope this will lead at least a tiny step towards figuring out what >> the problem is. Let me know if I can provide more information. I didn't >> do debug log on the other machines involved and didn't do any strace >> cause worried that slightly different/slower performance will make the >> problem disappear. >> >> erik >> >> On 11/20/2013 08:25 PM, David Lang wrote: >> >>> On Wed, 20 Nov 2013, Erik Steffl wrote: >>> >>> found out another failure scenario, by mistake. What happened that I >>>> ended up writing logs like this (all syslog traffic is rsyslog using >>>> RELP protocol): >>>> >>>> 6 hosts send messages to collector-prod >>>> >>>> 1 collector-prod writes logs to files (20-30 files at a time) >>>> >>>> 1 host sends messages to collector-test >>>> >>>> 1 collector-test writes logs to files (same image as production log >>>> collector) >>>> >>>> problem: burst of 100 log messages every 5 minutes sends from >>>> collector-prod to collector-prod (i.e. same machine) cause all 6 >>>> senders to stop sending messages until next burst of 100 messages >>>> (theory is that collector-prod stops confirming messages so nobody >>>> sends new messages) >>>> >>>> NEW INFO: burst of 100 messages sent from collector-TEST to >>>> collector-PROD shuts down traffic to collector-TEST, i.e. sender of >>>> the 100 messages. Traffic to collector-PROD continues without any >>>> (visible) hesitation. >>>> >>>> Seems that SENDING of the burst of messages somehow makes it stop >>>> sending confirmations to the messages that it is receiving from other >>>> servers. >>>> >>> >>> do you have pstats data from the machine during this time? >>> >>> David Lang >>> >>> erik >>>> >>>> On 11/19/2013 05:45 AM, Rainer Gerhards wrote: >>>> >>>>> OK, I am slowly catching up and can most probably look into this >>>>> problem >>>>> now. >>>>> >>>>> Erik, I think I need to create an instrumented version in any case so >>>>> that >>>>> we can get a bit more insight. Are you able to deploy such a version >>>>> from >>>>> git or source tarball? Also, I will need at least a partial debug >>>>> log, from >>>>> before things go wrong up until the situation recovers. If the debug >>>>> log >>>>> get's to large, "debug on demand" (turn on via signal) will probably be >>>>> helpful. >>>>> >>>>> Could you assist with these things? >>>>> >>>>> Thx, >>>>> Rainer >>>>> >>>>> >>>>> On Tue, Nov 19, 2013 at 9:08 AM, David Lang <[email protected]> wrote: >>>>> >>>>> On Mon, 18 Nov 2013, Erik Steffl wrote: >>>>>> >>>>>> On 11/18/2013 08:06 PM, David Lang wrote: >>>>>> >>>>>>> >>>>>>> could you run impstats with a rapid reporting cycle while you >>>>>>>> duplicate >>>>>>>> the problem? >>>>>>>> >>>>>>>> >>>>>>> run impstats how? I can add it, any suggested configuration? Just >>>>>>> log >>>>>>> into a file every minute? >>>>>>> >>>>>>> >>>>>> yes, log into a file every minute or so. We may end up shrinking >>>>>> this to a >>>>>> faster interval >>>>>> >>>>>> When you log as fast as you can, I expect that the queue will grow, >>>>>> and >>>>>> then take a little bit of time to drain off. If it gets stuck while >>>>>> draining, the pstats output will show us. >>>>>> >>>>>> David Lang >>>>>> >>>>>> >>>>>> With a rapid trigger like this, a new debug log may be helpful >>>>>> with less >>>>>> >>>>>>> 'stray noise' in it. >>>>>>>> >>>>>>>> >>>>>>> will try to set that up, at least it will be easier to get close >>>>>>> to the >>>>>>> time the problem starts (sometime between 1st and 100th burst >>>>>>> message) >>>>>>> >>>>>>> Speculation: >>>>>>> >>>>>>>> >>>>>>>> My guess is that something is getting 'stuck' processing the last >>>>>>>> message of the third batch, and this is stalling the system >>>>>>>> (probably >>>>>>>> something is holding a lock that's preventing RELP from finishing >>>>>>>> processing messages, which is causing the senders to pause until >>>>>>>> RELP >>>>>>>> finally acknowledges the sent messages) >>>>>>>> >>>>>>>> knowing if all the messages from the script make it to the log or if >>>>>>>> some of them don't make it until things startup again would be >>>>>>>> useful >>>>>>>> data. This may involve creating a custom output template to add >>>>>>>> extra >>>>>>>> time info (received time, now, etc) to the output line and see what >>>>>>>> happens. >>>>>>>> >>>>>>>> >>>>>>> sounds reasonable, will try to figure out if that's happening, >>>>>>> >>>>>>> thanks! >>>>>>> >>>>>>> erik >>>>>>> >>>>>>> >>>>>>> David Lang >>>>>>>> >>>>>>>> On Mon, 18 Nov 2013, Erik Steffl wrote: >>>>>>>> >>>>>>>> figured out more specific scenario that is causing the silence. >>>>>>>> >>>>>>>>> >>>>>>>>> to recap this is the current setup: >>>>>>>>> >>>>>>>>> - 6 machines sending logs to collector machine using RELP >>>>>>>>> >>>>>>>>> - 1 machine collecting all the logs on disk (20-30 files at a >>>>>>>>> time, >>>>>>>>> split by origin etc.) >>>>>>>>> >>>>>>>>> There is fairly steady traffic of 200-300 messages per second, >>>>>>>>> 200 - >>>>>>>>> 300 kB/s. >>>>>>>>> >>>>>>>>> In this scenario if I run a program that logs 100 messages as >>>>>>>>> fast as >>>>>>>>> possible the whole system goes silent (every THIRD time this >>>>>>>>> happens), >>>>>>>>> all six senders stop sending messages (or almost stop, the traffic >>>>>>>>> goes down to maybe 1kB/s) and doesn't start again until I run >>>>>>>>> another >>>>>>>>> burst of messages (which is equally weird, why would it start again >>>>>>>>> after next burst of messages?). >>>>>>>>> >>>>>>>>> The program that sends 100 messages is running on the log >>>>>>>>> collector >>>>>>>>> host (not sure if it's relevant, likely not). >>>>>>>>> >>>>>>>>> This produces the silences virtually 100% (every third time it >>>>>>>>> runs), >>>>>>>>> as long as I cron the 100-messages-log-sender the problem is >>>>>>>>> there, if >>>>>>>>> I comment it out it works fine (tested both number of times, >>>>>>>>> sometime >>>>>>>>> left it running for days). >>>>>>>>> >>>>>>>>> Previously I thought this might be related to renaming of the >>>>>>>>> files >>>>>>>>> or reload rsyslog (HUP signal) but that is completely irrelevant. >>>>>>>>> Regardless of renaming or HUP signal it's the burst of messages >>>>>>>>> that >>>>>>>>> makes it go silent (and somehow signaling to all senders to go >>>>>>>>> silent, >>>>>>>>> perhaps by not sending RELP confirmations?). >>>>>>>>> >>>>>>>>> The program that logs burst of 100 messages is a perl script that >>>>>>>>> does essentially this (some irrelevant parts left out, think it >>>>>>>>> would >>>>>>>>> run as is but it might need some fixes): >>>>>>>>> >>>>>>>>> use strict; >>>>>>>>> use Log::Log4perl; >>>>>>>>> >>>>>>>>> my $logConfig = ' >>>>>>>>> log4perl.rootLogger=DEBUG, SYSLOG >>>>>>>>> log4perl.appender.SYSLOG = Log::Dispatch::Syslog >>>>>>>>> log4perl.appender.SYSLOG.min_level = debug >>>>>>>>> log4perl.appender.SYSLOG.ident = myTag >>>>>>>>> log4perl.appender.SYSLOG.facility = local0 >>>>>>>>> log4perl.appender.SYSLOG.layout = >>>>>>>>> Log::Log4perl::Layout::PatternLayout >>>>>>>>> log4perl.appender.SYSLOG.layout.ConversionPattern=@cee:%m >>>>>>>>> '; >>>>>>>>> >>>>>>>>> Log::Log4perl::init(\$logConfig); >>>>>>>>> my $l = Log::Log4perl::get_logger(); >>>>>>>>> my $now = localtime(); >>>>>>>>> >>>>>>>>> for(my $i = 0; $i < 100; $i++) { >>>>>>>>> $l->info(JSON::encode_json({now=>$now,i=>$i})); >>>>>>>>> } >>>>>>>>> >>>>>>>>> it needs liblog-log4perl-perl and liblog-dispatch-perl ubuntu >>>>>>>>> packages (or equivalent on other distros). >>>>>>>>> >>>>>>>>> This does not seem to be related to rate-limiting (as documented) >>>>>>>>> since there are no dropped messages, didn't find any rsyslog >>>>>>>>> messages >>>>>>>>> about dropping messages due to rate limiting (grep'd all files in >>>>>>>>> /var/log/). Could this be related to Queue slowdown? We do not set >>>>>>>>> dequeueslowdown for any queues. >>>>>>>>> >>>>>>>>> Any ideas why would a short burst of 100 messages cause this >>>>>>>>> problem? >>>>>>>>> >>>>>>>>> thanks! >>>>>>>>> >>>>>>>>> erik >>>>>>>>> >>>>>>>>> On 11/14/2013 01:53 AM, Rainer Gerhards wrote: >>>>>>>>> >>>>>>>>> On Thu, Nov 14, 2013 at 6:58 AM, Erik Steffl <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> did you have a chance to look at debug log? >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> not yet, but I am finalizing the 8.1.0 release. Bear with me >>>>>>>>>>> another >>>>>>>>>>> >>>>>>>>>> couple >>>>>>>>>> of days pls. >>>>>>>>>> Rainer >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> did more testing in the meantime, figured out that: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 6 senders, 1 collector version 7.5.4, 300 kB/s always works >>>>>>>>>>> >>>>>>>>>>> 6 senders, 1 collector versions 7.5.5, 7.5.6 300 kB/s do >>>>>>>>>>> not work >>>>>>>>>>> >>>>>>>>>>> 6 senders, 2 collectors versions 7.5.5, 7.5.6 300 kB/s mostly >>>>>>>>>>> work but >>>>>>>>>>> not always (periods of silence are much more predicable if the >>>>>>>>>>> re is >>>>>>>>>>> no >>>>>>>>>>> reload/HUP signal) >>>>>>>>>>> >>>>>>>>>>> 1 sender, one collector any version always works (this has >>>>>>>>>>> less >>>>>>>>>>> files >>>>>>>>>>> open plusthe total traffic is around 100lB/s, not sure if that's >>>>>>>>>>> siginificant) >>>>>>>>>>> >>>>>>>>>>> Looked at strace on both sides (sender and receiver) during >>>>>>>>>>> time >>>>>>>>>>> when >>>>>>>>>>> traffic went from 300kB/s down to virtually zero but see nothing >>>>>>>>>>> suspicious, essentially there is lot of messages sent, then one >>>>>>>>>>> message >>>>>>>>>>> confirming all of them. Then at some point there is timeout, >>>>>>>>>>> then few >>>>>>>>>>> messages are confirmed one by one then nothing, then it starts >>>>>>>>>>> again. >>>>>>>>>>> >>>>>>>>>>> As far as I can tell it's the rename of the files that >>>>>>>>>>> causes the >>>>>>>>>>> problem (we rename athen remove them but if I comment out the >>>>>>>>>>> remove >>>>>>>>>>> part >>>>>>>>>>> the problem is still there). We never rename the files that >>>>>>>>>>> rsyslogd >>>>>>>>>>> is >>>>>>>>>>> actually writing to (even though they are still open). >>>>>>>>>>> >>>>>>>>>>> Given the behaviour in the scenarios above it seems >>>>>>>>>>> something got >>>>>>>>>>> broken >>>>>>>>>>> between 7.5.4 and 7.5.5. Did the code that handles renamed files >>>>>>>>>>> change in >>>>>>>>>>> any way? I looked at changelog (http://www.rsyslog.com/ >>>>>>>>>>> changelog-for-7-5-5-v7-devel/) before but didn't see anything >>>>>>>>>>> that >>>>>>>>>>> looks >>>>>>>>>>> related. >>>>>>>>>>> >>>>>>>>>>> Any ideas how to troubleshoot this further? >>>>>>>>>>> >>>>>>>>>>> thanks! >>>>>>>>>>> >>>>>>>>>>> erik >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 11/05/2013 10:04 AM, Rainer Gerhards wrote: >>>>>>>>>>> >>>>>>>>>>> On Tue, Nov 5, 2013 at 3:41 AM, Erik Steffl <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> See the debug log attached, the first line is the last >>>>>>>>>>>> message >>>>>>>>>>>> in one >>>>>>>>>>>> >>>>>>>>>>>> of >>>>>>>>>>>>> the files (maybe not last message overall, but all >>>>>>>>>>>>> communication >>>>>>>>>>>>> stops at >>>>>>>>>>>>> that second, there's number of files so it's hard to figure out >>>>>>>>>>>>> which >>>>>>>>>>>>> message is the last one). >>>>>>>>>>>>> >>>>>>>>>>>>> Also noticed that number of messages in log with same >>>>>>>>>>>>> number >>>>>>>>>>>>> at the >>>>>>>>>>>>> beginning drops right after this message, I assume the >>>>>>>>>>>>> beginning >>>>>>>>>>>>> of the >>>>>>>>>>>>> line has something to do with time, these are the counts and >>>>>>>>>>>>> values at >>>>>>>>>>>>> the >>>>>>>>>>>>> beginning of the debug log line: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> It's the last 5 digits of a Unix timestamp - so you don't >>>>>>>>>>>>> get an >>>>>>>>>>>>> >>>>>>>>>>>> absolute >>>>>>>>>>>> time, but an idea how things progress. After the dot I think >>>>>>>>>>>> there >>>>>>>>>>>> are ms >>>>>>>>>>>> (or so) and after the colon there is a thread ID (same ID = same >>>>>>>>>>>> thread). >>>>>>>>>>>> >>>>>>>>>>>> Hope that helps a bit, will try to look at the debug log >>>>>>>>>>>> tomorrow >>>>>>>>>>>> morning. >>>>>>>>>>>> >>>>>>>>>>>> Rainer >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 21264 9547. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 22378 9548. >>>>>>>>>>>> >>>>>>>>>>>> 2725 9549. >>>>>>>>>>>> >>>>>>>>>>>>> 1877 9550. >>>>>>>>>>>>> ...9551 to 9969 left out, low counts around 2000 >>>>>>>>>>>>> ...9970 to 0519 left out, low counts around 6 >>>>>>>>>>>>> 6 0520. >>>>>>>>>>>>> 6 0521. >>>>>>>>>>>>> 39302 0522. >>>>>>>>>>>>> 51997 0523. >>>>>>>>>>>>> >>>>>>>>>>>>> thanks! >>>>>>>>>>>>> >>>>>>>>>>>>> erik >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 11/03/2013 01:05 PM, David Lang wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> I would post everything in the log file starting with the >>>>>>>>>>>>> last >>>>>>>>>>>>> log that >>>>>>>>>>>>> >>>>>>>>>>>>> it successfully sent and continuing a while after that >>>>>>>>>>>>>> message (a >>>>>>>>>>>>>> while >>>>>>>>>>>>>> being a couple hundred lines or so to be safe) >>>>>>>>>>>>>> >>>>>>>>>>>>>> once we look at the log we may ask for more. >>>>>>>>>>>>>> >>>>>>>>>>>>>> David Lang >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sun, 3 Nov 2013, Erik Steffl wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Date: Sun, 03 Nov 2013 11:45:03 -0800 >>>>>>>>>>>>>> >>>>>>>>>>>>>> From: Erik Steffl <[email protected]> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Reply-To: rsyslog-users <[email protected]> >>>>>>>>>>>>>>> To: rsyslog-users <[email protected]> >>>>>>>>>>>>>>> Subject: Re: [rsyslog] Rsyslog with RELP not >>>>>>>>>>>>>>> sending/receiving >>>>>>>>>>>>>>> messages for >>>>>>>>>>>>>>> long intervals >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I have debug log where I was able to find last message >>>>>>>>>>>>>>> in one >>>>>>>>>>>>>>> of the >>>>>>>>>>>>>>> logs files (right before the silence period) but it's 3 GB :) >>>>>>>>>>>>>>> Any hint >>>>>>>>>>>>>>> how much to pick the relevant part of the debug log? Is the >>>>>>>>>>>>>>> timestamp >>>>>>>>>>>>>>> part of the log message? There is something in the >>>>>>>>>>>>>>> beginning of >>>>>>>>>>>>>>> each >>>>>>>>>>>>>>> debug log line but I don't know how to translate it to time >>>>>>>>>>>>>>> (part of >>>>>>>>>>>>>>> it is transaction id, the rest not sure about) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> thanks! (and of course I unserstand you can't drop >>>>>>>>>>>>>>> everything >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>> work on my problem, thanks for answers!) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> erik >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 11/02/2013 08:51 AM, Rainer Gerhards wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Nov 1, 2013 at 11:31 PM, Erik Steffl >>>>>>>>>>>>>>> <[email protected]> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> running reload every 43 seconds, behaviour is >>>>>>>>>>>>>>>> exactly the >>>>>>>>>>>>>>>> same >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> (30 min >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> traffic, 15 min silence) so I think it's not related to >>>>>>>>>>>>>>>>> reload. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> will test the latest version and I guess re-test >>>>>>>>>>>>>>>>> the older >>>>>>>>>>>>>>>>> version and >>>>>>>>>>>>>>>>> go from there... >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> It would probably be a good idea to try to capture a >>>>>>>>>>>>>>>>> debug >>>>>>>>>>>>>>>>> log so >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> that we >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> can see what rsyslog itself thinks what happens. Note that >>>>>>>>>>>>>>>> you >>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>> just enable it shortly before you expect the problem to >>>>>>>>>>>>>>>> happen >>>>>>>>>>>>>>>> (google for >>>>>>>>>>>>>>>> "debug on demand"). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Right now, I am totally busy with complex work and cannot >>>>>>>>>>>>>>>> look >>>>>>>>>>>>>>>> at it. >>>>>>>>>>>>>>>> If >>>>>>>>>>>>>>>> this is for a business, I would strongly suggest to purchase >>>>>>>>>>>>>>>> one of >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> low-priced support options. That would enable an engineer to >>>>>>>>>>>>>>>> begin >>>>>>>>>>>>>>>> working >>>>>>>>>>>>>>>> on the case within half a day or so. I would *really >>>>>>>>>>>>>>>> love* to >>>>>>>>>>>>>>>> drop >>>>>>>>>>>>>>>> this to >>>>>>>>>>>>>>>> someone else, as I really don't like the smell of this >>>>>>>>>>>>>>>> bug -- >>>>>>>>>>>>>>>> but if >>>>>>>>>>>>>>>> I do, >>>>>>>>>>>>>>>> I am pretty sure other bugs will creep up, and that would >>>>>>>>>>>>>>>> again >>>>>>>>>>>>>>>> be the >>>>>>>>>>>>>>>> death of the new work I am doing (and it took me 3 years >>>>>>>>>>>>>>>> or so >>>>>>>>>>>>>>>> to get >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> this point!). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> So it looks like non-paid work needs to stay focused on that >>>>>>>>>>>>>>>> work, >>>>>>>>>>>>>>>> which >>>>>>>>>>>>>>>> will be useful for all users in the hopefully not so distant >>>>>>>>>>>>>>>> future. >>>>>>>>>>>>>>>> All >>>>>>>>>>>>>>>> non-emergency bugs (usually security related things) or >>>>>>>>>>>>>>>> request >>>>>>>>>>>>>>>> thus >>>>>>>>>>>>>>>> need >>>>>>>>>>>>>>>> to be on hold for a couple of days or maybe till the end of >>>>>>>>>>>>>>>> month. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> If you get a debug log, I will have time to at least look >>>>>>>>>>>>>>>> quickly at >>>>>>>>>>>>>>>> it, >>>>>>>>>>>>>>>> but I can't say if that will be sufficient to provide a >>>>>>>>>>>>>>>> solution. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> sorry for that and I hope for your understanding. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Rainer >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> thanks! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> erik >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 11/01/2013 02:29 PM, David Lang wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Ok, this is definantly sounding like a bug in RELP, it >>>>>>>>>>>>>>>>> worked on an >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> earlier version, check if it works on the latest >>>>>>>>>>>>>>>>> version, and >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> then I >>>>>>>>>>>>>>>>>> would suggest fileing a bug on the www.rsyslog.com >>>>>>>>>>>>>>>>>> bugtracker >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> This sounds like something Rainer is going to need to look >>>>>>>>>>>>>>>>>> at. He's >>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>> Germany, so I believe his weekend has started, he may >>>>>>>>>>>>>>>>>> see it >>>>>>>>>>>>>>>>>> over >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> weekend or it may wait until his work week starts. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> David Lang >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Fri, 1 Nov 2013, Erik Steffl wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> pretty sure reload is sending HUP (that's upstart >>>>>>>>>>>>>>>>>> job as >>>>>>>>>>>>>>>>>> inlcluded in >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> adiscon packages), the rsyslog keeps the same PID, I >>>>>>>>>>>>>>>>>> even >>>>>>>>>>>>>>>>>> tried to >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> strace it while running reload, here's syslog message: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Nov 1 10:47:25 ip-10-238-198-103 rsyslogd: [origin >>>>>>>>>>>>>>>>>>> software="rsyslogd" swVersion="7.5.5" x-pid="3865" >>>>>>>>>>>>>>>>>>> x-info="http://www.rsyslog.com";] rsyslogd was HUPed >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> but I think it must be related to something else >>>>>>>>>>>>>>>>>>> cause >>>>>>>>>>>>>>>>>>> when I >>>>>>>>>>>>>>>>>>> removed >>>>>>>>>>>>>>>>>>> the reload (so the log mover was doing only rename, >>>>>>>>>>>>>>>>>>> upload, >>>>>>>>>>>>>>>>>>> remove) it >>>>>>>>>>>>>>>>>>> was still exhibiting the same works for 30 min, silent >>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>> 15 min >>>>>>>>>>>>>>>>>>> etc. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> running reload every 43 seconds now and will see >>>>>>>>>>>>>>>>>>> what >>>>>>>>>>>>>>>>>>> happens... >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> erik >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On 11/01/2013 12:57 PM, David Lang wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> a HUP of the server should not cause any >>>>>>>>>>>>>>>>>>> interruption in >>>>>>>>>>>>>>>>>>> processing >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> messages. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> are you sure upstart is seing HUP not doing a full >>>>>>>>>>>>>>>>>>>> stop/start? >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> could you try sending HUP to rsyslog manually a few >>>>>>>>>>>>>>>>>>>> times >>>>>>>>>>>>>>>>>>>> to see >>>>>>>>>>>>>>>>>>>> if you >>>>>>>>>>>>>>>>>>>> can duplicate the problem? >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> David Lang >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Fri, 1 Nov 2013, Erik Steffl wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> sorry, it wasn't entirely clear in my previous >>>>>>>>>>>>>>>>>>>> email >>>>>>>>>>>>>>>>>>>> but we >>>>>>>>>>>>>>>>>>>> are doing >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> ubuntu upstart reload which is sending HUP signal, >>>>>>>>>>>>>>>>>>>> pretty >>>>>>>>>>>>>>>>>>>> sure >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> it's >>>>>>>>>>>>>>>>>>>>> doing the right thing since logrotate scripts use it, >>>>>>>>>>>>>>>>>>>>> syslog >>>>>>>>>>>>>>>>>>>>> message >>>>>>>>>>>>>>>>>>>>> says rsyslog received HUP signal, rsyslogd keeps the >>>>>>>>>>>>>>>>>>>>> same >>>>>>>>>>>>>>>>>>>>> PID >>>>>>>>>>>>>>>>>>>>> and all >>>>>>>>>>>>>>>>>>>>> files are closed (as verified using lsof). >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On top of that if, during one of these quiet >>>>>>>>>>>>>>>>>>>>> periods, >>>>>>>>>>>>>>>>>>>>> I try to >>>>>>>>>>>>>>>>>>>>> send >>>>>>>>>>>>>>>>>>>>> messages from another client (on which I just restarted >>>>>>>>>>>>>>>>>>>>> rsyslog >>>>>>>>>>>>>>>>>>>>> so it >>>>>>>>>>>>>>>>>>>>> has no error counters etc.) I think it's also not >>>>>>>>>>>>>>>>>>>>> sending >>>>>>>>>>>>>>>>>>>>> anything, >>>>>>>>>>>>>>>>>>>>> when stracing it (the sender) I see it recieved >>>>>>>>>>>>>>>>>>>>> messages >>>>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>>> logger >>>>>>>>>>>>>>>>>>>>> but is not sending them anywhere. So I think that the >>>>>>>>>>>>>>>>>>>>> collector >>>>>>>>>>>>>>>>>>>>> rsyslog continues to send some kind of signal that >>>>>>>>>>>>>>>>>>>>> it's not >>>>>>>>>>>>>>>>>>>>> ready but >>>>>>>>>>>>>>>>>>>>> not sure why or how to figure out whether it actually >>>>>>>>>>>>>>>>>>>>> does >>>>>>>>>>>>>>>>>>>>> it. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Any ideas how to troubleshoot this? I think I'll >>>>>>>>>>>>>>>>>>>>> record more >>>>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>> strace output and try to reconstruct RELP messages, I >>>>>>>>>>>>>>>>>>>>> guess >>>>>>>>>>>>>>>>>>>>> there is >>>>>>>>>>>>>>>>>>>>> some RELP response that has something else than >>>>>>>>>>>>>>>>>>>>> RSP-CODE = >>>>>>>>>>>>>>>>>>>>> 200 >>>>>>>>>>>>>>>>>>>>> (looking at http://www.rsyslog.com/doc/relp.html) >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I was just about to publish the S3 upload >>>>>>>>>>>>>>>>>>>>> scripts but >>>>>>>>>>>>>>>>>>>>> since >>>>>>>>>>>>>>>>>>>>> they >>>>>>>>>>>>>>>>>>>>> cause rsyslog to misbehave this way I didn't do it yet, >>>>>>>>>>>>>>>>>>>>> maybe >>>>>>>>>>>>>>>>>>>>> I'll >>>>>>>>>>>>>>>>>>>>> post them as they are and fix later... >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> erik >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On 11/01/2013 01:02 AM, David Lang wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> the purpose of RELP is to tell the senders if the >>>>>>>>>>>>>>>>>>>>> receiver was >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> able to >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> accept the message or not, so that's working as >>>>>>>>>>>>>>>>>>>>>> designed. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> When you do a full restart of rsyslog, it stops >>>>>>>>>>>>>>>>>>>>>> processing new >>>>>>>>>>>>>>>>>>>>>> messages >>>>>>>>>>>>>>>>>>>>>> and disconnects all senders (you are doing a full >>>>>>>>>>>>>>>>>>>>>> stop of >>>>>>>>>>>>>>>>>>>>>> rsyslog and >>>>>>>>>>>>>>>>>>>>>> then starting it from scratch, this takes time). My >>>>>>>>>>>>>>>>>>>>>> guess >>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>> that the >>>>>>>>>>>>>>>>>>>>>> senders are detecting 'too many failures' when >>>>>>>>>>>>>>>>>>>>>> trying to >>>>>>>>>>>>>>>>>>>>>> send >>>>>>>>>>>>>>>>>>>>>> messages, >>>>>>>>>>>>>>>>>>>>>> so they are backing off and sleeping for a while >>>>>>>>>>>>>>>>>>>>>> rather >>>>>>>>>>>>>>>>>>>>>> tthan >>>>>>>>>>>>>>>>>>>>>> performing >>>>>>>>>>>>>>>>>>>>>> a mini DOS attack on the network and server. Every >>>>>>>>>>>>>>>>>>>>>> third >>>>>>>>>>>>>>>>>>>>>> restart is >>>>>>>>>>>>>>>>>>>>>> probably triggering a threshold. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Instead of doing a full restart, just send the rsyslog >>>>>>>>>>>>>>>>>>>>>> daemon a >>>>>>>>>>>>>>>>>>>>>> HUP >>>>>>>>>>>>>>>>>>>>>> signal instead. That will tell rsyslog to flush and >>>>>>>>>>>>>>>>>>>>>> close >>>>>>>>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>>>>> files so >>>>>>>>>>>>>>>>>>>>>> that you can rotate them (If you are doing >>>>>>>>>>>>>>>>>>>>>> compression on >>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> files, you >>>>>>>>>>>>>>>>>>>>>> may need to sleep for a few seconds to let the fluch >>>>>>>>>>>>>>>>>>>>>> complete). >>>>>>>>>>>>>>>>>>>>>> Rsyslog >>>>>>>>>>>>>>>>>>>>>> can continue to receive new messages during this >>>>>>>>>>>>>>>>>>>>>> time, so >>>>>>>>>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>>>>>>> senders >>>>>>>>>>>>>>>>>>>>>> will not see an outage. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> By the way, I'm needing a script to upload rsyslog >>>>>>>>>>>>>>>>>>>>>> archives to >>>>>>>>>>>>>>>>>>>>>> S3, >>>>>>>>>>>>>>>>>>>>>> could >>>>>>>>>>>>>>>>>>>>>> you send me a copy of yours? (remove any passwords >>>>>>>>>>>>>>>>>>>>>> first >>>>>>>>>>>>>>>>>>>>>> please >>>>>>>>>>>>>>>>>>>>>> :-) >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> David Lang >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Thu, 31 Oct 2013, Erik Steffl wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Date: Thu, 31 Oct 2013 18:44:23 -0700 >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> From: Erik Steffl <[email protected]> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Reply-To: rsyslog-users <[email protected]> >>>>>>>>>>>>>>>>>>>>>>> To: rsyslog-users <[email protected]> >>>>>>>>>>>>>>>>>>>>>>> Subject: [rsyslog] Rsyslog with RELP not >>>>>>>>>>>>>>>>>>>>>>> sending/receiving >>>>>>>>>>>>>>>>>>>>>>> messages >>>>>>>>>>>>>>>>>>>>>>> for long >>>>>>>>>>>>>>>>>>>>>>> intervals >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> We have a fairly simple setup of 6 hosts sending >>>>>>>>>>>>>>>>>>>>>>> syslog >>>>>>>>>>>>>>>>>>>>>>> messages to >>>>>>>>>>>>>>>>>>>>>>> one collector host, all of these run rsyslog >>>>>>>>>>>>>>>>>>>>>>> 7.5.5-0adiscon2 >>>>>>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>>>>> adiscon repo and use RELP to transfer messages. >>>>>>>>>>>>>>>>>>>>>>> There is >>>>>>>>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>>>>>> load >>>>>>>>>>>>>>>>>>>>>>> balancer in front of the collector machine but I >>>>>>>>>>>>>>>>>>>>>>> dont' >>>>>>>>>>>>>>>>>>>>>>> think it >>>>>>>>>>>>>>>>>>>>>>> matters in this case. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Rsyslog on collector machine is configured to >>>>>>>>>>>>>>>>>>>>>>> write to >>>>>>>>>>>>>>>>>>>>>>> files, >>>>>>>>>>>>>>>>>>>>>>> switching to a new file every 15 minute, using config >>>>>>>>>>>>>>>>>>>>>>> like this >>>>>>>>>>>>>>>>>>>>>>> (abbreviated a bit): >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> template(name="jsonFilename" type="list") { >>>>>>>>>>>>>>>>>>>>>>> constant(value="/path/") >>>>>>>>>>>>>>>>>>>>>>> property(name="$now") >>>>>>>>>>>>>>>>>>>>>>> constant(value="/") >>>>>>>>>>>>>>>>>>>>>>> property(name="$hour") >>>>>>>>>>>>>>>>>>>>>>> constant(value="/") >>>>>>>>>>>>>>>>>>>>>>> property(name="$qhour") >>>>>>>>>>>>>>>>>>>>>>> constant(value="/") >>>>>>>>>>>>>>>>>>>>>>> constant(value="log.json") >>>>>>>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> action(type="omfile" DynaFile="jsonFilename" >>>>>>>>>>>>>>>>>>>>>>> Template="jsonFormat") >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> We run a script at every 2, 17, 32, 47 minute >>>>>>>>>>>>>>>>>>>>>>> of the >>>>>>>>>>>>>>>>>>>>>>> hour >>>>>>>>>>>>>>>>>>>>>>> and upload >>>>>>>>>>>>>>>>>>>>>>> the just finished file to S3. The uploading works >>>>>>>>>>>>>>>>>>>>>>> like >>>>>>>>>>>>>>>>>>>>>>> this: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> - let's say it's 3:02:00, rsyslog is writing to >>>>>>>>>>>>>>>>>>>>>>> /path/2013-10-10/03/00/log.json >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> - get the filename log.json (anything that's not >>>>>>>>>>>>>>>>>>>>>>> current, >>>>>>>>>>>>>>>>>>>>>>> usually >>>>>>>>>>>>>>>>>>>>>>> just one previous file which in the example would be >>>>>>>>>>>>>>>>>>>>>>> /path/2013-10-10/02/03/log.json) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> - rename /path/2013-10-10/02/03/log.json to >>>>>>>>>>>>>>>>>>>>>>> /path/2013-10-10/02/03/log.json.uploading.0 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> - reload rsyslog (to make sure that even if >>>>>>>>>>>>>>>>>>>>>>> for some >>>>>>>>>>>>>>>>>>>>>>> reason >>>>>>>>>>>>>>>>>>>>>>> it was >>>>>>>>>>>>>>>>>>>>>>> writing to just renamed file it would close it and >>>>>>>>>>>>>>>>>>>>>>> open >>>>>>>>>>>>>>>>>>>>>>> a new >>>>>>>>>>>>>>>>>>>>>>> file) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> - upload >>>>>>>>>>>>>>>>>>>>>>> /path/2013-10-10/02/03/log.json.uploading.0 >>>>>>>>>>>>>>>>>>>>>>> to S3 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> - remove >>>>>>>>>>>>>>>>>>>>>>> /path/2013-10-10/02/03/log.json.uploading.0 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Here's what happens every third run (yes, regularly >>>>>>>>>>>>>>>>>>>>>>> EVERY >>>>>>>>>>>>>>>>>>>>>>> THIRD RUN) >>>>>>>>>>>>>>>>>>>>>>> of this script: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> - rsyslog stops writing to the CURRENT file >>>>>>>>>>>>>>>>>>>>>>> (/path/2013-10-10/03/00/log.json, the one that is >>>>>>>>>>>>>>>>>>>>>>> NOT >>>>>>>>>>>>>>>>>>>>>>> being >>>>>>>>>>>>>>>>>>>>>>> renamed) >>>>>>>>>>>>>>>>>>>>>>> few seconds into the run of the script (e.g. 3:02:04) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> - 6 hosts that were sending syslog messages to >>>>>>>>>>>>>>>>>>>>>>> the log >>>>>>>>>>>>>>>>>>>>>>> collector >>>>>>>>>>>>>>>>>>>>>>> STOP >>>>>>>>>>>>>>>>>>>>>>> sending anything (as verified by stracing rsyslogd, >>>>>>>>>>>>>>>>>>>>>>> tcpdump >>>>>>>>>>>>>>>>>>>>>>> and in >>>>>>>>>>>>>>>>>>>>>>> amazon AWS console metric for network in) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> - after this nothing is ever written into >>>>>>>>>>>>>>>>>>>>>>> /path/2013-10-10/03/00/log.json >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> - the 6 clients start sending sysog messages >>>>>>>>>>>>>>>>>>>>>>> again >>>>>>>>>>>>>>>>>>>>>>> when the >>>>>>>>>>>>>>>>>>>>>>> next >>>>>>>>>>>>>>>>>>>>>>> file >>>>>>>>>>>>>>>>>>>>>>> is created (in this example it would be >>>>>>>>>>>>>>>>>>>>>>> /path/2013-10-10/03/01/log.json) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I checked and double check the files, dates, >>>>>>>>>>>>>>>>>>>>>>> verified that >>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>> current file is not touched but can't figure out >>>>>>>>>>>>>>>>>>>>>>> what's >>>>>>>>>>>>>>>>>>>>>>> going >>>>>>>>>>>>>>>>>>>>>>> on. I >>>>>>>>>>>>>>>>>>>>>>> tried the script without reload rsyslog but it didn't >>>>>>>>>>>>>>>>>>>>>>> make any >>>>>>>>>>>>>>>>>>>>>>> difference. If I don't run this script rsyslog works >>>>>>>>>>>>>>>>>>>>>>> flawlessly. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> any ideas how to troubleshoot this? What >>>>>>>>>>>>>>>>>>>>>>> could be >>>>>>>>>>>>>>>>>>>>>>> causing >>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>> rsyslog >>>>>>>>>>>>>>>>>>>>>>> to stop writing to the file and for the senders to >>>>>>>>>>>>>>>>>>>>>>> stop >>>>>>>>>>>>>>>>>>>>>>> sending >>>>>>>>>>>>>>>>>>>>>>> syslog >>>>>>>>>>>>>>>>>>>>>>> messages? I assume the rsyslog on the collector host >>>>>>>>>>>>>>>>>>>>>>> somehow >>>>>>>>>>>>>>>>>>>>>>> signals >>>>>>>>>>>>>>>>>>>>>>> to the 6 hosts that send messages that it's not >>>>>>>>>>>>>>>>>>>>>>> ready or >>>>>>>>>>>>>>>>>>>>>>> something... >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> thanks! >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> erik >>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>>>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>>>>>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>>>>>>>>>>>>>> What's up with rsyslog? Follow >>>>>>>>>>>>>>>>>>>>>>> https://twitter.com/rgerhards >>>>>>>>>>>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are >>>>>>>>>>>>>>>>>>>>>>> ARCHIVED >>>>>>>>>>>>>>>>>>>>>>> by a >>>>>>>>>>>>>>>>>>>>>>> myriad of sites beyond our control. PLEASE >>>>>>>>>>>>>>>>>>>>>>> UNSUBSCRIBE >>>>>>>>>>>>>>>>>>>>>>> and DO >>>>>>>>>>>>>>>>>>>>>>> NOT >>>>>>>>>>>>>>>>>>>>>>> POST >>>>>>>>>>>>>>>>>>>>>>> if you DON'T LIKE THAT. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>>>>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>>>>>>>>>>>>> What's up with rsyslog? Follow >>>>>>>>>>>>>>>>>>>>>> https://twitter.com/rgerhards >>>>>>>>>>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are >>>>>>>>>>>>>>>>>>>>>> ARCHIVED by >>>>>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>> myriad >>>>>>>>>>>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO >>>>>>>>>>>>>>>>>>>>>> NOT POST >>>>>>>>>>>>>>>>>>>>>> if you >>>>>>>>>>>>>>>>>>>>>> DON'T LIKE THAT. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>>>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>>>>>>>>>>>> What's up with rsyslog? Follow >>>>>>>>>>>>>>>>>>>>> https://twitter.com/rgerhards >>>>>>>>>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are >>>>>>>>>>>>>>>>>>>>> ARCHIVED by a >>>>>>>>>>>>>>>>>>>>> myriad of sites beyond our control. PLEASE >>>>>>>>>>>>>>>>>>>>> UNSUBSCRIBE and >>>>>>>>>>>>>>>>>>>>> DO >>>>>>>>>>>>>>>>>>>>> NOT POST >>>>>>>>>>>>>>>>>>>>> if you DON'T LIKE THAT. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>>>>>>>>>>> What's up with rsyslog? Follow >>>>>>>>>>>>>>>>>>>> https://twitter.com/rgerhards >>>>>>>>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are >>>>>>>>>>>>>>>>>>>> ARCHIVED by a >>>>>>>>>>>>>>>>>>>> myriad >>>>>>>>>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO >>>>>>>>>>>>>>>>>>>> NOT >>>>>>>>>>>>>>>>>>>> POST >>>>>>>>>>>>>>>>>>>> if you >>>>>>>>>>>>>>>>>>>> DON'T LIKE THAT. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>>>>>>>>>> What's up with rsyslog? Follow >>>>>>>>>>>>>>>>>>> https://twitter.com/rgerhards >>>>>>>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are >>>>>>>>>>>>>>>>>>> ARCHIVED >>>>>>>>>>>>>>>>>>> by a >>>>>>>>>>>>>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> DO NOT >>>>>>>>>>>>>>>>>>> POST >>>>>>>>>>>>>>>>>>> if you DON'T LIKE THAT. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>>>>>>>>> What's up with rsyslog? Follow >>>>>>>>>>>>>>>>>> https://twitter.com/rgerhards >>>>>>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are >>>>>>>>>>>>>>>>>> ARCHIVED >>>>>>>>>>>>>>>>>> by a >>>>>>>>>>>>>>>>>> myriad >>>>>>>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>>>>>>>>>>>>>>> POST if >>>>>>>>>>>>>>>>>> you >>>>>>>>>>>>>>>>>> DON'T LIKE THAT. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>>>>>>>> What's up with rsyslog? Follow >>>>>>>>>>>>>>>>> https://twitter.com/rgerhards >>>>>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are >>>>>>>>>>>>>>>>> ARCHIVED by >>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>> myriad >>>>>>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>>>>>>>>>>>>>> POST if >>>>>>>>>>>>>>>>> you >>>>>>>>>>>>>>>>> DON'T LIKE THAT. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>>>>>>> What's up with rsyslog? Follow >>>>>>>>>>>>>>>> https://twitter.com/rgerhards >>>>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are >>>>>>>>>>>>>>>> ARCHIVED by a >>>>>>>>>>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and >>>>>>>>>>>>>>>> DO NOT >>>>>>>>>>>>>>>> POST if you DON'T LIKE THAT. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are >>>>>>>>>>>>>>> ARCHIVED by a >>>>>>>>>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO >>>>>>>>>>>>>>> NOT POST >>>>>>>>>>>>>>> if you DON'T LIKE THAT. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>>>>> >>>>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >>>>>>>>>>>>>> by a >>>>>>>>>>>>>> myriad >>>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>>>>>>>>>>> POST >>>>>>>>>>>>>> if you >>>>>>>>>>>>>> DON'T LIKE THAT. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >>>>>>>>>>>>> by a >>>>>>>>>>>>> myriad >>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>>>>>>>>>> POST if >>>>>>>>>>>>> you >>>>>>>>>>>>> DON'T LIKE THAT. >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> >>>>>>>>>>>>> rsyslog mailing list >>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >>>>>>>>>>>> by a >>>>>>>>>>>> myriad >>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>>>>>>>>> POST if >>>>>>>>>>>> you >>>>>>>>>>>> DON'T LIKE THAT. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> >>>>>>>>>>> rsyslog mailing list >>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>>>>> myriad >>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >>>>>>>>>>> if you >>>>>>>>>>> DON'T LIKE THAT. >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> >>>>>>>>>> rsyslog mailing list >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>>>>>>> POST if you DON'T LIKE THAT. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>>>>>> POST >>>>>>>>> if you DON'T LIKE THAT. >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>> myriad >>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>>> you >>>>>>>> DON'T LIKE THAT. >>>>>>>> >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>> myriad >>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>> you >>>>>>> DON'T LIKE THAT. >>>>>>> >>>>>>> _______________________________________________ >>>>>>> >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> myriad >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> >>>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>> POST if you DON'T LIKE THAT. >>>>> >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >>>> if you DON'T LIKE THAT. >>>> >>>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
