Todd, sorry, the web folks seem to have screwed up with the URL redirector on one of the recent site changes. The actual bug tracker URL is
http://bugzilla.adiscon.com/ I'll see that the redirector gets fixed. Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Todd Michael Bushnell > Sent: Tuesday, March 22, 2011 4:45 AM > To: rsyslog-users > Subject: Re: [rsyslog] Back logs from disk assisted > queuenotflowingtocentralloghost after service restored > > Thanks for the feedback Rainer. I understand you're busy. Just tried > via multiple browsers (chrome, safari) to access the Bugs page > (http://www.rsyslog.com/bugs), but no such luck (no error - just a > blank page on both occasions) Will try again tomorrow to file the bug. > > If I should be hitting a URL other than http://www.rsyslog.com/bugs let > me know and I'll get this filed. > > Todd > > > > > > On Mar 18, 2011, at 1:58 AM, Rainer Gerhards wrote: > > > Hi Todd, > > > > thanks for the detailed report. Unfortunately, I do not have time at > the > > moment to go through this longer debugging effort (I need to create > slides > > for a conference next week, plus have to do some paid work...). I'd > > appreciate if you could open a bug tracker with the info. I will look > at it > > asap, but that's probably after next week. > > > > Rainer > > > >> -----Original Message----- > >> From: [email protected] [mailto:rsyslog- > >> [email protected]] On Behalf Of Todd Michael Bushnell > >> Sent: Friday, March 18, 2011 6:34 AM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] Back logs from disk assisted queue > >> notflowingtocentralloghost after service restored > >> > >> Rainer, > >> > >> Will send you additional debug to your private email momentarily. > Here's > >> what I'm seeing: As expected, rsyslog starts to locally queue logs > in file > >> identified by ActionQueueFileName (e.g. failqueue-loghost#.0000n) if > > central > >> loghost is inaccessible. This is good. To simulate, I use iptables > to > > block traffic > >> to one of my loghosts and then blast 10,000 messages on that client. > > Here's > >> what $WorkDirectory looks like when I do this: > >> > >> [root@server1 rsyslog]# ls -al > >> total 4988 > >> drwxr-x--- 2 root wheel 4096 Mar 17 21:34 . > >> drwxr-xr-x 7 root root 4096 Mar 17 04:08 .. > >> -rw------- 1 root root 619948 Mar 17 21:34 failqueue- > loghost1.00000002 > >> -rw------- 1 root root 1048800 Mar 17 21:34 failqueue- > loghost2.00000001 > >> -rw------- 1 root root 1048850 Mar 17 21:34 failqueue- > loghost2.00000002 > >> -rw------- 1 root root 1048581 Mar 17 21:34 failqueue- > loghost2.00000003 > >> -rw------- 1 root root 1048988 Mar 17 21:34 failqueue- > loghost2.00000004 > >> -rw------- 1 root root 234515 Mar 17 21:34 failqueue- > loghost2.00000005 > >> > >> Note: loghost2 is the server I make inaccessible. loghost1 is still > > accessible. > >> assume it's queuing because loghost can't keep up with message > blast. > >> > >> I then restart iptables to make loghost2 accessible again. after a > minute > > or so > >> I check $WorkDirectory and it looks like this: > >> > >> [root@server1 rsyslog]# ls -al > >> total 860 > >> drwxr-x--- 2 root wheel 4096 Mar 17 21:36 . > >> drwxr-xr-x 7 root root 4096 Mar 17 04:08 .. > >> -rw------- 1 root root 621295 Mar 17 21:36 failqueue- > loghost1.00000002 > >> -rw------- 1 root root 236716 Mar 17 21:36 failqueue- > loghost2.00000005 > >> > >> So as you can see, most of the logs clear out as expected, but I'm > always > > left > >> with one logfile for each of my logservers. When I check the > central > > loghosts > >> they have already received all of the test messages so these > remaining > > files > >> contain messages that the central loghosts already have. > Furthermore, > >> future logs destined for the central loghosts get appended to these > files > >> even though they are arriving at the central loghosts. > >> > >> I then stop rsyslog (I clearly identify where I do this by echoing > "RSYSLOG > >> RESTART" in debug file) and start it back up. When I do this, both > files > > go > >> away. > >> > >> Note: though not represented in this debug, I'm sometimes seeing the > same > >> behavior with the my MainMsgQueue. The file will stick around and > all new > >> log entries get copied to it until rsyslog is restarted and the > files go > > away. > >> > >> Hopefully the debug log will provide some answers. Thx. > >> > >> Todd > >> > >> On Mar 17, 2011, at 8:53 AM, Rainer Gerhards wrote: > >> > >>> I have had a quick look at the debug log. Check line 133. It looks > >>> like there is some problem within the queue file. This makes > rsyslog > >>> switch over to using a pure memory queue. > >>> > >>> Rainer > >>> > >>>> -----Original Message----- > >>>> From: [email protected] [mailto:rsyslog- > >>>> [email protected]] On Behalf Of Rainer Gerhards > >>>> Sent: Thursday, March 17, 2011 4:01 PM > >>>> To: rsyslog-users > >>>> Subject: Re: [rsyslog] Back logs from disk assisted queue not > >>>> flowingtocentralloghost after service restored > >>>> > >>>> Please feel free to send to my private email address (the list > will > >>> probably > >>>> reject due to size anyway). I promise to have a quick look, but I > >>>> will > >>> probably > >>>> not be able to have an in-depth look until some time next week > (but > >>>> hopefully the quick look helps ;)) > >>>> > >>>> Rainer > >>>> > >>>>> -----Original Message----- > >>>>> From: [email protected] [mailto:rsyslog- > >>>>> [email protected]] On Behalf Of Todd Michael Bushnell > >>>>> Sent: Thursday, March 17, 2011 3:53 PM > >>>>> To: rsyslog-users > >>>>> Subject: Re: [rsyslog] Back logs from disk assisted queue not > >>>>> flowing tocentralloghost after service restored > >>>>> > >>>>> Will do Rainer. Just confirming, I should send zipped debug logs > to > >>>>> this > >>>> list or > >>>>> is there a private email address you prefer? Also, I ran debug > on > >>>>> an > >>>> existing > >>>>> system moments ago - a system that currently has several of these > >> "stuck" > >>>>> failqueue logfiles. Want to make sure that will give you what > you > >>>>> need or > >>>> if I > >>>>> need to start over, simulate a central loghost outage and grab > that > >>>>> information? If the former, I have what you need and will send > once > >>>>> I get confirm on location to send. The latter will take some > time > >>>>> so I can > >>>> simulate > >>>>> worthwhile test. Thx. > >>>>> > >>>>> todd > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On Mar 17, 2011, at 12:39 AM, Rainer Gerhards wrote: > >>>>> > >>>>>> This looks like we need a debug log... > >>>>>> > >>>>>> Rainer > >>>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: [email protected] [mailto:rsyslog- > >>>>>>> [email protected]] On Behalf Of Todd Michael Bushnell > >>>>>>> Sent: Thursday, March 17, 2011 1:18 AM > >>>>>>> To: rsyslog-users > >>>>>>> Subject: [rsyslog] Back logs from disk assisted queue not > flowing > >>>>>>> to centralloghost after service restored > >>>>>>> > >>>>>>> Have central loghost configured with disk assisted queue like > so: > >>>>>>> > >>>>>>> $WorkDirectory /var/log/rsyslog > >>>>>>> $ActionQueueType LinkedList > >>>>>>> $ActionQueueFileName failqueue-loghost2 > >> $ActionResumeRetryCount > >>>> -1 > >>>>>>> $ActionQueueSaveOnShutdown on > >>>>>>> > >>>>>>> # remote logging of everything > >>>>>>> *.* @@loghost1:5140 > >>>>>>> > >>>>>>> Central loghost still running syslog-ng. Had a problem with it > >>>>>>> that caused it to fail on multiple occasions over the past > couple > >>> days. > >>>>>>> Resolved the problem and logs are now flowing to it, but the > files > >>>>>>> that were created on the clients during this period are not > going > >>>>>>> away, nor are the back logs flowing to the central loghost. > For > >>> example: > >>>>>>> > >>>>>>> # syslog client > >>>>>>> #/var/log/syslog > >>>>>>> -rw------- 1 root root 1049189 Mar 16 01:13 > >>>>>>> failqueue-loghost2.00000002 > >>>>>>> -rw------- 1 root root 1048848 Mar 14 13:25 > >>>>>>> failqueue-loghost2.00000003 > >>>>>>> -rw------- 1 root root 1048648 Mar 14 17:20 > >>>>>>> failqueue-loghost2.00000004 > >>>>>>> -rw------- 1 root root 1049066 Mar 15 00:19 > >>>>>>> failqueue-loghost2.00000005 > >>>>>>> -rw------- 1 root root 1048619 Mar 15 00:27 > >>>>>>> failqueue-loghost2.00000006 > >>>>>>> -rw------- 1 root root 1048907 Mar 15 13:20 > >>>>>>> failqueue-loghost2.00000007 > >>>>>>> -rw------- 1 root root 949887 Mar 16 01:13 > >>> failqueue-loghost2.00000008 > >>>>>>> -rw------- 1 root root 1653 Mar 16 01:13 failqueue- > loghost2.qi > >>>>>>> > >>>>>>> Running rsyslog-5.6.4. > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> rsyslog mailing list > >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>> http://www.rsyslog.com > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com > >>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
