Earlier this year, I posted about our problem with this host, and delayed writes to Mysql databases. I noted, at that time, this imjournal error, as well as intermittent cached files in WorkDirectory for these ommysql queues.
After initial queries from the list, I answered, and received no answers. We continues to struggle with fifteen (15) minute db write delays periodically throughout the day. Interestingly enough, yesterday and today, after boot up, the queue files are there again: # ls -alrt /var/lib/rsyslog/ total 840 drwxr-xr-x. 34 root root 4096 Oct 3 11:30 .. -rw------- 1 root root 406707 Oct 20 08:06 ZenossQueue.00000001 -rw------- 1 root root 442098 Oct 20 08:06 SIEMQueue.00000001 -rw------- 1 root root 121 Oct 20 08:17 imjournal.state drwx------. 2 root root 80 Oct 20 08:17 . # /bin/systemctl restart rsyslog # ls -alrt /var/lib/rsyslog/ total 8 drwxr-xr-x. 34 root root 4096 Oct 3 11:30 .. -rw------- 1 root root 121 Oct 20 08:17 imjournal.state drwx------. 2 root root 28 Oct 20 08:17 . Please, advise. Thank you. ~ Mike On Fri, Oct 20, 2017 at 1:37 AM, Rainer Gerhards <rgerha...@hq.adiscon.com> wrote: > Mike, > > question: do you look at the error messages rsyslog emits? Or do you > throw them away (many distros do that by default)? I am asking because > I went through the debug log with the new information you gave. I see > these errors emitted by rsyslog's imjournal: > ``` > 'imjournal: couldn't seek to cursor > `s=dec6d981bf5647a2b6b7970597e4471d;i=455;b=b05da23ccaf04159888a0615326154 > 02;m=1337f528;t=55be6afe2d949;x=965813e66f54721f > sd_journal_next() failed: 'Success' > ``` > The second one is strange and most probably the root cause of the > missing information. > > Will be very interested to see what the log with the older version shows. > > In general, I strongly suggest to have a look at rsyslog error > messages, these can considerably ease your life ;-) > > Rainer > > 2017-10-19 21:23 GMT+02:00 Rainer Gerhards <rgerha...@hq.adiscon.com>: > > It would be great to have it as similar as possible. > > > > Sent from phone, thus brief. > > > > Am 19.10.2017 20:57 schrieb "Mike Schleif" <mike+rsys...@mdsresource.net > >: > >> > >> Rainer, > >> > >> Yes, I respect your time. Since it is running with 8.29, I can keep this > >> running as-is for a week or so; but, I do need the update fixes asap. > >> > >> For debug log from working system, do you need any system reboot? > >> > >> If not, I can turn on debug in rsyslog.conf, then simple restart > rsyslogd. > >> > >> Please, advise. Thank you. > >> > >> ~ Mike > >> > >> > >> > >> On Thu, Oct 19, 2017 at 1:35 PM, Rainer Gerhards > >> <rgerha...@hq.adiscon.com> > >> wrote: > >> > >> > I think David can probably answer that better. You need to check > systemd > >> > and journal conf. > >> > > >> > But you said it works with an older version. Can you create a Debug > log > >> > with that one as well so that I can compare? That would probably be > >> > useful. > >> > Again (due to time zone differences) I can look at this at earliest in > >> > roughly 12 hours - depending on what work has waiting for me in the > >> > morning. Having both logs by then would definitely be a plus. > >> > > >> > Rainer > >> > > >> > Sent from phone, thus brief. > >> > > >> > Am 19.10.2017 20:24 schrieb "Mike Schleif" > >> > <mike+rsys...@mdsresource.net>: > >> > > >> > > Rainer, > >> > > > >> > > Apparently, I wasn't explicit enough when submitting the debug log. > >> > > > >> > > You asked: Did something (systemd) steal the log socket? > >> > > > >> > > I don't know. How could I know? How can I find out? > >> > > > >> > > Please, advise. Thank you. > >> > > > >> > > ~ Mike > >> > > > >> > > > >> > > On Thu, Oct 19, 2017 at 1:18 PM, Rainer Gerhards < > >> > rgerha...@hq.adiscon.com > >> > > > > >> > > wrote: > >> > > > >> > > > Well it would have helped to have this information before wading > >> > through > >> > > > the log ;-). Now it needs to wait till tomorrow or Monday. > >> > > > > >> > > > Did something (systemd) steal the log socket? > >> > > > > >> > > > Räuber > >> > > > > >> > > > Sent from phone, thus brief. > >> > > > > >> > > > Am 19.10.2017 19:53 schrieb "Mike Schleif" < > >> > mike+rsys...@mdsresource.net > >> > > >: > >> > > > > >> > > > > Look at line: 32697 - That is the LAST line of debug as the > system > >> > > booted > >> > > > > up. > >> > > > > > >> > > > > Now, look at the next line: 32698 - That is the first line after > >> > > > > the > >> > > > > sysadmin pressed Enter after typing "reboot." > >> > > > > > >> > > > > I don't understand the time encoding prior to the first colon > (:) > >> > > > > of > >> > > each > >> > > > > line; but, this host was up for ten (10) minutes or more before > >> > backing > >> > > > out > >> > > > > of the update patches and reboot. > >> > > > > > >> > > > > How can I provide missing messages, when they are missing? > >> > > > > > >> > > > > The only way to get to this host is via SSH. During the period > of > >> > > > > the > >> > > > debug > >> > > > > log, another sysadmin and I logged onto that host at least three > >> > > > > (3) > >> > > > times > >> > > > > each - not one write to /var/log/secure !?!? > >> > > > > > >> > > > > Yes, there are /var/log/* writes up until the system fully > booted > >> > > > > - > >> > > then > >> > > > > nothing - until sysadmin pressed Enter, more than ten (10) > minutes > >> > > later. > >> > > > > The ONLY /var/log/ files to get written to during that period > were > >> > > > > /var/log/lastlog and /var/log/wtmp - NOT one other log was > written > >> > > > > to > >> > > in > >> > > > > more than ten (10) minutes ... > >> > > > > > >> > > > > Please, advise. Thank you. > >> > > > > > >> > > > > ~ Mike > >> > > > > > >> > > > > > >> > > > > > >> > > > > On Thu, Oct 19, 2017 at 12:32 PM, Rainer Gerhards < > >> > > > > rgerha...@hq.adiscon.com> > >> > > > > wrote: > >> > > > > > >> > > > > > 2017-10-19 16:14 GMT+02:00 Mike Schleif < > >> > > mike+rsys...@mdsresource.net> > >> > > > : > >> > > > > > > Rainer, > >> > > > > > > > >> > > > > > > Debug attached. Full reboot follows each update and roll > back. > >> > > > > > > > >> > > > > > > It looks like nothing under /var/log/ gets written to after > >> > reboot > >> > > > > > > complete, except lastlog and wtmp. > >> > > > > > > >> > > > > > mmhhh... I see at least writes to > >> > > > > > > >> > > > > > /var/log/messages: > >> > > > > > Reg/w0 : strm 0x7f81fc005290: stream.c: opened file > >> > > > > > '/var/log/messages' for WRITE as 12 > >> > > > > > Reg/w0 : strm 0x7f81fc005290: stream.c: file 12 write wrote > >> > > > > > 4041 > >> > > bytes > >> > > > > > > >> > > > > > from the embedded pstats, I see that no other action received > >> > > > > > messages. So far, everything looks ok. > >> > > > > > > >> > > > > > Can you point me to a specific message that you think is > >> > > > > > missing? I > >> > > > > > could then try to follow its flow inside the debug log. > >> > > > > > > >> > > > > > Rainer > >> > > > > > > > >> > > > > > > Event rsyslog-stats is not written to after boot complete. > >> > > > > > > > >> > > > > > > Please, advise. Thank you. > >> > > > > > > > >> > > > > > > ~ Mike > >> > > > > > > > >> > > > > > > > >> > > > > > > On Wed, Oct 18, 2017 at 10:43 AM, Rainer Gerhards < > >> > > > > > rgerha...@hq.adiscon.com> > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > >> Do you mean some logs were written to and some not? > >> > > > > > >> > >> > > > > > >> If so, I need a Debug log to diagnose what is going on. > >> > > > > > >> > >> > > > > > >> Rainer > >> > > > > > >> > >> > > > > > >> Sent from phone, thus brief. > >> > > > > > >> > >> > > > > > >> Am 18.10.2017 17:36 schrieb "Mike Schleif" < > >> > > > > > mike+rsys...@mdsresource.net>: > >> > > > > > >> > >> > > > > > >> > # cat /etc/centos-release > >> > > > > > >> > CentOS Linux release 7.4.1708 (Core) > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > After yum updates yesterday (see below,) several logs no > >> > longer > >> > > > > > logged, > >> > > > > > >> > including /var/log/secure > >> > > > > > >> > > >> > > > > > >> > In the last hour, we rolled back that entire yum update, > >> > > > > > >> > and > >> > > > logging > >> > > > > > >> > appears to be as expected > >> > > > > > >> > > >> > > > > > >> > Please, advise. Thank you. > >> > > > > > >> > > >> > > > > > >> > ~ Mike > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > # yum history info 62 > >> > > > > > >> > Loaded plugins: fastestmirror > >> > > > > > >> > Transaction ID : 62 > >> > > > > > >> > Begin time : Tue Oct 17 07:42:51 2017 > >> > > > > > >> > Begin rpmdb : > >> > > > > > >> > 597:442a35918ca922c515d3f9bbc38cb3733341358a > >> > > > > > >> > End time : 07:43:00 2017 (9 seconds) > >> > > > > > >> > End rpmdb : > >> > > > > > >> > 597:f817c423ae76bafaafaab823cfca6d4030e069f0 > >> > > > > > >> > User : Jeffrey Reed <jreed> > >> > > > > > >> > Return-Code : Success > >> > > > > > >> > Command Line : update > >> > > > > > >> > Transaction performed with: > >> > > > > > >> > Installed rpm-4.11.3-25.el7.x86_64 > >> > > > > @base > >> > > > > > >> > Installed yum-3.4.3-154.el7.centos.noarch > >> > > > > @base > >> > > > > > >> > Installed yum-plugin-fastestmirror-1.1. > >> > 31-42.el7.noarch > >> > > > > @base > >> > > > > > >> > Packages Altered: > >> > > > > > >> > Updated epel-release-7-10.noarch > >> > > > > > >> > @epel > >> > > > > > >> > Update 7-11.noarch > >> > > > @epel-testing > >> > > > > > >> > Updated libfastjson4-0.99.5-1.el7.x86_64 > >> > > > @rsyslog_v8 > >> > > > > > >> > Update 0.99.7-1.el7.x86_64 > >> > > @rsyslog_v8 > >> > > > > > >> > Updated mysql-community-client-5.6.37-2.el7.x86_64 > >> > > > > > >> @mysql56-community > >> > > > > > >> > Update 5.6.38-2.el7.x86_64 > >> > > > > > @mysql56-community > >> > > > > > >> > Updated mysql-community-common-5.6.37-2.el7.x86_64 > >> > > > > > >> @mysql56-community > >> > > > > > >> > Update 5.6.38-2.el7.x86_64 > >> > > > > > @mysql56-community > >> > > > > > >> > Updated mysql-community-libs-5.6.37-2.el7.x86_64 > >> > > > > > >> @mysql56-community > >> > > > > > >> > Update 5.6.38-2.el7.x86_64 > >> > > > > > @mysql56-community > >> > > > > > >> > Updated rsyslog-8.29.0-2.el7.x86_64 > >> > > @rsyslog_v8 > >> > > > > > >> > Update 8.30.0-1.el7.x86_64 > >> > > @rsyslog_v8 > >> > > > > > >> > Updated rsyslog-mysql-8.29.0-2.el7.x86_64 > >> > > > @rsyslog_v8 > >> > > > > > >> > Update 8.30.0-1.el7.x86_64 > >> > > @rsyslog_v8 > >> > > > > > >> > history info > >> > > > > > >> > _______________________________________________ > >> > > > > > >> > rsyslog mailing list > >> > > > > > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > > > > > >> > http://www.rsyslog.com/professional-services/ > >> > > > > > >> > What's up with rsyslog? Follow > >> > > > > > >> > https://twitter.com/rgerhards > >> > > > > > >> > NOTE WELL: This is a PUBLIC mailing list, posts are > >> > > > > > >> > ARCHIVED > >> > by > >> > > a > >> > > > > > myriad > >> > > > > > >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO > NOT > >> > POST > >> > > if > >> > > > > you > >> > > > > > >> > DON'T LIKE THAT. > >> > > > > > >> > > >> > > > > > >> _______________________________________________ > >> > > > > > >> rsyslog mailing list > >> > > > > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > > > > > >> http://www.rsyslog.com/professional-services/ > >> > > > > > >> What's up with rsyslog? Follow > https://twitter.com/rgerhards > >> > > > > > >> NOTE WELL: This is a PUBLIC mailing list, posts are > ARCHIVED > >> > > > > > >> by > >> > a > >> > > > > myriad > >> > > > > > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > >> > > > > > >> POST > >> > if > >> > > > you > >> > > > > > >> DON'T LIKE THAT. > >> > > > > > >> > >> > > > > > > > >> > > > > > > _______________________________________________ > >> > > > > > > rsyslog mailing list > >> > > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > > > > > > http://www.rsyslog.com/professional-services/ > >> > > > > > > What's up with rsyslog? Follow > https://twitter.com/rgerhards > >> > > > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED > >> > > > > > > by a > >> > > > > myriad > >> > > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST > >> > > > > > if > >> > > you > >> > > > > > DON'T LIKE THAT. > >> > > > > > _______________________________________________ > >> > > > > > rsyslog mailing list > >> > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > > > > > http://www.rsyslog.com/professional-services/ > >> > > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED > by > >> > > > > > a > >> > > > myriad > >> > > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST > >> > > > > > if > >> > > you > >> > > > > > DON'T LIKE THAT. > >> > > > > > > >> > > > > _______________________________________________ > >> > > > > rsyslog mailing list > >> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > > > > http://www.rsyslog.com/professional-services/ > >> > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by > a > >> > > myriad > >> > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if > >> > you > >> > > > > DON'T LIKE THAT. > >> > > > > > >> > > > _______________________________________________ > >> > > > rsyslog mailing list > >> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > > > http://www.rsyslog.com/professional-services/ > >> > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> > myriad > >> > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > >> > > > you > >> > > > DON'T LIKE THAT. > >> > > > > >> > > _______________________________________________ > >> > > rsyslog mailing list > >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > > http://www.rsyslog.com/professional-services/ > >> > > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> > > myriad > >> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >> > > DON'T LIKE THAT. > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com/professional-services/ > >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> > DON'T LIKE THAT. > >> > > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T > >> LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
