Mike, question: do you look at the error messages rsyslog emits? Or do you throw them away (many distros do that by default)? I am asking because I went through the debug log with the new information you gave. I see these errors emitted by rsyslog's imjournal: ``` 'imjournal: couldn't seek to cursor `s=dec6d981bf5647a2b6b7970597e4471d;i=455;b=b05da23ccaf04159888a061532615402;m=1337f528;t=55be6afe2d949;x=965813e66f54721f sd_journal_next() failed: 'Success' ``` The second one is strange and most probably the root cause of the missing information.
Will be very interested to see what the log with the older version shows. In general, I strongly suggest to have a look at rsyslog error messages, these can considerably ease your life ;-) Rainer 2017-10-19 21:23 GMT+02:00 Rainer Gerhards <rgerha...@hq.adiscon.com>: > It would be great to have it as similar as possible. > > Sent from phone, thus brief. > > Am 19.10.2017 20:57 schrieb "Mike Schleif" <mike+rsys...@mdsresource.net>: >> >> Rainer, >> >> Yes, I respect your time. Since it is running with 8.29, I can keep this >> running as-is for a week or so; but, I do need the update fixes asap. >> >> For debug log from working system, do you need any system reboot? >> >> If not, I can turn on debug in rsyslog.conf, then simple restart rsyslogd. >> >> Please, advise. Thank you. >> >> ~ Mike >> >> >> >> On Thu, Oct 19, 2017 at 1:35 PM, Rainer Gerhards >> <rgerha...@hq.adiscon.com> >> wrote: >> >> > I think David can probably answer that better. You need to check systemd >> > and journal conf. >> > >> > But you said it works with an older version. Can you create a Debug log >> > with that one as well so that I can compare? That would probably be >> > useful. >> > Again (due to time zone differences) I can look at this at earliest in >> > roughly 12 hours - depending on what work has waiting for me in the >> > morning. Having both logs by then would definitely be a plus. >> > >> > Rainer >> > >> > Sent from phone, thus brief. >> > >> > Am 19.10.2017 20:24 schrieb "Mike Schleif" >> > <mike+rsys...@mdsresource.net>: >> > >> > > Rainer, >> > > >> > > Apparently, I wasn't explicit enough when submitting the debug log. >> > > >> > > You asked: Did something (systemd) steal the log socket? >> > > >> > > I don't know. How could I know? How can I find out? >> > > >> > > Please, advise. Thank you. >> > > >> > > ~ Mike >> > > >> > > >> > > On Thu, Oct 19, 2017 at 1:18 PM, Rainer Gerhards < >> > rgerha...@hq.adiscon.com >> > > > >> > > wrote: >> > > >> > > > Well it would have helped to have this information before wading >> > through >> > > > the log ;-). Now it needs to wait till tomorrow or Monday. >> > > > >> > > > Did something (systemd) steal the log socket? >> > > > >> > > > Räuber >> > > > >> > > > Sent from phone, thus brief. >> > > > >> > > > Am 19.10.2017 19:53 schrieb "Mike Schleif" < >> > mike+rsys...@mdsresource.net >> > > >: >> > > > >> > > > > Look at line: 32697 - That is the LAST line of debug as the system >> > > booted >> > > > > up. >> > > > > >> > > > > Now, look at the next line: 32698 - That is the first line after >> > > > > the >> > > > > sysadmin pressed Enter after typing "reboot." >> > > > > >> > > > > I don't understand the time encoding prior to the first colon (:) >> > > > > of >> > > each >> > > > > line; but, this host was up for ten (10) minutes or more before >> > backing >> > > > out >> > > > > of the update patches and reboot. >> > > > > >> > > > > How can I provide missing messages, when they are missing? >> > > > > >> > > > > The only way to get to this host is via SSH. During the period of >> > > > > the >> > > > debug >> > > > > log, another sysadmin and I logged onto that host at least three >> > > > > (3) >> > > > times >> > > > > each - not one write to /var/log/secure !?!? >> > > > > >> > > > > Yes, there are /var/log/* writes up until the system fully booted >> > > > > - >> > > then >> > > > > nothing - until sysadmin pressed Enter, more than ten (10) minutes >> > > later. >> > > > > The ONLY /var/log/ files to get written to during that period were >> > > > > /var/log/lastlog and /var/log/wtmp - NOT one other log was written >> > > > > to >> > > in >> > > > > more than ten (10) minutes ... >> > > > > >> > > > > Please, advise. Thank you. >> > > > > >> > > > > ~ Mike >> > > > > >> > > > > >> > > > > >> > > > > On Thu, Oct 19, 2017 at 12:32 PM, Rainer Gerhards < >> > > > > rgerha...@hq.adiscon.com> >> > > > > wrote: >> > > > > >> > > > > > 2017-10-19 16:14 GMT+02:00 Mike Schleif < >> > > mike+rsys...@mdsresource.net> >> > > > : >> > > > > > > Rainer, >> > > > > > > >> > > > > > > Debug attached. Full reboot follows each update and roll back. >> > > > > > > >> > > > > > > It looks like nothing under /var/log/ gets written to after >> > reboot >> > > > > > > complete, except lastlog and wtmp. >> > > > > > >> > > > > > mmhhh... I see at least writes to >> > > > > > >> > > > > > /var/log/messages: >> > > > > > Reg/w0 : strm 0x7f81fc005290: stream.c: opened file >> > > > > > '/var/log/messages' for WRITE as 12 >> > > > > > Reg/w0 : strm 0x7f81fc005290: stream.c: file 12 write wrote >> > > > > > 4041 >> > > bytes >> > > > > > >> > > > > > from the embedded pstats, I see that no other action received >> > > > > > messages. So far, everything looks ok. >> > > > > > >> > > > > > Can you point me to a specific message that you think is >> > > > > > missing? I >> > > > > > could then try to follow its flow inside the debug log. >> > > > > > >> > > > > > Rainer >> > > > > > > >> > > > > > > Event rsyslog-stats is not written to after boot complete. >> > > > > > > >> > > > > > > Please, advise. Thank you. >> > > > > > > >> > > > > > > ~ Mike >> > > > > > > >> > > > > > > >> > > > > > > On Wed, Oct 18, 2017 at 10:43 AM, Rainer Gerhards < >> > > > > > rgerha...@hq.adiscon.com> >> > > > > > > wrote: >> > > > > > > >> > > > > > >> Do you mean some logs were written to and some not? >> > > > > > >> >> > > > > > >> If so, I need a Debug log to diagnose what is going on. >> > > > > > >> >> > > > > > >> Rainer >> > > > > > >> >> > > > > > >> Sent from phone, thus brief. >> > > > > > >> >> > > > > > >> Am 18.10.2017 17:36 schrieb "Mike Schleif" < >> > > > > > mike+rsys...@mdsresource.net>: >> > > > > > >> >> > > > > > >> > # cat /etc/centos-release >> > > > > > >> > CentOS Linux release 7.4.1708 (Core) >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > After yum updates yesterday (see below,) several logs no >> > longer >> > > > > > logged, >> > > > > > >> > including /var/log/secure >> > > > > > >> > >> > > > > > >> > In the last hour, we rolled back that entire yum update, >> > > > > > >> > and >> > > > logging >> > > > > > >> > appears to be as expected >> > > > > > >> > >> > > > > > >> > Please, advise. Thank you. >> > > > > > >> > >> > > > > > >> > ~ Mike >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > # yum history info 62 >> > > > > > >> > Loaded plugins: fastestmirror >> > > > > > >> > Transaction ID : 62 >> > > > > > >> > Begin time : Tue Oct 17 07:42:51 2017 >> > > > > > >> > Begin rpmdb : >> > > > > > >> > 597:442a35918ca922c515d3f9bbc38cb3733341358a >> > > > > > >> > End time : 07:43:00 2017 (9 seconds) >> > > > > > >> > End rpmdb : >> > > > > > >> > 597:f817c423ae76bafaafaab823cfca6d4030e069f0 >> > > > > > >> > User : Jeffrey Reed <jreed> >> > > > > > >> > Return-Code : Success >> > > > > > >> > Command Line : update >> > > > > > >> > Transaction performed with: >> > > > > > >> > Installed rpm-4.11.3-25.el7.x86_64 >> > > > > @base >> > > > > > >> > Installed yum-3.4.3-154.el7.centos.noarch >> > > > > @base >> > > > > > >> > Installed yum-plugin-fastestmirror-1.1. >> > 31-42.el7.noarch >> > > > > @base >> > > > > > >> > Packages Altered: >> > > > > > >> > Updated epel-release-7-10.noarch >> > > > > > >> > @epel >> > > > > > >> > Update 7-11.noarch >> > > > @epel-testing >> > > > > > >> > Updated libfastjson4-0.99.5-1.el7.x86_64 >> > > > @rsyslog_v8 >> > > > > > >> > Update 0.99.7-1.el7.x86_64 >> > > @rsyslog_v8 >> > > > > > >> > Updated mysql-community-client-5.6.37-2.el7.x86_64 >> > > > > > >> @mysql56-community >> > > > > > >> > Update 5.6.38-2.el7.x86_64 >> > > > > > @mysql56-community >> > > > > > >> > Updated mysql-community-common-5.6.37-2.el7.x86_64 >> > > > > > >> @mysql56-community >> > > > > > >> > Update 5.6.38-2.el7.x86_64 >> > > > > > @mysql56-community >> > > > > > >> > Updated mysql-community-libs-5.6.37-2.el7.x86_64 >> > > > > > >> @mysql56-community >> > > > > > >> > Update 5.6.38-2.el7.x86_64 >> > > > > > @mysql56-community >> > > > > > >> > Updated rsyslog-8.29.0-2.el7.x86_64 >> > > @rsyslog_v8 >> > > > > > >> > Update 8.30.0-1.el7.x86_64 >> > > @rsyslog_v8 >> > > > > > >> > Updated rsyslog-mysql-8.29.0-2.el7.x86_64 >> > > > @rsyslog_v8 >> > > > > > >> > Update 8.30.0-1.el7.x86_64 >> > > @rsyslog_v8 >> > > > > > >> > history info >> > > > > > >> > _______________________________________________ >> > > > > > >> > rsyslog mailing list >> > > > > > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > > > > >> > http://www.rsyslog.com/professional-services/ >> > > > > > >> > What's up with rsyslog? Follow >> > > > > > >> > https://twitter.com/rgerhards >> > > > > > >> > NOTE WELL: This is a PUBLIC mailing list, posts are >> > > > > > >> > ARCHIVED >> > by >> > > a >> > > > > > myriad >> > > > > > >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >> > POST >> > > if >> > > > > you >> > > > > > >> > DON'T LIKE THAT. >> > > > > > >> > >> > > > > > >> _______________________________________________ >> > > > > > >> rsyslog mailing list >> > > > > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > > > > >> http://www.rsyslog.com/professional-services/ >> > > > > > >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> > > > > > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >> > > > > > >> by >> > a >> > > > > myriad >> > > > > > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >> > > > > > >> POST >> > if >> > > > you >> > > > > > >> DON'T LIKE THAT. >> > > > > > >> >> > > > > > > >> > > > > > > _______________________________________________ >> > > > > > > rsyslog mailing list >> > > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > > > > > http://www.rsyslog.com/professional-services/ >> > > > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > > > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >> > > > > > > by a >> > > > > myriad >> > > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >> > > > > > if >> > > you >> > > > > > DON'T LIKE THAT. >> > > > > > _______________________________________________ >> > > > > > rsyslog mailing list >> > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > > > > http://www.rsyslog.com/professional-services/ >> > > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by >> > > > > > a >> > > > myriad >> > > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >> > > > > > if >> > > you >> > > > > > DON'T LIKE THAT. >> > > > > > >> > > > > _______________________________________________ >> > > > > rsyslog mailing list >> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > > > http://www.rsyslog.com/professional-services/ >> > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> > > myriad >> > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> > you >> > > > > DON'T LIKE THAT. >> > > > > >> > > > _______________________________________________ >> > > > rsyslog mailing list >> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > > http://www.rsyslog.com/professional-services/ >> > > > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> > myriad >> > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> > > > you >> > > > DON'T LIKE THAT. >> > > > >> > > _______________________________________________ >> > > rsyslog mailing list >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > http://www.rsyslog.com/professional-services/ >> > > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> > > myriad >> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > > DON'T LIKE THAT. >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > DON'T LIKE THAT. >> > >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
