FYI: created https://github.com/rsyslog/rsyslog/issues/1867
2017-10-20 8:37 GMT+02:00 Rainer Gerhards <rgerha...@hq.adiscon.com>: > Mike, > > question: do you look at the error messages rsyslog emits? Or do you > throw them away (many distros do that by default)? I am asking because > I went through the debug log with the new information you gave. I see > these errors emitted by rsyslog's imjournal: > ``` > 'imjournal: couldn't seek to cursor > `s=dec6d981bf5647a2b6b7970597e4471d;i=455;b=b05da23ccaf04159888a061532615402;m=1337f528;t=55be6afe2d949;x=965813e66f54721f > sd_journal_next() failed: 'Success' > ``` > The second one is strange and most probably the root cause of the > missing information. > > Will be very interested to see what the log with the older version shows. > > In general, I strongly suggest to have a look at rsyslog error > messages, these can considerably ease your life ;-) > > Rainer > > 2017-10-19 21:23 GMT+02:00 Rainer Gerhards <rgerha...@hq.adiscon.com>: >> It would be great to have it as similar as possible. >> >> Sent from phone, thus brief. >> >> Am 19.10.2017 20:57 schrieb "Mike Schleif" <mike+rsys...@mdsresource.net>: >>> >>> Rainer, >>> >>> Yes, I respect your time. Since it is running with 8.29, I can keep this >>> running as-is for a week or so; but, I do need the update fixes asap. >>> >>> For debug log from working system, do you need any system reboot? >>> >>> If not, I can turn on debug in rsyslog.conf, then simple restart rsyslogd. >>> >>> Please, advise. Thank you. >>> >>> ~ Mike >>> >>> >>> >>> On Thu, Oct 19, 2017 at 1:35 PM, Rainer Gerhards >>> <rgerha...@hq.adiscon.com> >>> wrote: >>> >>> > I think David can probably answer that better. You need to check systemd >>> > and journal conf. >>> > >>> > But you said it works with an older version. Can you create a Debug log >>> > with that one as well so that I can compare? That would probably be >>> > useful. >>> > Again (due to time zone differences) I can look at this at earliest in >>> > roughly 12 hours - depending on what work has waiting for me in the >>> > morning. Having both logs by then would definitely be a plus. >>> > >>> > Rainer >>> > >>> > Sent from phone, thus brief. >>> > >>> > Am 19.10.2017 20:24 schrieb "Mike Schleif" >>> > <mike+rsys...@mdsresource.net>: >>> > >>> > > Rainer, >>> > > >>> > > Apparently, I wasn't explicit enough when submitting the debug log. >>> > > >>> > > You asked: Did something (systemd) steal the log socket? >>> > > >>> > > I don't know. How could I know? How can I find out? >>> > > >>> > > Please, advise. Thank you. >>> > > >>> > > ~ Mike >>> > > >>> > > >>> > > On Thu, Oct 19, 2017 at 1:18 PM, Rainer Gerhards < >>> > rgerha...@hq.adiscon.com >>> > > > >>> > > wrote: >>> > > >>> > > > Well it would have helped to have this information before wading >>> > through >>> > > > the log ;-). Now it needs to wait till tomorrow or Monday. >>> > > > >>> > > > Did something (systemd) steal the log socket? >>> > > > >>> > > > Räuber >>> > > > >>> > > > Sent from phone, thus brief. >>> > > > >>> > > > Am 19.10.2017 19:53 schrieb "Mike Schleif" < >>> > mike+rsys...@mdsresource.net >>> > > >: >>> > > > >>> > > > > Look at line: 32697 - That is the LAST line of debug as the system >>> > > booted >>> > > > > up. >>> > > > > >>> > > > > Now, look at the next line: 32698 - That is the first line after >>> > > > > the >>> > > > > sysadmin pressed Enter after typing "reboot." >>> > > > > >>> > > > > I don't understand the time encoding prior to the first colon (:) >>> > > > > of >>> > > each >>> > > > > line; but, this host was up for ten (10) minutes or more before >>> > backing >>> > > > out >>> > > > > of the update patches and reboot. >>> > > > > >>> > > > > How can I provide missing messages, when they are missing? >>> > > > > >>> > > > > The only way to get to this host is via SSH. During the period of >>> > > > > the >>> > > > debug >>> > > > > log, another sysadmin and I logged onto that host at least three >>> > > > > (3) >>> > > > times >>> > > > > each - not one write to /var/log/secure !?!? >>> > > > > >>> > > > > Yes, there are /var/log/* writes up until the system fully booted >>> > > > > - >>> > > then >>> > > > > nothing - until sysadmin pressed Enter, more than ten (10) minutes >>> > > later. >>> > > > > The ONLY /var/log/ files to get written to during that period were >>> > > > > /var/log/lastlog and /var/log/wtmp - NOT one other log was written >>> > > > > to >>> > > in >>> > > > > more than ten (10) minutes ... >>> > > > > >>> > > > > Please, advise. Thank you. >>> > > > > >>> > > > > ~ Mike >>> > > > > >>> > > > > >>> > > > > >>> > > > > On Thu, Oct 19, 2017 at 12:32 PM, Rainer Gerhards < >>> > > > > rgerha...@hq.adiscon.com> >>> > > > > wrote: >>> > > > > >>> > > > > > 2017-10-19 16:14 GMT+02:00 Mike Schleif < >>> > > mike+rsys...@mdsresource.net> >>> > > > : >>> > > > > > > Rainer, >>> > > > > > > >>> > > > > > > Debug attached. Full reboot follows each update and roll back. >>> > > > > > > >>> > > > > > > It looks like nothing under /var/log/ gets written to after >>> > reboot >>> > > > > > > complete, except lastlog and wtmp. >>> > > > > > >>> > > > > > mmhhh... I see at least writes to >>> > > > > > >>> > > > > > /var/log/messages: >>> > > > > > Reg/w0 : strm 0x7f81fc005290: stream.c: opened file >>> > > > > > '/var/log/messages' for WRITE as 12 >>> > > > > > Reg/w0 : strm 0x7f81fc005290: stream.c: file 12 write wrote >>> > > > > > 4041 >>> > > bytes >>> > > > > > >>> > > > > > from the embedded pstats, I see that no other action received >>> > > > > > messages. So far, everything looks ok. >>> > > > > > >>> > > > > > Can you point me to a specific message that you think is >>> > > > > > missing? I >>> > > > > > could then try to follow its flow inside the debug log. >>> > > > > > >>> > > > > > Rainer >>> > > > > > > >>> > > > > > > Event rsyslog-stats is not written to after boot complete. >>> > > > > > > >>> > > > > > > Please, advise. Thank you. >>> > > > > > > >>> > > > > > > ~ Mike >>> > > > > > > >>> > > > > > > >>> > > > > > > On Wed, Oct 18, 2017 at 10:43 AM, Rainer Gerhards < >>> > > > > > rgerha...@hq.adiscon.com> >>> > > > > > > wrote: >>> > > > > > > >>> > > > > > >> Do you mean some logs were written to and some not? >>> > > > > > >> >>> > > > > > >> If so, I need a Debug log to diagnose what is going on. >>> > > > > > >> >>> > > > > > >> Rainer >>> > > > > > >> >>> > > > > > >> Sent from phone, thus brief. >>> > > > > > >> >>> > > > > > >> Am 18.10.2017 17:36 schrieb "Mike Schleif" < >>> > > > > > mike+rsys...@mdsresource.net>: >>> > > > > > >> >>> > > > > > >> > # cat /etc/centos-release >>> > > > > > >> > CentOS Linux release 7.4.1708 (Core) >>> > > > > > >> > >>> > > > > > >> > >>> > > > > > >> > After yum updates yesterday (see below,) several logs no >>> > longer >>> > > > > > logged, >>> > > > > > >> > including /var/log/secure >>> > > > > > >> > >>> > > > > > >> > In the last hour, we rolled back that entire yum update, >>> > > > > > >> > and >>> > > > logging >>> > > > > > >> > appears to be as expected >>> > > > > > >> > >>> > > > > > >> > Please, advise. Thank you. >>> > > > > > >> > >>> > > > > > >> > ~ Mike >>> > > > > > >> > >>> > > > > > >> > >>> > > > > > >> > # yum history info 62 >>> > > > > > >> > Loaded plugins: fastestmirror >>> > > > > > >> > Transaction ID : 62 >>> > > > > > >> > Begin time : Tue Oct 17 07:42:51 2017 >>> > > > > > >> > Begin rpmdb : >>> > > > > > >> > 597:442a35918ca922c515d3f9bbc38cb3733341358a >>> > > > > > >> > End time : 07:43:00 2017 (9 seconds) >>> > > > > > >> > End rpmdb : >>> > > > > > >> > 597:f817c423ae76bafaafaab823cfca6d4030e069f0 >>> > > > > > >> > User : Jeffrey Reed <jreed> >>> > > > > > >> > Return-Code : Success >>> > > > > > >> > Command Line : update >>> > > > > > >> > Transaction performed with: >>> > > > > > >> > Installed rpm-4.11.3-25.el7.x86_64 >>> > > > > @base >>> > > > > > >> > Installed yum-3.4.3-154.el7.centos.noarch >>> > > > > @base >>> > > > > > >> > Installed yum-plugin-fastestmirror-1.1. >>> > 31-42.el7.noarch >>> > > > > @base >>> > > > > > >> > Packages Altered: >>> > > > > > >> > Updated epel-release-7-10.noarch >>> > > > > > >> > @epel >>> > > > > > >> > Update 7-11.noarch >>> > > > @epel-testing >>> > > > > > >> > Updated libfastjson4-0.99.5-1.el7.x86_64 >>> > > > @rsyslog_v8 >>> > > > > > >> > Update 0.99.7-1.el7.x86_64 >>> > > @rsyslog_v8 >>> > > > > > >> > Updated mysql-community-client-5.6.37-2.el7.x86_64 >>> > > > > > >> @mysql56-community >>> > > > > > >> > Update 5.6.38-2.el7.x86_64 >>> > > > > > @mysql56-community >>> > > > > > >> > Updated mysql-community-common-5.6.37-2.el7.x86_64 >>> > > > > > >> @mysql56-community >>> > > > > > >> > Update 5.6.38-2.el7.x86_64 >>> > > > > > @mysql56-community >>> > > > > > >> > Updated mysql-community-libs-5.6.37-2.el7.x86_64 >>> > > > > > >> @mysql56-community >>> > > > > > >> > Update 5.6.38-2.el7.x86_64 >>> > > > > > @mysql56-community >>> > > > > > >> > Updated rsyslog-8.29.0-2.el7.x86_64 >>> > > @rsyslog_v8 >>> > > > > > >> > Update 8.30.0-1.el7.x86_64 >>> > > @rsyslog_v8 >>> > > > > > >> > Updated rsyslog-mysql-8.29.0-2.el7.x86_64 >>> > > > @rsyslog_v8 >>> > > > > > >> > Update 8.30.0-1.el7.x86_64 >>> > > @rsyslog_v8 >>> > > > > > >> > history info >>> > > > > > >> > _______________________________________________ >>> > > > > > >> > rsyslog mailing list >>> > > > > > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >>> > > > > > >> > http://www.rsyslog.com/professional-services/ >>> > > > > > >> > What's up with rsyslog? Follow >>> > > > > > >> > https://twitter.com/rgerhards >>> > > > > > >> > NOTE WELL: This is a PUBLIC mailing list, posts are >>> > > > > > >> > ARCHIVED >>> > by >>> > > a >>> > > > > > myriad >>> > > > > > >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>> > POST >>> > > if >>> > > > > you >>> > > > > > >> > DON'T LIKE THAT. >>> > > > > > >> > >>> > > > > > >> _______________________________________________ >>> > > > > > >> rsyslog mailing list >>> > > > > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> > > > > > >> http://www.rsyslog.com/professional-services/ >>> > > > > > >> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> > > > > > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >>> > > > > > >> by >>> > a >>> > > > > myriad >>> > > > > > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>> > > > > > >> POST >>> > if >>> > > > you >>> > > > > > >> DON'T LIKE THAT. >>> > > > > > >> >>> > > > > > > >>> > > > > > > _______________________________________________ >>> > > > > > > rsyslog mailing list >>> > > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog >>> > > > > > > http://www.rsyslog.com/professional-services/ >>> > > > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards >>> > > > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >>> > > > > > > by a >>> > > > > myriad >>> > > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >>> > > > > > if >>> > > you >>> > > > > > DON'T LIKE THAT. >>> > > > > > _______________________________________________ >>> > > > > > rsyslog mailing list >>> > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog >>> > > > > > http://www.rsyslog.com/professional-services/ >>> > > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards >>> > > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by >>> > > > > > a >>> > > > myriad >>> > > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >>> > > > > > if >>> > > you >>> > > > > > DON'T LIKE THAT. >>> > > > > > >>> > > > > _______________________________________________ >>> > > > > rsyslog mailing list >>> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog >>> > > > > http://www.rsyslog.com/professional-services/ >>> > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards >>> > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> > > myriad >>> > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> > you >>> > > > > DON'T LIKE THAT. >>> > > > > >>> > > > _______________________________________________ >>> > > > rsyslog mailing list >>> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog >>> > > > http://www.rsyslog.com/professional-services/ >>> > > > What's up with rsyslog? Follow https://twitter.com/rgerhards >>> > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> > myriad >>> > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> > > > you >>> > > > DON'T LIKE THAT. >>> > > > >>> > > _______________________________________________ >>> > > rsyslog mailing list >>> > > http://lists.adiscon.net/mailman/listinfo/rsyslog >>> > > http://www.rsyslog.com/professional-services/ >>> > > What's up with rsyslog? Follow https://twitter.com/rgerhards >>> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> > > myriad >>> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> > > DON'T LIKE THAT. >>> > _______________________________________________ >>> > rsyslog mailing list >>> > http://lists.adiscon.net/mailman/listinfo/rsyslog >>> > http://www.rsyslog.com/professional-services/ >>> > What's up with rsyslog? Follow https://twitter.com/rgerhards >>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> > DON'T LIKE THAT. >>> > >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
