date:20171019

Re: [rsyslog] Updates 8.29 -> 8.30 broke several logs

2017-10-19 Thread Mike Schleif

Rainer,

Apparently, I wasn't explicit enough when submitting the debug log.

You asked: Did something (systemd) steal the log socket?

I don't know. How could I know? How can I find out?

Please, advise. Thank you.

~ Mike


On Thu, Oct 19, 2017 at 1:18 PM, Rainer Gerhards 
wrote:

> Well it would have helped to have this information before wading through
> the log ;-). Now it needs to wait till tomorrow or Monday.
>
> Did something (systemd) steal the log socket?
>
> Räuber
>
> Sent from phone, thus brief.
>
> Am 19.10.2017 19:53 schrieb "Mike Schleif" :
>
> > Look at line: 32697 - That is the LAST line of debug as the system booted
> > up.
> >
> > Now, look at the next line: 32698 - That is the first line after the
> > sysadmin pressed Enter after typing "reboot."
> >
> > I don't understand the time encoding prior to the first colon (:) of each
> > line; but, this host was up for ten (10) minutes or more before backing
> out
> > of the update patches and reboot.
> >
> > How can I provide missing messages, when they are missing?
> >
> > The only way to get to this host is via SSH. During the period of the
> debug
> > log, another sysadmin and I logged onto that host at least three (3)
> times
> > each - not one write to /var/log/secure !?!?
> >
> > Yes, there are /var/log/* writes up until the system fully booted - then
> > nothing - until sysadmin pressed Enter, more than ten (10) minutes later.
> > The ONLY /var/log/ files to get written to during that period were
> > /var/log/lastlog and /var/log/wtmp - NOT one other log was written to in
> > more than ten (10) minutes ...
> >
> > Please, advise. Thank you.
> >
> > ~ Mike
> >
> >
> >
> > On Thu, Oct 19, 2017 at 12:32 PM, Rainer Gerhards <
> > rgerha...@hq.adiscon.com>
> > wrote:
> >
> > > 2017-10-19 16:14 GMT+02:00 Mike Schleif 
> :
> > > > Rainer,
> > > >
> > > > Debug attached. Full reboot follows each update and roll back.
> > > >
> > > > It looks like nothing under /var/log/ gets written to after reboot
> > > > complete, except lastlog and wtmp.
> > >
> > > mmhhh... I see at least writes to
> > >
> > > /var/log/messages:
> > > Reg/w0  : strm 0x7f81fc005290: stream.c: opened file
> > > '/var/log/messages' for WRITE as 12
> > > Reg/w0  : strm 0x7f81fc005290: stream.c: file 12 write wrote 4041 bytes
> > >
> > > from the embedded pstats, I see that no other action received
> > > messages. So far, everything looks ok.
> > >
> > > Can you point me to a specific message that you think is missing? I
> > > could then try to follow its flow inside the debug log.
> > >
> > > Rainer
> > > >
> > > > Event rsyslog-stats is not written to after boot complete.
> > > >
> > > > Please, advise. Thank you.
> > > >
> > > > ~ Mike
> > > >
> > > >
> > > > On Wed, Oct 18, 2017 at 10:43 AM, Rainer Gerhards <
> > > rgerha...@hq.adiscon.com>
> > > > wrote:
> > > >
> > > >> Do you mean some logs were written to and some not?
> > > >>
> > > >> If so, I need a Debug log to diagnose what is going on.
> > > >>
> > > >> Rainer
> > > >>
> > > >> Sent from phone, thus brief.
> > > >>
> > > >> Am 18.10.2017 17:36 schrieb "Mike Schleif" <
> > > mike+rsys...@mdsresource.net>:
> > > >>
> > > >> > # cat /etc/centos-release
> > > >> > CentOS Linux release 7.4.1708 (Core)
> > > >> >
> > > >> >
> > > >> > After yum updates yesterday (see below,) several logs no longer
> > > logged,
> > > >> > including /var/log/secure
> > > >> >
> > > >> > In the last hour, we rolled back that entire yum update, and
> logging
> > > >> > appears to be as expected
> > > >> >
> > > >> > Please, advise. Thank you.
> > > >> >
> > > >> > ~ Mike
> > > >> >
> > > >> >
> > > >> > # yum history info 62
> > > >> > Loaded plugins: fastestmirror
> > > >> > Transaction ID : 62
> > > >> > Begin time : Tue Oct 17 07:42:51 2017
> > > >> > Begin rpmdb: 597:442a35918ca922c515d3f9bbc38cb3733341358a
> > > >> > End time   :07:43:00 2017 (9 seconds)
> > > >> > End rpmdb  : 597:f817c423ae76bafaafaab823cfca6d4030e069f0
> > > >> > User   : Jeffrey Reed 
> > > >> > Return-Code: Success
> > > >> > Command Line   : update
> > > >> > Transaction performed with:
> > > >> > Installed rpm-4.11.3-25.el7.x86_64
> > @base
> > > >> > Installed yum-3.4.3-154.el7.centos.noarch
> >  @base
> > > >> > Installed yum-plugin-fastestmirror-1.1.31-42.el7.noarch
> > @base
> > > >> > Packages Altered:
> > > >> > Updated epel-release-7-10.noarch   @epel
> > > >> > Update   7-11.noarch
>  @epel-testing
> > > >> > Updated libfastjson4-0.99.5-1.el7.x86_64
>  @rsyslog_v8
> > > >> > Update   0.99.7-1.el7.x86_64   @rsyslog_v8
> > > >> > Updated mysql-community-client-5.6.37-2.el7.x86_64
> > > >> @mysql56-community
> > > >> > Update 5.6.38-2.el7.x86_64
> > > @mysql56-community
> > > >> > Updated

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-19 Thread David Lang


On Thu, 19 Oct 2017, deoren wrote:


On 10/18/2017 8:10 PM, David Lang wrote:

On Wed, 18 Oct 2017, deoren wrote:


On 10/18/2017 3:15 PM, David Lang wrote:

On Wed, 18 Oct 2017, deoren wrote:


On 10/18/2017 1:36 PM, David Lang wrote:

On Wed, 18 Oct 2017, deoren wrote:
Since the sender and receiver in this are both the latest versions of 
rsyslog (with the plan for the setup to remain that way), can I scale 
the accepted message size values to properly accommodate non-standard 
message sizes (delivered via JSON payloads)?


up to 128K should not be a problem, I believe that to scale the message 
size >128K you need to change a setting in the source.


Do you have experience delivering messages that large? I wonder whether 
I'm going about this the right way.


I wasn't using relp, but I did see logs hit 128k and get truncated a few 
times.


Do you use it now? If not, is it because it lacks a feature you need?


I changed jobs, and have not yet setup rsyslog here :-)

RELP has it's place, but most of the time I'm willing to loose some logs under 
rare failure conditions and so haven't bothered to use it.


large maxmessagesize leads to wasted memory in rsyslog, but nothing more 
severe than that.


Thanks for confirming. I'll likely go ahead and increase that value to 128K 
to see if the problem goes away.



if your maxmessagesize was 64k, that should not have been a problem.


Acknowledged. Do you know if all inputs honor the global() maxmessagesize 
value, or only certain ones? Does that value need to be specified using the 
legacy configuration syntax?


That's really a question Rainer will need to answer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Updates 8.29 -> 8.30 broke several logs

2017-10-19 Thread Rainer Gerhards

I think David can probably answer that better. You need to check systemd
and journal conf.

But you said it works with an older version. Can you create a Debug log
with that one as well so that I can compare? That would probably be useful.
Again (due to time zone differences) I can look at this at earliest in
roughly 12 hours - depending on what work has waiting for me in the
morning. Having both logs by then would definitely be a plus.

Rainer

Sent from phone, thus brief.

Am 19.10.2017 20:24 schrieb "Mike Schleif" :

> Rainer,
>
> Apparently, I wasn't explicit enough when submitting the debug log.
>
> You asked: Did something (systemd) steal the log socket?
>
> I don't know. How could I know? How can I find out?
>
> Please, advise. Thank you.
>
> ~ Mike
>
>
> On Thu, Oct 19, 2017 at 1:18 PM, Rainer Gerhards  >
> wrote:
>
> > Well it would have helped to have this information before wading through
> > the log ;-). Now it needs to wait till tomorrow or Monday.
> >
> > Did something (systemd) steal the log socket?
> >
> > Räuber
> >
> > Sent from phone, thus brief.
> >
> > Am 19.10.2017 19:53 schrieb "Mike Schleif"  >:
> >
> > > Look at line: 32697 - That is the LAST line of debug as the system
> booted
> > > up.
> > >
> > > Now, look at the next line: 32698 - That is the first line after the
> > > sysadmin pressed Enter after typing "reboot."
> > >
> > > I don't understand the time encoding prior to the first colon (:) of
> each
> > > line; but, this host was up for ten (10) minutes or more before backing
> > out
> > > of the update patches and reboot.
> > >
> > > How can I provide missing messages, when they are missing?
> > >
> > > The only way to get to this host is via SSH. During the period of the
> > debug
> > > log, another sysadmin and I logged onto that host at least three (3)
> > times
> > > each - not one write to /var/log/secure !?!?
> > >
> > > Yes, there are /var/log/* writes up until the system fully booted -
> then
> > > nothing - until sysadmin pressed Enter, more than ten (10) minutes
> later.
> > > The ONLY /var/log/ files to get written to during that period were
> > > /var/log/lastlog and /var/log/wtmp - NOT one other log was written to
> in
> > > more than ten (10) minutes ...
> > >
> > > Please, advise. Thank you.
> > >
> > > ~ Mike
> > >
> > >
> > >
> > > On Thu, Oct 19, 2017 at 12:32 PM, Rainer Gerhards <
> > > rgerha...@hq.adiscon.com>
> > > wrote:
> > >
> > > > 2017-10-19 16:14 GMT+02:00 Mike Schleif <
> mike+rsys...@mdsresource.net>
> > :
> > > > > Rainer,
> > > > >
> > > > > Debug attached. Full reboot follows each update and roll back.
> > > > >
> > > > > It looks like nothing under /var/log/ gets written to after reboot
> > > > > complete, except lastlog and wtmp.
> > > >
> > > > mmhhh... I see at least writes to
> > > >
> > > > /var/log/messages:
> > > > Reg/w0  : strm 0x7f81fc005290: stream.c: opened file
> > > > '/var/log/messages' for WRITE as 12
> > > > Reg/w0  : strm 0x7f81fc005290: stream.c: file 12 write wrote 4041
> bytes
> > > >
> > > > from the embedded pstats, I see that no other action received
> > > > messages. So far, everything looks ok.
> > > >
> > > > Can you point me to a specific message that you think is missing? I
> > > > could then try to follow its flow inside the debug log.
> > > >
> > > > Rainer
> > > > >
> > > > > Event rsyslog-stats is not written to after boot complete.
> > > > >
> > > > > Please, advise. Thank you.
> > > > >
> > > > > ~ Mike
> > > > >
> > > > >
> > > > > On Wed, Oct 18, 2017 at 10:43 AM, Rainer Gerhards <
> > > > rgerha...@hq.adiscon.com>
> > > > > wrote:
> > > > >
> > > > >> Do you mean some logs were written to and some not?
> > > > >>
> > > > >> If so, I need a Debug log to diagnose what is going on.
> > > > >>
> > > > >> Rainer
> > > > >>
> > > > >> Sent from phone, thus brief.
> > > > >>
> > > > >> Am 18.10.2017 17:36 schrieb "Mike Schleif" <
> > > > mike+rsys...@mdsresource.net>:
> > > > >>
> > > > >> > # cat /etc/centos-release
> > > > >> > CentOS Linux release 7.4.1708 (Core)
> > > > >> >
> > > > >> >
> > > > >> > After yum updates yesterday (see below,) several logs no longer
> > > > logged,
> > > > >> > including /var/log/secure
> > > > >> >
> > > > >> > In the last hour, we rolled back that entire yum update, and
> > logging
> > > > >> > appears to be as expected
> > > > >> >
> > > > >> > Please, advise. Thank you.
> > > > >> >
> > > > >> > ~ Mike
> > > > >> >
> > > > >> >
> > > > >> > # yum history info 62
> > > > >> > Loaded plugins: fastestmirror
> > > > >> > Transaction ID : 62
> > > > >> > Begin time : Tue Oct 17 07:42:51 2017
> > > > >> > Begin rpmdb: 597:442a35918ca922c515d3f9bbc38cb3733341358a
> > > > >> > End time   :07:43:00 2017 (9 seconds)
> > > > >> > End rpmdb  : 597:f817c423ae76bafaafaab823cfca6d4030e069f0
> > > > >> > User   : Jeffrey Reed 
> > > > >> > Return-Code: Success
>

Re: [rsyslog] Updates 8.29 -> 8.30 broke several logs

2017-10-19 Thread Rainer Gerhards

It would be great to have it as similar as possible.

Sent from phone, thus brief.

Am 19.10.2017 20:57 schrieb "Mike Schleif" :

> Rainer,
>
> Yes, I respect your time. Since it is running with 8.29, I can keep this
> running as-is for a week or so; but, I do need the update fixes asap.
>
> For debug log from working system, do you need any system reboot?
>
> If not, I can turn on debug in rsyslog.conf, then simple restart rsyslogd.
>
> Please, advise. Thank you.
>
> ~ Mike
>
>
>
> On Thu, Oct 19, 2017 at 1:35 PM, Rainer Gerhards  >
> wrote:
>
> > I think David can probably answer that better. You need to check systemd
> > and journal conf.
> >
> > But you said it works with an older version. Can you create a Debug log
> > with that one as well so that I can compare? That would probably be
> useful.
> > Again (due to time zone differences) I can look at this at earliest in
> > roughly 12 hours - depending on what work has waiting for me in the
> > morning. Having both logs by then would definitely be a plus.
> >
> > Rainer
> >
> > Sent from phone, thus brief.
> >
> > Am 19.10.2017 20:24 schrieb "Mike Schleif"  >:
> >
> > > Rainer,
> > >
> > > Apparently, I wasn't explicit enough when submitting the debug log.
> > >
> > > You asked: Did something (systemd) steal the log socket?
> > >
> > > I don't know. How could I know? How can I find out?
> > >
> > > Please, advise. Thank you.
> > >
> > > ~ Mike
> > >
> > >
> > > On Thu, Oct 19, 2017 at 1:18 PM, Rainer Gerhards <
> > rgerha...@hq.adiscon.com
> > > >
> > > wrote:
> > >
> > > > Well it would have helped to have this information before wading
> > through
> > > > the log ;-). Now it needs to wait till tomorrow or Monday.
> > > >
> > > > Did something (systemd) steal the log socket?
> > > >
> > > > Räuber
> > > >
> > > > Sent from phone, thus brief.
> > > >
> > > > Am 19.10.2017 19:53 schrieb "Mike Schleif" <
> > mike+rsys...@mdsresource.net
> > > >:
> > > >
> > > > > Look at line: 32697 - That is the LAST line of debug as the system
> > > booted
> > > > > up.
> > > > >
> > > > > Now, look at the next line: 32698 - That is the first line after
> the
> > > > > sysadmin pressed Enter after typing "reboot."
> > > > >
> > > > > I don't understand the time encoding prior to the first colon (:)
> of
> > > each
> > > > > line; but, this host was up for ten (10) minutes or more before
> > backing
> > > > out
> > > > > of the update patches and reboot.
> > > > >
> > > > > How can I provide missing messages, when they are missing?
> > > > >
> > > > > The only way to get to this host is via SSH. During the period of
> the
> > > > debug
> > > > > log, another sysadmin and I logged onto that host at least three
> (3)
> > > > times
> > > > > each - not one write to /var/log/secure !?!?
> > > > >
> > > > > Yes, there are /var/log/* writes up until the system fully booted -
> > > then
> > > > > nothing - until sysadmin pressed Enter, more than ten (10) minutes
> > > later.
> > > > > The ONLY /var/log/ files to get written to during that period were
> > > > > /var/log/lastlog and /var/log/wtmp - NOT one other log was written
> to
> > > in
> > > > > more than ten (10) minutes ...
> > > > >
> > > > > Please, advise. Thank you.
> > > > >
> > > > > ~ Mike
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Oct 19, 2017 at 12:32 PM, Rainer Gerhards <
> > > > > rgerha...@hq.adiscon.com>
> > > > > wrote:
> > > > >
> > > > > > 2017-10-19 16:14 GMT+02:00 Mike Schleif <
> > > mike+rsys...@mdsresource.net>
> > > > :
> > > > > > > Rainer,
> > > > > > >
> > > > > > > Debug attached. Full reboot follows each update and roll back.
> > > > > > >
> > > > > > > It looks like nothing under /var/log/ gets written to after
> > reboot
> > > > > > > complete, except lastlog and wtmp.
> > > > > >
> > > > > > mmhhh... I see at least writes to
> > > > > >
> > > > > > /var/log/messages:
> > > > > > Reg/w0  : strm 0x7f81fc005290: stream.c: opened file
> > > > > > '/var/log/messages' for WRITE as 12
> > > > > > Reg/w0  : strm 0x7f81fc005290: stream.c: file 12 write wrote 4041
> > > bytes
> > > > > >
> > > > > > from the embedded pstats, I see that no other action received
> > > > > > messages. So far, everything looks ok.
> > > > > >
> > > > > > Can you point me to a specific message that you think is
> missing? I
> > > > > > could then try to follow its flow inside the debug log.
> > > > > >
> > > > > > Rainer
> > > > > > >
> > > > > > > Event rsyslog-stats is not written to after boot complete.
> > > > > > >
> > > > > > > Please, advise. Thank you.
> > > > > > >
> > > > > > > ~ Mike
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Oct 18, 2017 at 10:43 AM, Rainer Gerhards <
> > > > > > rgerha...@hq.adiscon.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Do you mean some logs were written to and some not?
> > > > > > >>
> > > > > > >> If so, I need a Debug log to diagnose what is going on.
> > > > > > >>
> > > >

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-19 Thread Rainer Gerhards

Am 19.10.2017 21:55 schrieb "David Lang" :

On Thu, 19 Oct 2017, deoren wrote:

On 10/18/2017 8:10 PM, David Lang wrote:
>
>> On Wed, 18 Oct 2017, deoren wrote:
>>
>> On 10/18/2017 3:15 PM, David Lang wrote:
>>>
 On Wed, 18 Oct 2017, deoren wrote:

 On 10/18/2017 1:36 PM, David Lang wrote:
>
>> On Wed, 18 Oct 2017, deoren wrote:
>>
>>> Since the sender and receiver in this are both the latest versions
>>> of rsyslog (with the plan for the setup to remain that way), can I scale
>>> the accepted message size values to properly accommodate non-standard
>>> message sizes (delivered via JSON payloads)?
>>>
>>
>> up to 128K should not be a problem, I believe that to scale the
>> message size >128K you need to change a setting in the source.
>>
>
>>> Do you have experience delivering messages that large? I wonder whether
>>> I'm going about this the right way.
>>>
>>
>> I wasn't using relp, but I did see logs hit 128k and get truncated a few
>> times.
>>
>
> Do you use it now? If not, is it because it lacks a feature you need?
>

I changed jobs, and have not yet setup rsyslog here :-)

RELP has it's place, but most of the time I'm willing to loose some logs
under rare failure conditions and so haven't bothered to use it.

large maxmessagesize leads to wasted memory in rsyslog, but nothing more
>> severe than that.
>>
>
> Thanks for confirming. I'll likely go ahead and increase that value to
> 128K to see if the problem goes away.
>

if your maxmessagesize was 64k, that should not have been a problem.
>>
>
> Acknowledged. Do you know if all inputs honor the global() maxmessagesize
> value, or only certain ones? Does that value need to be specified using the
> legacy configuration syntax?
>

That's really a question Rainer will need to answer

In theory "yes", if not, its a bug. I ironed out some of those bugs the
past releases, most notably in imfile. I won't outrule I have overlooked
some.

Rainer

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Updates 8.29 -> 8.30 broke several logs

2017-10-19 Thread Rainer Gerhards

Well it would have helped to have this information before wading through
the log ;-). Now it needs to wait till tomorrow or Monday.

Did something (systemd) steal the log socket?

Räuber

Sent from phone, thus brief.

Am 19.10.2017 19:53 schrieb "Mike Schleif" :

> Look at line: 32697 - That is the LAST line of debug as the system booted
> up.
>
> Now, look at the next line: 32698 - That is the first line after the
> sysadmin pressed Enter after typing "reboot."
>
> I don't understand the time encoding prior to the first colon (:) of each
> line; but, this host was up for ten (10) minutes or more before backing out
> of the update patches and reboot.
>
> How can I provide missing messages, when they are missing?
>
> The only way to get to this host is via SSH. During the period of the debug
> log, another sysadmin and I logged onto that host at least three (3) times
> each - not one write to /var/log/secure !?!?
>
> Yes, there are /var/log/* writes up until the system fully booted - then
> nothing - until sysadmin pressed Enter, more than ten (10) minutes later.
> The ONLY /var/log/ files to get written to during that period were
> /var/log/lastlog and /var/log/wtmp - NOT one other log was written to in
> more than ten (10) minutes ...
>
> Please, advise. Thank you.
>
> ~ Mike
>
>
>
> On Thu, Oct 19, 2017 at 12:32 PM, Rainer Gerhards <
> rgerha...@hq.adiscon.com>
> wrote:
>
> > 2017-10-19 16:14 GMT+02:00 Mike Schleif :
> > > Rainer,
> > >
> > > Debug attached. Full reboot follows each update and roll back.
> > >
> > > It looks like nothing under /var/log/ gets written to after reboot
> > > complete, except lastlog and wtmp.
> >
> > mmhhh... I see at least writes to
> >
> > /var/log/messages:
> > Reg/w0  : strm 0x7f81fc005290: stream.c: opened file
> > '/var/log/messages' for WRITE as 12
> > Reg/w0  : strm 0x7f81fc005290: stream.c: file 12 write wrote 4041 bytes
> >
> > from the embedded pstats, I see that no other action received
> > messages. So far, everything looks ok.
> >
> > Can you point me to a specific message that you think is missing? I
> > could then try to follow its flow inside the debug log.
> >
> > Rainer
> > >
> > > Event rsyslog-stats is not written to after boot complete.
> > >
> > > Please, advise. Thank you.
> > >
> > > ~ Mike
> > >
> > >
> > > On Wed, Oct 18, 2017 at 10:43 AM, Rainer Gerhards <
> > rgerha...@hq.adiscon.com>
> > > wrote:
> > >
> > >> Do you mean some logs were written to and some not?
> > >>
> > >> If so, I need a Debug log to diagnose what is going on.
> > >>
> > >> Rainer
> > >>
> > >> Sent from phone, thus brief.
> > >>
> > >> Am 18.10.2017 17:36 schrieb "Mike Schleif" <
> > mike+rsys...@mdsresource.net>:
> > >>
> > >> > # cat /etc/centos-release
> > >> > CentOS Linux release 7.4.1708 (Core)
> > >> >
> > >> >
> > >> > After yum updates yesterday (see below,) several logs no longer
> > logged,
> > >> > including /var/log/secure
> > >> >
> > >> > In the last hour, we rolled back that entire yum update, and logging
> > >> > appears to be as expected
> > >> >
> > >> > Please, advise. Thank you.
> > >> >
> > >> > ~ Mike
> > >> >
> > >> >
> > >> > # yum history info 62
> > >> > Loaded plugins: fastestmirror
> > >> > Transaction ID : 62
> > >> > Begin time : Tue Oct 17 07:42:51 2017
> > >> > Begin rpmdb: 597:442a35918ca922c515d3f9bbc38cb3733341358a
> > >> > End time   :07:43:00 2017 (9 seconds)
> > >> > End rpmdb  : 597:f817c423ae76bafaafaab823cfca6d4030e069f0
> > >> > User   : Jeffrey Reed 
> > >> > Return-Code: Success
> > >> > Command Line   : update
> > >> > Transaction performed with:
> > >> > Installed rpm-4.11.3-25.el7.x86_64
> @base
> > >> > Installed yum-3.4.3-154.el7.centos.noarch
>  @base
> > >> > Installed yum-plugin-fastestmirror-1.1.31-42.el7.noarch
> @base
> > >> > Packages Altered:
> > >> > Updated epel-release-7-10.noarch   @epel
> > >> > Update   7-11.noarch   @epel-testing
> > >> > Updated libfastjson4-0.99.5-1.el7.x86_64   @rsyslog_v8
> > >> > Update   0.99.7-1.el7.x86_64   @rsyslog_v8
> > >> > Updated mysql-community-client-5.6.37-2.el7.x86_64
> > >> @mysql56-community
> > >> > Update 5.6.38-2.el7.x86_64
> > @mysql56-community
> > >> > Updated mysql-community-common-5.6.37-2.el7.x86_64
> > >> @mysql56-community
> > >> > Update 5.6.38-2.el7.x86_64
> > @mysql56-community
> > >> > Updated mysql-community-libs-5.6.37-2.el7.x86_64
> > >>  @mysql56-community
> > >> > Update   5.6.38-2.el7.x86_64
> >  @mysql56-community
> > >> > Updated rsyslog-8.29.0-2.el7.x86_64@rsyslog_v8
> > >> > Update  8.30.0-1.el7.x86_64@rsyslog_v8
> > >> > Updated rsyslog-mysql-8.29.0-2.el7.x86_64  @rsyslog_v8
> > >> >

Re: [rsyslog] Updates 8.29 -> 8.30 broke several logs

2017-10-19 Thread Mike Schleif

Rainer,

Yes, I respect your time. Since it is running with 8.29, I can keep this
running as-is for a week or so; but, I do need the update fixes asap.

For debug log from working system, do you need any system reboot?

If not, I can turn on debug in rsyslog.conf, then simple restart rsyslogd.

Please, advise. Thank you.

~ Mike



On Thu, Oct 19, 2017 at 1:35 PM, Rainer Gerhards 
wrote:

> I think David can probably answer that better. You need to check systemd
> and journal conf.
>
> But you said it works with an older version. Can you create a Debug log
> with that one as well so that I can compare? That would probably be useful.
> Again (due to time zone differences) I can look at this at earliest in
> roughly 12 hours - depending on what work has waiting for me in the
> morning. Having both logs by then would definitely be a plus.
>
> Rainer
>
> Sent from phone, thus brief.
>
> Am 19.10.2017 20:24 schrieb "Mike Schleif" :
>
> > Rainer,
> >
> > Apparently, I wasn't explicit enough when submitting the debug log.
> >
> > You asked: Did something (systemd) steal the log socket?
> >
> > I don't know. How could I know? How can I find out?
> >
> > Please, advise. Thank you.
> >
> > ~ Mike
> >
> >
> > On Thu, Oct 19, 2017 at 1:18 PM, Rainer Gerhards <
> rgerha...@hq.adiscon.com
> > >
> > wrote:
> >
> > > Well it would have helped to have this information before wading
> through
> > > the log ;-). Now it needs to wait till tomorrow or Monday.
> > >
> > > Did something (systemd) steal the log socket?
> > >
> > > Räuber
> > >
> > > Sent from phone, thus brief.
> > >
> > > Am 19.10.2017 19:53 schrieb "Mike Schleif" <
> mike+rsys...@mdsresource.net
> > >:
> > >
> > > > Look at line: 32697 - That is the LAST line of debug as the system
> > booted
> > > > up.
> > > >
> > > > Now, look at the next line: 32698 - That is the first line after the
> > > > sysadmin pressed Enter after typing "reboot."
> > > >
> > > > I don't understand the time encoding prior to the first colon (:) of
> > each
> > > > line; but, this host was up for ten (10) minutes or more before
> backing
> > > out
> > > > of the update patches and reboot.
> > > >
> > > > How can I provide missing messages, when they are missing?
> > > >
> > > > The only way to get to this host is via SSH. During the period of the
> > > debug
> > > > log, another sysadmin and I logged onto that host at least three (3)
> > > times
> > > > each - not one write to /var/log/secure !?!?
> > > >
> > > > Yes, there are /var/log/* writes up until the system fully booted -
> > then
> > > > nothing - until sysadmin pressed Enter, more than ten (10) minutes
> > later.
> > > > The ONLY /var/log/ files to get written to during that period were
> > > > /var/log/lastlog and /var/log/wtmp - NOT one other log was written to
> > in
> > > > more than ten (10) minutes ...
> > > >
> > > > Please, advise. Thank you.
> > > >
> > > > ~ Mike
> > > >
> > > >
> > > >
> > > > On Thu, Oct 19, 2017 at 12:32 PM, Rainer Gerhards <
> > > > rgerha...@hq.adiscon.com>
> > > > wrote:
> > > >
> > > > > 2017-10-19 16:14 GMT+02:00 Mike Schleif <
> > mike+rsys...@mdsresource.net>
> > > :
> > > > > > Rainer,
> > > > > >
> > > > > > Debug attached. Full reboot follows each update and roll back.
> > > > > >
> > > > > > It looks like nothing under /var/log/ gets written to after
> reboot
> > > > > > complete, except lastlog and wtmp.
> > > > >
> > > > > mmhhh... I see at least writes to
> > > > >
> > > > > /var/log/messages:
> > > > > Reg/w0  : strm 0x7f81fc005290: stream.c: opened file
> > > > > '/var/log/messages' for WRITE as 12
> > > > > Reg/w0  : strm 0x7f81fc005290: stream.c: file 12 write wrote 4041
> > bytes
> > > > >
> > > > > from the embedded pstats, I see that no other action received
> > > > > messages. So far, everything looks ok.
> > > > >
> > > > > Can you point me to a specific message that you think is missing? I
> > > > > could then try to follow its flow inside the debug log.
> > > > >
> > > > > Rainer
> > > > > >
> > > > > > Event rsyslog-stats is not written to after boot complete.
> > > > > >
> > > > > > Please, advise. Thank you.
> > > > > >
> > > > > > ~ Mike
> > > > > >
> > > > > >
> > > > > > On Wed, Oct 18, 2017 at 10:43 AM, Rainer Gerhards <
> > > > > rgerha...@hq.adiscon.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Do you mean some logs were written to and some not?
> > > > > >>
> > > > > >> If so, I need a Debug log to diagnose what is going on.
> > > > > >>
> > > > > >> Rainer
> > > > > >>
> > > > > >> Sent from phone, thus brief.
> > > > > >>
> > > > > >> Am 18.10.2017 17:36 schrieb "Mike Schleif" <
> > > > > mike+rsys...@mdsresource.net>:
> > > > > >>
> > > > > >> > # cat /etc/centos-release
> > > > > >> > CentOS Linux release 7.4.1708 (Core)
> > > > > >> >
> > > > > >> >
> > > > > >> > After yum updates yesterday (see below,) several logs no
> longer
> > > > > logged,
> > > > > >> > including /var/log/secure
> > > > >

Re: [rsyslog] Updates 8.29 -> 8.30 broke several logs

2017-10-19 Thread Mike Schleif

Look at line: 32697 - That is the LAST line of debug as the system booted
up.

Now, look at the next line: 32698 - That is the first line after the
sysadmin pressed Enter after typing "reboot."

I don't understand the time encoding prior to the first colon (:) of each
line; but, this host was up for ten (10) minutes or more before backing out
of the update patches and reboot.

How can I provide missing messages, when they are missing?

The only way to get to this host is via SSH. During the period of the debug
log, another sysadmin and I logged onto that host at least three (3) times
each - not one write to /var/log/secure !?!?

Yes, there are /var/log/* writes up until the system fully booted - then
nothing - until sysadmin pressed Enter, more than ten (10) minutes later.
The ONLY /var/log/ files to get written to during that period were
/var/log/lastlog and /var/log/wtmp - NOT one other log was written to in
more than ten (10) minutes ...

Please, advise. Thank you.

~ Mike



On Thu, Oct 19, 2017 at 12:32 PM, Rainer Gerhards 
wrote:

> 2017-10-19 16:14 GMT+02:00 Mike Schleif :
> > Rainer,
> >
> > Debug attached. Full reboot follows each update and roll back.
> >
> > It looks like nothing under /var/log/ gets written to after reboot
> > complete, except lastlog and wtmp.
>
> mmhhh... I see at least writes to
>
> /var/log/messages:
> Reg/w0  : strm 0x7f81fc005290: stream.c: opened file
> '/var/log/messages' for WRITE as 12
> Reg/w0  : strm 0x7f81fc005290: stream.c: file 12 write wrote 4041 bytes
>
> from the embedded pstats, I see that no other action received
> messages. So far, everything looks ok.
>
> Can you point me to a specific message that you think is missing? I
> could then try to follow its flow inside the debug log.
>
> Rainer
> >
> > Event rsyslog-stats is not written to after boot complete.
> >
> > Please, advise. Thank you.
> >
> > ~ Mike
> >
> >
> > On Wed, Oct 18, 2017 at 10:43 AM, Rainer Gerhards <
> rgerha...@hq.adiscon.com>
> > wrote:
> >
> >> Do you mean some logs were written to and some not?
> >>
> >> If so, I need a Debug log to diagnose what is going on.
> >>
> >> Rainer
> >>
> >> Sent from phone, thus brief.
> >>
> >> Am 18.10.2017 17:36 schrieb "Mike Schleif" <
> mike+rsys...@mdsresource.net>:
> >>
> >> > # cat /etc/centos-release
> >> > CentOS Linux release 7.4.1708 (Core)
> >> >
> >> >
> >> > After yum updates yesterday (see below,) several logs no longer
> logged,
> >> > including /var/log/secure
> >> >
> >> > In the last hour, we rolled back that entire yum update, and logging
> >> > appears to be as expected
> >> >
> >> > Please, advise. Thank you.
> >> >
> >> > ~ Mike
> >> >
> >> >
> >> > # yum history info 62
> >> > Loaded plugins: fastestmirror
> >> > Transaction ID : 62
> >> > Begin time : Tue Oct 17 07:42:51 2017
> >> > Begin rpmdb: 597:442a35918ca922c515d3f9bbc38cb3733341358a
> >> > End time   :07:43:00 2017 (9 seconds)
> >> > End rpmdb  : 597:f817c423ae76bafaafaab823cfca6d4030e069f0
> >> > User   : Jeffrey Reed 
> >> > Return-Code: Success
> >> > Command Line   : update
> >> > Transaction performed with:
> >> > Installed rpm-4.11.3-25.el7.x86_64  @base
> >> > Installed yum-3.4.3-154.el7.centos.noarch   @base
> >> > Installed yum-plugin-fastestmirror-1.1.31-42.el7.noarch @base
> >> > Packages Altered:
> >> > Updated epel-release-7-10.noarch   @epel
> >> > Update   7-11.noarch   @epel-testing
> >> > Updated libfastjson4-0.99.5-1.el7.x86_64   @rsyslog_v8
> >> > Update   0.99.7-1.el7.x86_64   @rsyslog_v8
> >> > Updated mysql-community-client-5.6.37-2.el7.x86_64
> >> @mysql56-community
> >> > Update 5.6.38-2.el7.x86_64
> @mysql56-community
> >> > Updated mysql-community-common-5.6.37-2.el7.x86_64
> >> @mysql56-community
> >> > Update 5.6.38-2.el7.x86_64
> @mysql56-community
> >> > Updated mysql-community-libs-5.6.37-2.el7.x86_64
> >>  @mysql56-community
> >> > Update   5.6.38-2.el7.x86_64
>  @mysql56-community
> >> > Updated rsyslog-8.29.0-2.el7.x86_64@rsyslog_v8
> >> > Update  8.30.0-1.el7.x86_64@rsyslog_v8
> >> > Updated rsyslog-mysql-8.29.0-2.el7.x86_64  @rsyslog_v8
> >> > Update8.30.0-1.el7.x86_64  @rsyslog_v8
> >> > history info
> >> > ___
> >> > rsyslog mailing list
> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> > http://www.rsyslog.com/professional-services/
> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> > DON'T

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-19 Thread deoren


On 10/19/2017 3:12 PM, Rainer Gerhards wrote:

Am 19.10.2017 21:55 schrieb "David Lang" :
RELP has it's place, but most of the time I'm willing to loose some logs
under rare failure conditions and so haven't bothered to use it.


large maxmessagesize leads to wasted memory in rsyslog, but nothing more

severe than that.



Thanks for confirming. I'll likely go ahead and increase that value to
128K to see if the problem goes away.



if your maxmessagesize was 64k, that should not have been a problem.




Acknowledged. Do you know if all inputs honor the global() maxmessagesize
value, or only certain ones? Does that value need to be specified using the
legacy configuration syntax?



That's really a question Rainer will need to answer


In theory "yes", if not, its a bug. I ironed out some of those bugs the
past releases, most notably in imfile. I won't outrule I have overlooked
some.

Rainer


It's far too soon to say that the issue with using RELP has been worked 
around (other than using omfwd and imptcp in its place), but I modified 
the imprelp input() definition on the receiver to explicitly specify 
MaxDataSize="128k" instead of relying on the global parameter 
configuration of maxMessageSize="128k".


That seems to have made a difference, but I won't know more until we go 
a full day on Friday with heavy activity on the sending system again.


On a related note, I noticed that I enabled Keep Alive for the imrelp 
input() definition. Does RELP benefit from enabling Keep Alive? If so, 
do you specify it for both imrelp (input) and omrelp (action)?


I noticed that you can configure both ends of the connection when using 
plain TCP forwarding (omfwd on sender and imptcp on receiver). Does RELP 
not operate that way?

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Updates 8.29 -> 8.30 broke several logs

2017-10-19 Thread Rainer Gerhards

2017-10-19 16:14 GMT+02:00 Mike Schleif :
> Rainer,
>
> Debug attached. Full reboot follows each update and roll back.
>
> It looks like nothing under /var/log/ gets written to after reboot
> complete, except lastlog and wtmp.

mmhhh... I see at least writes to

/var/log/messages:
Reg/w0  : strm 0x7f81fc005290: stream.c: opened file
'/var/log/messages' for WRITE as 12
Reg/w0  : strm 0x7f81fc005290: stream.c: file 12 write wrote 4041 bytes

from the embedded pstats, I see that no other action received
messages. So far, everything looks ok.

Can you point me to a specific message that you think is missing? I
could then try to follow its flow inside the debug log.

Rainer
>
> Event rsyslog-stats is not written to after boot complete.
>
> Please, advise. Thank you.
>
> ~ Mike
>
>
> On Wed, Oct 18, 2017 at 10:43 AM, Rainer Gerhards 
> wrote:
>
>> Do you mean some logs were written to and some not?
>>
>> If so, I need a Debug log to diagnose what is going on.
>>
>> Rainer
>>
>> Sent from phone, thus brief.
>>
>> Am 18.10.2017 17:36 schrieb "Mike Schleif" :
>>
>> > # cat /etc/centos-release
>> > CentOS Linux release 7.4.1708 (Core)
>> >
>> >
>> > After yum updates yesterday (see below,) several logs no longer logged,
>> > including /var/log/secure
>> >
>> > In the last hour, we rolled back that entire yum update, and logging
>> > appears to be as expected
>> >
>> > Please, advise. Thank you.
>> >
>> > ~ Mike
>> >
>> >
>> > # yum history info 62
>> > Loaded plugins: fastestmirror
>> > Transaction ID : 62
>> > Begin time : Tue Oct 17 07:42:51 2017
>> > Begin rpmdb: 597:442a35918ca922c515d3f9bbc38cb3733341358a
>> > End time   :07:43:00 2017 (9 seconds)
>> > End rpmdb  : 597:f817c423ae76bafaafaab823cfca6d4030e069f0
>> > User   : Jeffrey Reed 
>> > Return-Code: Success
>> > Command Line   : update
>> > Transaction performed with:
>> > Installed rpm-4.11.3-25.el7.x86_64  @base
>> > Installed yum-3.4.3-154.el7.centos.noarch   @base
>> > Installed yum-plugin-fastestmirror-1.1.31-42.el7.noarch @base
>> > Packages Altered:
>> > Updated epel-release-7-10.noarch   @epel
>> > Update   7-11.noarch   @epel-testing
>> > Updated libfastjson4-0.99.5-1.el7.x86_64   @rsyslog_v8
>> > Update   0.99.7-1.el7.x86_64   @rsyslog_v8
>> > Updated mysql-community-client-5.6.37-2.el7.x86_64
>> @mysql56-community
>> > Update 5.6.38-2.el7.x86_64 @mysql56-community
>> > Updated mysql-community-common-5.6.37-2.el7.x86_64
>> @mysql56-community
>> > Update 5.6.38-2.el7.x86_64 @mysql56-community
>> > Updated mysql-community-libs-5.6.37-2.el7.x86_64
>>  @mysql56-community
>> > Update   5.6.38-2.el7.x86_64   @mysql56-community
>> > Updated rsyslog-8.29.0-2.el7.x86_64@rsyslog_v8
>> > Update  8.30.0-1.el7.x86_64@rsyslog_v8
>> > Updated rsyslog-mysql-8.29.0-2.el7.x86_64  @rsyslog_v8
>> > Update8.30.0-1.el7.x86_64  @rsyslog_v8
>> > history info
>> > ___
>> > rsyslog mailing list
>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com/professional-services/
>> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> > DON'T LIKE THAT.
>> >
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

2017-10-19 Thread deoren


On 10/18/2017 8:10 PM, David Lang wrote:

On Wed, 18 Oct 2017, deoren wrote:


On 10/18/2017 3:15 PM, David Lang wrote:

On Wed, 18 Oct 2017, deoren wrote:


On 10/18/2017 1:36 PM, David Lang wrote:

On Wed, 18 Oct 2017, deoren wrote:
Since the sender and receiver in this are both the latest versions 
of rsyslog (with the plan for the setup to remain that way), can I 
scale the accepted message size values to properly accommodate 
non-standard message sizes (delivered via JSON payloads)?


up to 128K should not be a problem, I believe that to scale the 
message size >128K you need to change a setting in the source.


Do you have experience delivering messages that large? I wonder 
whether I'm going about this the right way.


I wasn't using relp, but I did see logs hit 128k and get truncated a few 
times.


Do you use it now? If not, is it because it lacks a feature you need?



large maxmessagesize leads to wasted memory in rsyslog, but nothing more 
severe than that.


Thanks for confirming. I'll likely go ahead and increase that value to 
128K to see if the problem goes away.




The particular box I'm having trouble with runs a Redmine instance 
that is heavily used and has lots of activity within its 
production log and development log files (large entries). I'd like 
to deliver entire log entries instead of truncated versions if 
possible.


how large are the lines?


At present, the development log file has a line with a length of 
2997 characters and the production.log file has a line with a length 
of 8608 characters.


those are quite reasonable


I checked again and found a single line of 24405 characters. I've not 
yet checked back through historical log files to see what some of the 
longer lines were.


if your maxmessagesize was 64k, that should not have been a problem.


Acknowledged. Do you know if all inputs honor the global() 
maxmessagesize value, or only certain ones? Does that value need to be 
specified using the legacy configuration syntax?


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Updates 8.29 -> 8.30 broke several logs

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

Re: [rsyslog] Updates 8.29 -> 8.30 broke several logs

Re: [rsyslog] Updates 8.29 -> 8.30 broke several logs

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

Re: [rsyslog] Updates 8.29 -> 8.30 broke several logs

Re: [rsyslog] Updates 8.29 -> 8.30 broke several logs

Re: [rsyslog] Updates 8.29 -> 8.30 broke several logs

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

Re: [rsyslog] Updates 8.29 -> 8.30 broke several logs

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

11 matches

Site Navigation

Mail list logo

Footer information