On Wed, 20 Aug 2014, Ivan Lezhnjov IV wrote:

The very much disliked and unsupported here version 5.8.6 :)

So, one way around this would be to set $RepeatedMsgReduction to off? Otherwise 
not much can be done about this, as it sounds like it is a sort of unresolved 
bug in v 5?

Yes, turn off repeated message reduction

This was a good idea 20+ years ago when logs were viewed by humans, but when you have tools analysing the logs, this 'feature' makes it impossible for the tools to notice that the message is repeating (especially after logs are sent to a remote machine)

David Lang

Ivan

On Aug 20, 2014, at 4:32 PM, Rainer Gerhards <[email protected]> wrote:

This sounds like a problem with the ugly "last message repeated n times"
feature. Which version are the clients? If not current, you should at least
update one to 8.4.0 and check if the problem persists.

Rainer


On Wed, Aug 20, 2014 at 3:28 PM, Ivan Lezhnjov IV <
[email protected]> wrote:


On Aug 19, 2014, at 6:33 PM, David Lang <[email protected]> wrote:

On Tue, 19 Aug 2014, Ivan Lezhnjov IV wrote:

Hello,

On Aug 15, 2014, at 6:17 PM, Rainer Gerhards <[email protected]>
wrote:

On Fri, Aug 15, 2014 at 5:13 PM, Mike Hoskins (michoski) <
[email protected]
wrote:

I thought %FROMHOST% caused a DNS lookup on rsyslog's side, while
%HOSTNAME% just used the hostname sent in the message...others will
correct if my memory is bad.


That's right, but I think we fall back to a dns lookup if there is no
detectable hostname in the message(not 100% sure, though).


So if %HOSTNAME% is not right, it must be
something on the client side.


can very well be, but sounded more like DNS resolution.


I think you just use %rawmsg% to get the raw message.  :-)

http://www.rsyslog.com/doc/property_replacer.html


yup or use

*.* /var/log/messagedebug;RSYSLOG_DebugFormat

which will write out all properties.

This is how a normal message looks like:

Debug line with all properties:
FROMHOST: '172.16.16.4', fromhost-ip: '172.16.16.4', HOSTNAME:
'xyz-DDDD-02', PRI: 86,
syslogtag 'su[42661]:', programname: 'su', APP-NAME: 'su', PROCID:
'42661', MSGID: '-',
TIMESTAMP: 'Aug 19 02:11:58', STRUCTURED-DATA: '-',
msg: ' pam_unix(su:session): session closed for user postgres'
escaped msg: ' pam_unix(su:session): session closed for user postgres'
inputname: imtcp rawmsg: '<86>Aug 19 02:11:58 xyz-DDDD-02 su[42661]:
pam_unix(su:session): session closed for user postgres'

Are we interested in this only, or also what debug message is going to
look like when the suspected DNS resolution failure occurs again?

It would be good to get one of a failing message

In this case, the HOSTNAME is pulled directly from the rawmsg, so no DNS
lookup is done there.

normally FROMHOST is a DNS lookup of fromhost-ip, so this message shows
a "normal" DNS failure.

This makes me think that you have a situation where the sender isn't
properly populating the hostname field of the message under some conditions.

Alright, it happened again yesterday and these two debug messages follow
exactly one after another:

Debug line with all properties:
FROMHOST: '172.16.16.5', fromhost-ip: '172.16.16.5', HOSTNAME:
'xyz-DDDD-03', PRI: 29,
syslogtag 'dbus[1767]:', programname: 'dbus', APP-NAME: 'dbus', PROCID:
'1767', MSGID: '-',
TIMESTAMP: 'Aug 19 18:11:57', STRUCTURED-DATA: '-',
msg: ' [system] Reloaded configuration'
escaped msg: ' [system] Reloaded configuration'
inputname: imtcp rawmsg: '<29>Aug 19 18:11:57 xyz-DDDD-03 dbus[1767]:
[system] Reloaded configuration'

Debug line with all properties:
FROMHOST: '172.16.16.5', fromhost-ip: '172.16.16.5', HOSTNAME:
'172.16.16.5', PRI: 29,
syslogtag '', programname: '', APP-NAME: '', PROCID: '-', MSGID: '-',
TIMESTAMP: 'Aug 19 18:11:57', STRUCTURED-DATA: '-',
msg: ' dbus[1767]: last message repeated 4 times'
escaped msg: ' dbus[1767]: last message repeated 4 times'
inputname: imtcp rawmsg: '<29>Aug 19 18:11:57  dbus[1767]: last message
repeated 4 times'

Then it goes back to proper host name resolution and fails again after
some time:

Debug line with all properties:
FROMHOST: '172.16.16.5', fromhost-ip: '172.16.16.5', HOSTNAME:
'xyz-DDDD-03', PRI: 13,
syslogtag 'dataeng:', programname: 'dataeng', APP-NAME: 'dataeng', PROCID:
'-', MSGID: '-',
TIMESTAMP: 'Aug 19 18:12:03', STRUCTURED-DATA: '-',
msg: ' NOTE: OS SNMP support not installed'
escaped msg: ' NOTE: OS SNMP support not installed'
inputname: imtcp rawmsg: '<13>Aug 19 18:12:03 xyz-DDDD-03 dataeng: NOTE:
OS SNMP support not installed'

Debug line with all properties:
FROMHOST: '172.16.16.5', fromhost-ip: '172.16.16.5', HOSTNAME:
'172.16.16.5', PRI: 29,
syslogtag '', programname: '', APP-NAME: '', PROCID: '-', MSGID: '-',
TIMESTAMP: 'Aug 19 18:12:03', STRUCTURED-DATA: '-',
msg: ' dbus[1767]: last message repeated 4 times'
escaped msg: ' dbus[1767]: last message repeated 4 times'
inputname: imtcp rawmsg: '<29>Aug 19 18:12:03  dbus[1767]: last message
repeated 4 times'

Note how APP-NAME: '', PROCID: '-', MSGID: '-', are just the "-" sign when
HOSTNAME becomes an IP address.

Then it happened again:

Debug line with all properties:
FROMHOST: '172.16.16.5', fromhost-ip: '172.16.16.5', HOSTNAME:
'172.16.16.5', PRI: 29,
syslogtag '', programname: '', APP-NAME: '', PROCID: '-', MSGID: '-',
TIMESTAMP: 'Aug 19 18:12:17', STRUCTURED-DATA: '-',
msg: ' dbus[1767]: last message repeated 11 times'
escaped msg: ' dbus[1767]: last message repeated 11 times'
inputname: imtcp rawmsg: '<29>Aug 19 18:12:17  dbus[1767]: last message
repeated 11 times'

The /var/log/syslog on 172.16.16.5 has these entries that match the time
of the ones mentioned above:

Aug 19 17:55:01 xyz-DDDD-03 CRON[46232]: (root) CMD (command -v debian-sa1
/dev/null && debian-sa1 1 1)
Aug 19 18:05:01 xyz-DDDD-03 CRON[49559]: (root) CMD (command -v debian-sa1
/dev/null && debian-sa1 1 1)
Aug 19 18:11:57 xyz-DDDD-03 dbus[1767]: [system] Reloaded configuration
Aug 19 18:12:03  dbus[1767]: last message repeated 4 times
Aug 19 18:12:03 xyz-DDDD-03 dataeng: NOTE: OS SNMP support not installed
Aug 19 18:12:04 xyz-DDDD-03 dbus[1767]: [system] Reloaded configuration
Aug 19 18:12:17  dbus[1767]: last message repeated 11 times
Aug 19 18:12:17 xyz-DDDD-03 ata_id[55947]: HDIO_GET_IDENTITY failed for
'/dev/sdc': Invalid argument
Aug 19 18:12:17 xyz-DDDD-03 dbus[1767]: [system] Reloaded configuration
Aug 19 18:12:17 xyz-DDDD-03 dbus[1767]: [system] Reloaded configuration

So, as we can see the hostname never appears in the client log file
actually.

Ivan
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Reply via email to