Hello,
I'm having an issue where messages sent from rsyslog relays via RELP to
rsyslog collectors that write them to disk via DynFile (%HOSTNAME%
template) are not looking right and causing problems. I don't think the
RELP protocol is negotiating properly as the syslog messages are not
seen on the wire and the queue is building up on the relays.
$ ls -l /var/spool/rsyslog/
total 69892
-rw------- 1 root root 1049135 Sep 24 11:36 rsyslog-buffer.00000001
-rw------- 1 root root 1048593 Sep 24 12:09 rsyslog-buffer.00000002
-rw------- 1 root root 1049021 Sep 18 22:19 rsyslog-buffer.00000003
-rw------- 1 root root 1049181 Sep 19 00:32 rsyslog-buffer.00000004
-rw------- 1 root root 1048907 Sep 19 02:01 rsyslog-buffer.00000005
-rw------- 1 root root 1049049 Sep 19 04:13 rsyslog-buffer.00000006
-rw------- 1 root root 1048661 Sep 19 06:25 rsyslog-buffer.00000007
....
Systems are running compiled with RELP via rsyslog's rpm repo.
$ rsyslogd -v
rsyslogd 7.4.4, compiled with:
FEATURE_REGEXP: Yes
FEATURE_LARGEFILE: No
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
Runtime Instrumentation (slow code): No
uuid support: Yes
Here is dump of the syslog messages seen by the collectors. This message is
seen over and over.
115, options [nop,nop,TS val 141682 ecr 152580], length 96
0x0000: 4500 0094 c32b 4000 4006 2585 0a8e 9e47 E....+@.@.%....G
0x0010: 0a8e 9e50 e6a9 0202 6808 ea5c 8961 44fd ...P....h..\.aD.
0x0020: 8018 0073 5936 0000 0101 080a 0002 2972 ...sY6........)r
0x0030: 0002 5404 3120 6f70 656e 2038 3520 7265 ..T.1.open.85.re
0x0040: 6c70 5f76 6572 7369 6f6e 3d30 0a72 656c lp_version=0.rel
0x0050: 705f 736f 6674 7761 7265 3d6c 6962 7265 p_software=libre
0x0060: 6c70 2c31 2e30 2e36 2c68 7474 703a 2f2f lp,1.0.6,http://
0x0070: 6c69 6272 656c 702e 6164 6973 636f 6e2e librelp.adiscon.
0x0080: 636f 6d0a 636f 6d6d 616e 6473 3d73 7973 com.commands=sys
0x0090: 6c6f 670a log.
Each of the rsyslog machines have multiple connections that end up in
TIME_WAIT state,
over and over which suggests that the relays connect to the collectors but
the application
data isn't liked and thus connection finishes and is then retried again.
$ netstat -an -A inet | grep 514
tcp 0 0 0.0.0.0:514 0.0.0.0:*
LISTEN
tcp 0 0 10.142.158.71:55249 10.142.158.81:514
TIME_WAIT
tcp 0 0 10.142.158.71:59079 10.142.158.80:514
TIME_WAIT
tcp 0 0 10.142.158.71:55239 10.142.158.81:514
TIME_WAIT
tcp 0 0 10.142.158.71:59073 10.142.158.80:514
TIME_WAIT
tcp 0 0 10.142.158.71:55247 10.142.158.81:514
TIME_WAIT
tcp 0 0 10.142.158.71:59071 10.142.158.80:514
TIME_WAIT
tcp 0 0 10.142.158.71:59077 10.142.158.80:514
TIME_WAIT
tcp 0 0 10.142.158.71:59069 10.142.158.80:514
TIME_WAIT
tcp 0 0 10.142.158.71:55241 10.142.158.81:514
TIME_WAIT
tcp 0 0 10.142.158.71:55251 10.142.158.81:514
ESTABLISHED
tcp 0 0 10.142.158.71:55245 10.142.158.81:514
TIME_WAIT
tcp 0 0 10.142.158.71:55243 10.142.158.81:514
TIME_WAIT
tcp 0 0 10.142.158.71:59075 10.142.158.80:514
TIME_WAIT
udp 0 0 0.0.0.0:514 0.0.0.0:*
RSYSLOG RELAY CONFIGURATION:
Relays should receive messages via UDP, TCP, and RELP.
#### MODULES ####
$ModLoad imuxsock # provides support for local system logging (e.g. via
logger command)
$ModLoad imklog # provides kernel logging support (previously done by
rklogd)
#$ModLoad immark # provides --MARK-- message capability
# Loaded by default
# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514
# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
# Load input module relp
$ModLoad imrelp
$InputRELPServerRun 514
# Load output module relp
$ModLoad omrelp
#### GLOBAL DIRECTIVES ####
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Spool files
$WorkDirectory /var/spool/rsyslog
#### RULES ####
(default local log stuff cut out for brevity)
action(Name="collect02-sec"
Type="omrelp"
Target="10.142.158.81"
Port="514"
Action.ResumeInterval="5"
Timeout="5")
action(Name="collect02-sec"
Type="omrelp"
Target="10.142.158.80"
Port="514"
Action.ExecOnlyWhenPreviousIsSuspended="on"
Action.ResumeInterval="5"
Timeout="5")
RSYSLOG COLLECTOR CONFIGURATON:
#### MODULES ####
$ModLoad imuxsock # provides support for local system logging (e.g. via
logger command)
$ModLoad imklog # provides kernel logging support (previously done by
rklogd)
#$ModLoad immark # provides --MARK-- message capability
# Load (i)nput and (o)utput (m)odules
$ModLoad imudp
$ModLoad imtcp
$ModLoad imrelp
$ModLoad omrelp
#### TEMPLATES ####
template(name="ByHost" type="string"
string="/var/log/remotelogs/%HOSTNAME%.log")
#### GLOBAL DIRECTIVES ####
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Spool files
$WorkDirectory /var/spool/rsyslog
#### RULES ####
$AllowedSender TCP, 10.142.158.70, 10.142.158.71
$AllowedSender UDP, 10.142.158.70, 10.142.158.71
#### Remote Logging
if $fromhost-ip == ["10.142.158.70","10.142.158.71"]
then {
action(type="omfile" DynaFile="ByHost")
stop
}
#### Local Logging
(default local log stuff cut out for brevity)
#### LISTENERS ####
# Provides UDP syslog reception
$UDPServerRun 514
# Provides TCP syslog reception
$InputTCPServerRun 514
# Provides Reception of RELP
$InputRELPServerRun 514
I've been working on this for awhile so any help is appreciated.
Thanks
--
Jon Schipp,
jonschipp.com, sickbits.net
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
