Update: I have logs in the remote log directory but they're strange. o and pen are not hosts, I imagine the word "open" was separated. Also, the logs from the relays are nothing but the relp version lines. Seems that this problem may be on the relays now.
$ head -3 * ==> 10.142.158.70.log <== Sep 24 15:34:25 10.142.158.70 relp_software=librelp,1.0.6,http: // librelp.adiscon.com Sep 24 15:34:25 10.142.158.70 commands=syslog Sep 24 15:34:45 10.142.158.70 relp_software=librelp,1.0.6,http: // librelp.adiscon.com ==> 10.142.158.71.log <== Sep 24 15:34:35 10.142.158.71 relp_software=librelp,1.0.6,http: // librelp.adiscon.com Sep 24 15:34:35 10.142.158.71 commands=syslog Sep 24 15:34:55 10.142.158.71 relp_software=librelp,1.0.6,http: // librelp.adiscon.com ==> o.log <== Sep 24 15:34:25 o Sep 24 15:34:35 o Sep 24 15:34:45 o ==> pen.log <== Sep 24 15:34:25 pen 85 relp_version=0 Sep 24 15:34:35 pen 85 relp_version=0 Sep 24 15:34:45 pen 85 relp_version=0 On Tue, Sep 24, 2013 at 4:04 PM, Jon Schipp <[email protected]> wrote: > Hello, > > I'm having an issue where messages sent from rsyslog relays via RELP to > rsyslog collectors that write them to disk via DynFile (%HOSTNAME% > template) are not looking right and causing problems. I don't think the > RELP protocol is negotiating properly as the syslog messages are not > seen on the wire and the queue is building up on the relays. > > $ ls -l /var/spool/rsyslog/ > total 69892 > -rw------- 1 root root 1049135 Sep 24 11:36 rsyslog-buffer.00000001 > -rw------- 1 root root 1048593 Sep 24 12:09 rsyslog-buffer.00000002 > -rw------- 1 root root 1049021 Sep 18 22:19 rsyslog-buffer.00000003 > -rw------- 1 root root 1049181 Sep 19 00:32 rsyslog-buffer.00000004 > -rw------- 1 root root 1048907 Sep 19 02:01 rsyslog-buffer.00000005 > -rw------- 1 root root 1049049 Sep 19 04:13 rsyslog-buffer.00000006 > -rw------- 1 root root 1048661 Sep 19 06:25 rsyslog-buffer.00000007 > .... > > Systems are running compiled with RELP via rsyslog's rpm repo. > $ rsyslogd -v > rsyslogd 7.4.4, compiled with: > FEATURE_REGEXP: Yes > FEATURE_LARGEFILE: No > GSSAPI Kerberos 5 support: Yes > FEATURE_DEBUG (debug build, slow code): No > 32bit Atomic operations supported: Yes > 64bit Atomic operations supported: Yes > Runtime Instrumentation (slow code): No > uuid support: Yes > > Here is dump of the syslog messages seen by the collectors. This message > is seen over and over. > > 115, options [nop,nop,TS val 141682 ecr 152580], length 96 > 0x0000: 4500 0094 c32b 4000 4006 2585 0a8e 9e47 E....+@.@.%....G > 0x0010: 0a8e 9e50 e6a9 0202 6808 ea5c 8961 44fd ...P....h..\.aD. > 0x0020: 8018 0073 5936 0000 0101 080a 0002 2972 ...sY6........)r > 0x0030: 0002 5404 3120 6f70 656e 2038 3520 7265 ..T.1.open.85.re > 0x0040: 6c70 5f76 6572 7369 6f6e 3d30 0a72 656c lp_version=0.rel > 0x0050: 705f 736f 6674 7761 7265 3d6c 6962 7265 p_software=libre > 0x0060: 6c70 2c31 2e30 2e36 2c68 7474 703a 2f2f lp,1.0.6,http:// > 0x0070: 6c69 6272 656c 702e 6164 6973 636f 6e2e librelp.adiscon. > 0x0080: 636f 6d0a 636f 6d6d 616e 6473 3d73 7973 com.commands=sys > 0x0090: 6c6f 670a log. > > Each of the rsyslog machines have multiple connections that end up in > TIME_WAIT state, > over and over which suggests that the relays connect to the collectors but > the application > data isn't liked and thus connection finishes and is then retried again. > > $ netstat -an -A inet | grep 514 > tcp 0 0 0.0.0.0:514 0.0.0.0:* > LISTEN > tcp 0 0 10.142.158.71:55249 10.142.158.81:514 > TIME_WAIT > tcp 0 0 10.142.158.71:59079 10.142.158.80:514 > TIME_WAIT > tcp 0 0 10.142.158.71:55239 10.142.158.81:514 > TIME_WAIT > tcp 0 0 10.142.158.71:59073 10.142.158.80:514 > TIME_WAIT > tcp 0 0 10.142.158.71:55247 10.142.158.81:514 > TIME_WAIT > tcp 0 0 10.142.158.71:59071 10.142.158.80:514 > TIME_WAIT > tcp 0 0 10.142.158.71:59077 10.142.158.80:514 > TIME_WAIT > tcp 0 0 10.142.158.71:59069 10.142.158.80:514 > TIME_WAIT > tcp 0 0 10.142.158.71:55241 10.142.158.81:514 > TIME_WAIT > tcp 0 0 10.142.158.71:55251 10.142.158.81:514 > ESTABLISHED > tcp 0 0 10.142.158.71:55245 10.142.158.81:514 > TIME_WAIT > tcp 0 0 10.142.158.71:55243 10.142.158.81:514 > TIME_WAIT > tcp 0 0 10.142.158.71:59075 10.142.158.80:514 > TIME_WAIT > udp 0 0 0.0.0.0:514 0.0.0.0:* > > > RSYSLOG RELAY CONFIGURATION: > Relays should receive messages via UDP, TCP, and RELP. > > #### MODULES #### > > $ModLoad imuxsock # provides support for local system logging (e.g. via > logger command) > $ModLoad imklog # provides kernel logging support (previously done by > rklogd) > #$ModLoad immark # provides --MARK-- message capability > > # Loaded by default > # Provides UDP syslog reception > #$ModLoad imudp > #$UDPServerRun 514 > > # Provides TCP syslog reception > $ModLoad imtcp > $InputTCPServerRun 514 > > # Provides UDP syslog reception > $ModLoad imudp > $UDPServerRun 514 > > # Load input module relp > $ModLoad imrelp > $InputRELPServerRun 514 > > # Load output module relp > $ModLoad omrelp > > #### GLOBAL DIRECTIVES #### > > # Use default timestamp format > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > # Include all config files in /etc/rsyslog.d/ > $IncludeConfig /etc/rsyslog.d/*.conf > > # Spool files > $WorkDirectory /var/spool/rsyslog > > #### RULES #### > > (default local log stuff cut out for brevity) > > action(Name="collect02-sec" > Type="omrelp" > Target="10.142.158.81" > Port="514" > Action.ResumeInterval="5" > Timeout="5") > > action(Name="collect02-sec" > Type="omrelp" > Target="10.142.158.80" > Port="514" > Action.ExecOnlyWhenPreviousIsSuspended="on" > Action.ResumeInterval="5" > Timeout="5") > > RSYSLOG COLLECTOR CONFIGURATON: > > #### MODULES #### > > $ModLoad imuxsock # provides support for local system logging (e.g. via > logger command) > $ModLoad imklog # provides kernel logging support (previously done by > rklogd) > #$ModLoad immark # provides --MARK-- message capability > > # Load (i)nput and (o)utput (m)odules > $ModLoad imudp > $ModLoad imtcp > $ModLoad imrelp > $ModLoad omrelp > > #### TEMPLATES #### > > template(name="ByHost" type="string" > string="/var/log/remotelogs/%HOSTNAME%.log") > > #### GLOBAL DIRECTIVES #### > > # Use default timestamp format > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > # Include all config files in /etc/rsyslog.d/ > $IncludeConfig /etc/rsyslog.d/*.conf > > # Spool files > $WorkDirectory /var/spool/rsyslog > > #### RULES #### > > $AllowedSender TCP, 10.142.158.70, 10.142.158.71 > $AllowedSender UDP, 10.142.158.70, 10.142.158.71 > > #### Remote Logging > > if $fromhost-ip == ["10.142.158.70","10.142.158.71"] > then { > action(type="omfile" DynaFile="ByHost") > stop > } > > #### Local Logging > > (default local log stuff cut out for brevity) > > #### LISTENERS #### > > # Provides UDP syslog reception > $UDPServerRun 514 > > # Provides TCP syslog reception > $InputTCPServerRun 514 > > # Provides Reception of RELP > $InputRELPServerRun 514 > > I've been working on this for awhile so any help is appreciated. > > Thanks > -- > Jon Schipp, > jonschipp.com, sickbits.net > -- Jon Schipp, jonschipp.com, sickbits.net _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
