It looks like the syslog receiver is setup for TCP on port 514, but the senders are sending RELP (over TCP) on 514.
You have both plain TCP and RELP on port 514 on the receiver. Disable the TCP or pick different ports for plain TCP and RELP.... It is interesting that rsyslog didn't complain - are there any messages during rsyslog start on the receiver? Alan Edmonds ________________________________________ From: [email protected] [[email protected]] On Behalf Of Jon Schipp [[email protected]] Sent: 24 September 2013 22:53 To: [email protected] Subject: Re: [rsyslog] RELP messages not sending/receiving properly Update: I have logs in the remote log directory but they're strange. o and pen are not hosts, I imagine the word "open" was separated. Also, the logs from the relays are nothing but the relp version lines. Seems that this problem may be on the relays now. $ head -3 * ==> 10.142.158.70.log <== Sep 24 15:34:25 10.142.158.70 relp_software=librelp,1.0.6,http: // librelp.adiscon.com Sep 24 15:34:25 10.142.158.70 commands=syslog Sep 24 15:34:45 10.142.158.70 relp_software=librelp,1.0.6,http: // librelp.adiscon.com ==> 10.142.158.71.log <== Sep 24 15:34:35 10.142.158.71 relp_software=librelp,1.0.6,http: // librelp.adiscon.com Sep 24 15:34:35 10.142.158.71 commands=syslog Sep 24 15:34:55 10.142.158.71 relp_software=librelp,1.0.6,http: // librelp.adiscon.com ==> o.log <== Sep 24 15:34:25 o Sep 24 15:34:35 o Sep 24 15:34:45 o ==> pen.log <== Sep 24 15:34:25 pen 85 relp_version=0 Sep 24 15:34:35 pen 85 relp_version=0 Sep 24 15:34:45 pen 85 relp_version=0 On Tue, Sep 24, 2013 at 4:04 PM, Jon Schipp <[email protected]> wrote: > Hello, > > I'm having an issue where messages sent from rsyslog relays via RELP to > rsyslog collectors that write them to disk via DynFile (%HOSTNAME% > template) are not looking right and causing problems. I don't think the > RELP protocol is negotiating properly as the syslog messages are not > seen on the wire and the queue is building up on the relays. > > $ ls -l /var/spool/rsyslog/ > total 69892 > -rw------- 1 root root 1049135 Sep 24 11:36 rsyslog-buffer.00000001 > -rw------- 1 root root 1048593 Sep 24 12:09 rsyslog-buffer.00000002 > -rw------- 1 root root 1049021 Sep 18 22:19 rsyslog-buffer.00000003 > -rw------- 1 root root 1049181 Sep 19 00:32 rsyslog-buffer.00000004 > -rw------- 1 root root 1048907 Sep 19 02:01 rsyslog-buffer.00000005 > -rw------- 1 root root 1049049 Sep 19 04:13 rsyslog-buffer.00000006 > -rw------- 1 root root 1048661 Sep 19 06:25 rsyslog-buffer.00000007 > .... > > Systems are running compiled with RELP via rsyslog's rpm repo. > $ rsyslogd -v > rsyslogd 7.4.4, compiled with: > FEATURE_REGEXP: Yes > FEATURE_LARGEFILE: No > GSSAPI Kerberos 5 support: Yes > FEATURE_DEBUG (debug build, slow code): No > 32bit Atomic operations supported: Yes > 64bit Atomic operations supported: Yes > Runtime Instrumentation (slow code): No > uuid support: Yes > > Here is dump of the syslog messages seen by the collectors. This message > is seen over and over. > > 115, options [nop,nop,TS val 141682 ecr 152580], length 96 > 0x0000: 4500 0094 c32b 4000 4006 2585 0a8e 9e47 E....+@.@.%....G > 0x0010: 0a8e 9e50 e6a9 0202 6808 ea5c 8961 44fd ...P....h..\.aD. > 0x0020: 8018 0073 5936 0000 0101 080a 0002 2972 ...sY6........)r > 0x0030: 0002 5404 3120 6f70 656e 2038 3520 7265 ..T.1.open.85.re > 0x0040: 6c70 5f76 6572 7369 6f6e 3d30 0a72 656c lp_version=0.rel > 0x0050: 705f 736f 6674 7761 7265 3d6c 6962 7265 p_software=libre > 0x0060: 6c70 2c31 2e30 2e36 2c68 7474 703a 2f2f lp,1.0.6,http:// > 0x0070: 6c69 6272 656c 702e 6164 6973 636f 6e2e librelp.adiscon. > 0x0080: 636f 6d0a 636f 6d6d 616e 6473 3d73 7973 com.commands=sys > 0x0090: 6c6f 670a log. > > Each of the rsyslog machines have multiple connections that end up in > TIME_WAIT state, > over and over which suggests that the relays connect to the collectors but > the application > data isn't liked and thus connection finishes and is then retried again. > > $ netstat -an -A inet | grep 514 > tcp 0 0 0.0.0.0:514 0.0.0.0:* > LISTEN > tcp 0 0 10.142.158.71:55249 10.142.158.81:514 > TIME_WAIT > tcp 0 0 10.142.158.71:59079 10.142.158.80:514 > TIME_WAIT > tcp 0 0 10.142.158.71:55239 10.142.158.81:514 > TIME_WAIT > tcp 0 0 10.142.158.71:59073 10.142.158.80:514 > TIME_WAIT > tcp 0 0 10.142.158.71:55247 10.142.158.81:514 > TIME_WAIT > tcp 0 0 10.142.158.71:59071 10.142.158.80:514 > TIME_WAIT > tcp 0 0 10.142.158.71:59077 10.142.158.80:514 > TIME_WAIT > tcp 0 0 10.142.158.71:59069 10.142.158.80:514 > TIME_WAIT > tcp 0 0 10.142.158.71:55241 10.142.158.81:514 > TIME_WAIT > tcp 0 0 10.142.158.71:55251 10.142.158.81:514 > ESTABLISHED > tcp 0 0 10.142.158.71:55245 10.142.158.81:514 > TIME_WAIT > tcp 0 0 10.142.158.71:55243 10.142.158.81:514 > TIME_WAIT > tcp 0 0 10.142.158.71:59075 10.142.158.80:514 > TIME_WAIT > udp 0 0 0.0.0.0:514 0.0.0.0:* > > > RSYSLOG RELAY CONFIGURATION: > Relays should receive messages via UDP, TCP, and RELP. > > #### MODULES #### > > $ModLoad imuxsock # provides support for local system logging (e.g. via > logger command) > $ModLoad imklog # provides kernel logging support (previously done by > rklogd) > #$ModLoad immark # provides --MARK-- message capability > > # Loaded by default > # Provides UDP syslog reception > #$ModLoad imudp > #$UDPServerRun 514 > > # Provides TCP syslog reception > $ModLoad imtcp > $InputTCPServerRun 514 > > # Provides UDP syslog reception > $ModLoad imudp > $UDPServerRun 514 > > # Load input module relp > $ModLoad imrelp > $InputRELPServerRun 514 > > # Load output module relp > $ModLoad omrelp > > #### GLOBAL DIRECTIVES #### > > # Use default timestamp format > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > # Include all config files in /etc/rsyslog.d/ > $IncludeConfig /etc/rsyslog.d/*.conf > > # Spool files > $WorkDirectory /var/spool/rsyslog > > #### RULES #### > > (default local log stuff cut out for brevity) > > action(Name="collect02-sec" > Type="omrelp" > Target="10.142.158.81" > Port="514" > Action.ResumeInterval="5" > Timeout="5") > > action(Name="collect02-sec" > Type="omrelp" > Target="10.142.158.80" > Port="514" > Action.ExecOnlyWhenPreviousIsSuspended="on" > Action.ResumeInterval="5" > Timeout="5") > > RSYSLOG COLLECTOR CONFIGURATON: > > #### MODULES #### > > $ModLoad imuxsock # provides support for local system logging (e.g. via > logger command) > $ModLoad imklog # provides kernel logging support (previously done by > rklogd) > #$ModLoad immark # provides --MARK-- message capability > > # Load (i)nput and (o)utput (m)odules > $ModLoad imudp > $ModLoad imtcp > $ModLoad imrelp > $ModLoad omrelp > > #### TEMPLATES #### > > template(name="ByHost" type="string" > string="/var/log/remotelogs/%HOSTNAME%.log") > > #### GLOBAL DIRECTIVES #### > > # Use default timestamp format > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > # Include all config files in /etc/rsyslog.d/ > $IncludeConfig /etc/rsyslog.d/*.conf > > # Spool files > $WorkDirectory /var/spool/rsyslog > > #### RULES #### > > $AllowedSender TCP, 10.142.158.70, 10.142.158.71 > $AllowedSender UDP, 10.142.158.70, 10.142.158.71 > > #### Remote Logging > > if $fromhost-ip == ["10.142.158.70","10.142.158.71"] > then { > action(type="omfile" DynaFile="ByHost") > stop > } > > #### Local Logging > > (default local log stuff cut out for brevity) > > #### LISTENERS #### > > # Provides UDP syslog reception > $UDPServerRun 514 > > # Provides TCP syslog reception > $InputTCPServerRun 514 > > # Provides Reception of RELP > $InputRELPServerRun 514 > > I've been working on this for awhile so any help is appreciated. > > Thanks > -- > Jon Schipp, > jonschipp.com, sickbits.net > -- Jon Schipp, jonschipp.com, sickbits.net _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
