a HUP of the server should not cause any interruption in processing messages.
are you sure upstart is seing HUP not doing a full stop/start?
could you try sending HUP to rsyslog manually a few times to see if you can
duplicate the problem?
David Lang
On Fri, 1 Nov 2013, Erik Steffl wrote:
sorry, it wasn't entirely clear in my previous email but we are doing
ubuntu upstart reload which is sending HUP signal, pretty sure it's doing the
right thing since logrotate scripts use it, syslog message says rsyslog
received HUP signal, rsyslogd keeps the same PID and all files are closed (as
verified using lsof).
On top of that if, during one of these quiet periods, I try to send
messages from another client (on which I just restarted rsyslog so it has no
error counters etc.) I think it's also not sending anything, when stracing it
(the sender) I see it recieved messages from logger but is not sending them
anywhere. So I think that the collector rsyslog continues to send some kind
of signal that it's not ready but not sure why or how to figure out whether
it actually does it.
Any ideas how to troubleshoot this? I think I'll record more of strace
output and try to reconstruct RELP messages, I guess there is some RELP
response that has something else than RSP-CODE = 200 (looking at
http://www.rsyslog.com/doc/relp.html)
I was just about to publish the S3 upload scripts but since they cause
rsyslog to misbehave this way I didn't do it yet, maybe I'll post them as
they are and fix later...
erik
On 11/01/2013 01:02 AM, David Lang wrote:
the purpose of RELP is to tell the senders if the receiver was able to
accept the message or not, so that's working as designed.
When you do a full restart of rsyslog, it stops processing new messages
and disconnects all senders (you are doing a full stop of rsyslog and
then starting it from scratch, this takes time). My guess is that the
senders are detecting 'too many failures' when trying to send messages,
so they are backing off and sleeping for a while rather tthan performing
a mini DOS attack on the network and server. Every third restart is
probably triggering a threshold.
Instead of doing a full restart, just send the rsyslog daemon a HUP
signal instead. That will tell rsyslog to flush and close all files so
that you can rotate them (If you are doing compression on the files, you
may need to sleep for a few seconds to let the fluch complete). Rsyslog
can continue to receive new messages during this time, so your senders
will not see an outage.
By the way, I'm needing a script to upload rsyslog archives to S3, could
you send me a copy of yours? (remove any passwords first please :-)
David Lang
On Thu, 31 Oct 2013, Erik Steffl wrote:
Date: Thu, 31 Oct 2013 18:44:23 -0700
From: Erik Steffl <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: [rsyslog] Rsyslog with RELP not sending/receiving messages
for long
intervals
We have a fairly simple setup of 6 hosts sending syslog messages to
one collector host, all of these run rsyslog 7.5.5-0adiscon2 from
adiscon repo and use RELP to transfer messages. There is also load
balancer in front of the collector machine but I dont' think it
matters in this case.
Rsyslog on collector machine is configured to write to files,
switching to a new file every 15 minute, using config like this
(abbreviated a bit):
template(name="jsonFilename" type="list") {
constant(value="/path/")
property(name="$now")
constant(value="/")
property(name="$hour")
constant(value="/")
property(name="$qhour")
constant(value="/")
constant(value="log.json")
}
action(type="omfile" DynaFile="jsonFilename" Template="jsonFormat")
We run a script at every 2, 17, 32, 47 minute of the hour and upload
the just finished file to S3. The uploading works like this:
- let's say it's 3:02:00, rsyslog is writing to
/path/2013-10-10/03/00/log.json
- get the filename log.json (anything that's not current, usually
just one previous file which in the example would be
/path/2013-10-10/02/03/log.json)
- rename /path/2013-10-10/02/03/log.json to
/path/2013-10-10/02/03/log.json.uploading.0
- reload rsyslog (to make sure that even if for some reason it was
writing to just renamed file it would close it and open a new file)
- upload /path/2013-10-10/02/03/log.json.uploading.0 to S3
- remove /path/2013-10-10/02/03/log.json.uploading.0
Here's what happens every third run (yes, regularly EVERY THIRD RUN)
of this script:
- rsyslog stops writing to the CURRENT file
(/path/2013-10-10/03/00/log.json, the one that is NOT being renamed)
few seconds into the run of the script (e.g. 3:02:04)
- 6 hosts that were sending syslog messages to the log collector STOP
sending anything (as verified by stracing rsyslogd, tcpdump and in
amazon AWS console metric for network in)
- after this nothing is ever written into
/path/2013-10-10/03/00/log.json
- the 6 clients start sending sysog messages again when the next file
is created (in this example it would be /path/2013-10-10/03/01/log.json)
I checked and double check the files, dates, verified that the
current file is not touched but can't figure out what's going on. I
tried the script without reload rsyslog but it didn't make any
difference. If I don't run this script rsyslog works flawlessly.
any ideas how to troubleshoot this? What could be causing the rsyslog
to stop writing to the file and for the senders to stop sending syslog
messages? I assume the rsyslog on the collector host somehow signals
to the 6 hosts that send messages that it's not ready or something...
thanks!
erik
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
