Ok, that sounds like what RELP is designed to do, now we just need to figure out
why the connection is interrupted. a HUP is only supposed to close outbound
connections and files, it's not supposed to affect inbound traffic at all.
I'm suspicious of upstart because 'reload' can imply that it's trying to update
for any config changes, and that requires a complete shutdown and restart.
David Lang
On Thu, 31 Oct 2013, Erik Steffl wrote:
forgot one imporant piece of information: it seems that no messages are
lost, the messages not delivered during the 15 minutes when there's no
traffic are delivered right after it starts working again (just like RELP
should)
the traffic pattern is 30 minutes of regular traffic, 15 minutes of
nothing, few minutes of very high traffic followed by 30 minutes of regular
traffic, then 15 minutes of not traffic etc.
erik
On 10/31/2013 06:44 PM, Erik Steffl wrote:
We have a fairly simple setup of 6 hosts sending syslog messages to one
collector host, all of these run rsyslog 7.5.5-0adiscon2 from adiscon
repo and use RELP to transfer messages. There is also load balancer in
front of the collector machine but I dont' think it matters in this case.
Rsyslog on collector machine is configured to write to files, switching
to a new file every 15 minute, using config like this (abbreviated a bit):
template(name="jsonFilename" type="list") {
constant(value="/path/")
property(name="$now")
constant(value="/")
property(name="$hour")
constant(value="/")
property(name="$qhour")
constant(value="/")
constant(value="log.json")
}
action(type="omfile" DynaFile="jsonFilename" Template="jsonFormat")
We run a script at every 2, 17, 32, 47 minute of the hour and upload
the just finished file to S3. The uploading works like this:
- let's say it's 3:02:00, rsyslog is writing to
/path/2013-10-10/03/00/log.json
- get the filename log.json (anything that's not current, usually
just one previous file which in the example would be
/path/2013-10-10/02/03/log.json)
- rename /path/2013-10-10/02/03/log.json to
/path/2013-10-10/02/03/log.json.uploading.0
- reload rsyslog (to make sure that even if for some reason it was
writing to just renamed file it would close it and open a new file)
- upload /path/2013-10-10/02/03/log.json.uploading.0 to S3
- remove /path/2013-10-10/02/03/log.json.uploading.0
Here's what happens every third run (yes, regularly EVERY THIRD RUN) of
this script:
- rsyslog stops writing to the CURRENT file
(/path/2013-10-10/03/00/log.json, the one that is NOT being renamed) few
seconds into the run of the script (e.g. 3:02:04)
- 6 hosts that were sending syslog messages to the log collector STOP
sending anything (as verified by stracing rsyslogd, tcpdump and in
amazon AWS console metric for network in)
- after this nothing is ever written into
/path/2013-10-10/03/00/log.json
- the 6 clients start sending sysog messages again when the next file
is created (in this example it would be /path/2013-10-10/03/01/log.json)
I checked and double check the files, dates, verified that the
current file is not touched but can't figure out what's going on. I
tried the script without reload rsyslog but it didn't make any
difference. If I don't run this script rsyslog works flawlessly.
any ideas how to troubleshoot this? What could be causing the rsyslog
to stop writing to the file and for the senders to stop sending syslog
messages? I assume the rsyslog on the collector host somehow signals to
the 6 hosts that send messages that it's not ready or something...
thanks!
erik
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
