it's defintely the log mover script (which does what I described in
the original email). If I don't run it the rsyslog continues to work
without any problems.
I am trying to isolate what could be cause the problem but it's
tricky, as far as I can tell there are these actions that can have
impact on rsyslog:
- rename of the file
- reload (i.e. HUP signal)
- remove of the file
If I don't do reload it still behaves the same way. I guess I can try
to not remove the file either and see what happens.
erik
On 11/01/2013 01:06 AM, Radu Gheorghe wrote:
Hi Erik,
I don't have any idea on why this problem appears (maybe others will), but
I'd try to "isolate" the problematic bit. Maybe it's the cron job that
provokes something. Or the dynafile or the 6 hosts or...
2013/11/1 Erik Steffl <[email protected]>
We have a fairly simple setup of 6 hosts sending syslog messages to one
collector host, all of these run rsyslog 7.5.5-0adiscon2 from adiscon repo
and use RELP to transfer messages. There is also load balancer in front of
the collector machine but I dont' think it matters in this case.
Rsyslog on collector machine is configured to write to files, switching to
a new file every 15 minute, using config like this (abbreviated a bit):
template(name="jsonFilename" type="list") {
constant(value="/path/")
property(name="$now")
constant(value="/")
property(name="$hour")
constant(value="/")
property(name="$qhour")
constant(value="/")
constant(value="log.json")
}
action(type="omfile" DynaFile="jsonFilename" Template="jsonFormat")
We run a script at every 2, 17, 32, 47 minute of the hour and upload the
just finished file to S3. The uploading works like this:
- let's say it's 3:02:00, rsyslog is writing to
/path/2013-10-10/03/00/log.**json
- get the filename log.json (anything that's not current, usually just
one previous file which in the example would be /path/2013-10-10/02/03/log.
**json)
- rename /path/2013-10-10/02/03/log.**json to
/path/2013-10-10/02/03/log.**json.uploading.0
- reload rsyslog (to make sure that even if for some reason it was
writing to just renamed file it would close it and open a new file)
- upload /path/2013-10-10/02/03/log.**json.uploading.0 to S3
- remove /path/2013-10-10/02/03/log.**json.uploading.0
Here's what happens every third run (yes, regularly EVERY THIRD RUN) of
this script:
- rsyslog stops writing to the CURRENT file (/path/2013-10-10/03/00/log.
**json, the one that is NOT being renamed) few seconds into the run of
the script (e.g. 3:02:04)
- 6 hosts that were sending syslog messages to the log collector STOP
sending anything (as verified by stracing rsyslogd, tcpdump and in amazon
AWS console metric for network in)
- after this nothing is ever written into /path/2013-10-10/03/00/log.**
json
- the 6 clients start sending sysog messages again when the next file is
created (in this example it would be /path/2013-10-10/03/01/log.**json)
I checked and double check the files, dates, verified that the current
file is not touched but can't figure out what's going on. I tried the
script without reload rsyslog but it didn't make any difference. If I don't
run this script rsyslog works flawlessly.
any ideas how to troubleshoot this? What could be causing the rsyslog to
stop writing to the file and for the senders to stop sending syslog
messages? I assume the rsyslog on the collector host somehow signals to the
6 hosts that send messages that it's not ready or something...
thanks!
erik
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
