On Mon, 9 Sep 2013, Erik Steffl wrote:
On 09/07/2013 06:01 AM, Rainer Gerhards wrote:
On Sat, Sep 7, 2013 at 12:10 PM, Erik Steffl <[email protected]> wrote:
Is there any way to make rsyslog change the filename when it's done
writing to it?
Well, let me ask a counter-question: how do you think rsyslog can know it
is done with a file?
:) exactly
In fact, it doesn't know, as whenever it thinks it is done, the very next
message may come up with data that invalidates that assumption. So from the
rsyslog PoV, a file is never "done", but may be evicted from cache due to
not being used frequently enough (but even cache eviction doesn't mean the
file will be re-opened again).
I agree that with some system properties - mostly $month and friends -
rsyslog may be able to get a clue when it is "done". But even there are
subtle issue, like DST, where being done based on "$HOUR" isn't really sure
at 2:30 in the morning ;).
If I were to add a feature for "being done", it would probably materialize
around a single fixed-format timestamp property, which would clearly
indicate that the same file cannot be used again (but even then people
could set the system clock back in time...). Such a feature would probably
require a sponsor and may not be as easy to implement as it sounds.
Note that e.g. $hour inside a template is not sufficient - e.g. a template
"/var/log/mail%$hour$%.log" would never be done, as the same file name
would be re-used each 24 hours!
I get that it's pretty hard to mark files as done (given that time can
change on machine and rsyslog does not have a persistent storage where it
could atomically update the filenames that are done). Maybe combination of
time and random or sort of random number would work, together with renaming
of the file after it's done (so if rsyslog needs to write to same timestamp
it would get new random number). Of course adding a random number would mean
one cannot order the files by filename only, pretty big disadvantage...
It's not just this, how would rsyslog know that you are never going to send it a
log that needs to go to that file so that it would know to create a new file in
the first place.
For right now I decided to go with timestmap being part of pathname but
also do what logrotate does (rename the file, reload rsyslog, then move file
to S3).
The cases where I move the file but rsyslog writes to same filename again
will have to be handled during upload. If the S3 object already exist then
create new S3 object (same key but add a sequence number). This should happen
very rarely, virtually never.
this is a really good reason to run your log servers on GMT so that they are not
affected by DST. you may still have problems, but only if something goes wrong,
not as part of normal operations.
David Lang
thanks!
erik
Rainer
Our current strategy is to dynamically create files (using omfile and
DynaFile with date/time).
The obvious solution is to look whether the file is old enough (compare
the date/time in filename to current time).
Is there a better solution? Ideally I would have rsyslog rename the
file
once it's done with it.
Using logrotate would not work in our case, we need the date/time (and
other information) in the filename, it is used as S3 key.
thanks!
erik
______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
