Whoops, this fell through the cracks, sorry about that. Responses
in-line, below:
On 08/21/2014 09:21 PM, Sebastian Kurfürst wrote:
Hey Rob and Tom,
I would definitely try using --inplace and/or --append and/or
--append-verify. And please report back how it goes, if that resolves
the issue then we can just add a note to the documentation so people
will know about this gotcha.
I did some more tests now; but the results vary. First the good news:
Using "--inplace" or "--append" works nicely as long as the log file is
not rotated/truncated. Changes are picked up by heka, all good...
However, as soon as a log file is truncated/rotated, you cannot use
rsync --append or --append-verify anymore to sync it -- because the file
on the target is bigger than on the source, it will be skipped
altogether. So that leaves us with rsync without any options that way.
Ah, yes, I see the problem. But the behavior you're describing only
happens with --append, if you use --inplace rsync will happily shorten
the file. Have you tried that?
As fas as I understand, Heka does *only* detect logfile rotation when:
a) the file size *shrinks* and b) the file handle is different.
This is close, but it's not exactly right. The "is a newer log file
available?" implementation is in this function:
https://github.com/mozilla-services/heka/blob/v0.7.0/logstreamer/reader.go#L154
Reading that code, I can see there are two cases where the file on the
file system is considered to be different than the file handle we're
holding:
1. The file system file is shorter than the one our handle is pointing to.
2. The seekjournal hash for our file no longer matches.
In the case where the file has shortened in place, #1 would not be true
(b/c our file handle still points to the same file), but #2 would.
It'd be nice if we had a better way to check if the file descriptor is
still pointing to the same file, but there's no easy way to do this
across all of the various platforms and file systems that people will
want to use, unfortunately.
While I think that generally is sensible for Heka, this breaks the rsync
way of syncing log files; as one would essentially need to do the following:
IF target file size > source file size
# logrotate took place
do a normal rsync; adding a new file, and replacing the target file
completely - leading to a new file handle and Heka to pick up the file
correctly
ELSE
do rsync --inplace (or --append); as otherwise, Heka won't see the new
messages.
END
I wrote a small script (see below) which does exactly that; maybe it's
useful for others as well. This allows to effectively sync log files in
a way that Heka picks them up.
Are *both* conditions needed for Heka to work, or would just a) work
without problems? This would solve the rsync issue effectively in the
long run, and remove the need for any additional scripts.
It should be the case that just the file getting shorter triggers that
the file is new. Really you'll have to experiment a bit more to see what
works in practice, though.
If --inplace doesn't end up working for you, it's worth mentioning that
there are some other workarounds that are worth considering:
If you stop actually rotating files in place, but instead set your
processes up to write out log files with timestamps, always appending
new files and deleting old ones, you avoid a lot of this complication.
In other words, instead of:
access.log
access.log.1
access.log.2
access.log.3
...etc...
You can use:
access.log.2014.08.29
access.log.2014.08.28
access.log.2014.08.27
access.log.2014.08.26
...etc...
This way you can still delete files as they get too old, but files are
never changing out from under an active file descriptor, and there are
far fewer edge cases to worry about. If you really want to be sure you
have to most robust logging possible, this is the way to go.
Another possible solution is to use Heka instead of rsync as your log
shipper. Then you can have Heka reading the files directly off of the
machine where they're being generated, sending the records across to the
machine where you parse them over TCP, using TLS and client certs if you
want encryption.
Anyway, hope this has been helpful. Let us know what ends up working out
best for you.
-r
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
