On 04/13/2015 08:53 AM, M GS wrote:
Is it possible to start the LogstreamerInput without catching up old logs?
No, but it's becoming clear that we should maybe add this. I'm thinking a boolean setting that, if true *and* there's no cursor file for the logstream, will cause the processing to start at the tail rather than the head of the stream. I've opened an issue for this: https://github.com/mozilla-services/heka/issues/1482
In my use case oldest_duration is not enough because the log is constantly being updated.
Not sure I understand what this means. Is it that even the current log file is too big to process? If so, maybe you want to consider rotation your logs a bit more often?
One wrinkle in all of this is that we've realized there's a bug in the oldest_duration code, causing it not to be honored; I've got it fixed on a branch, am expecting to land that on the versions/0.9 branch today, so it will go out in the 0.9.2 release that I'm hoping to make later today or tomorrow. Maybe oldest_duration would be sufficient, if it were actually working correctly.
The issue this causes is that parsing large log file attempts to inject too many messages when heka first starts up. I realize that we will end up loosing messages this way, but as it stands its very hard to deploy (ie: we have to rotate the logs manually, then start heka). Ideally what I do want is to parse all our logs, but I can't do this on a production machine anyway. What I would rather do is have heka start fresh each time when in production.
Hopefully the oldest_duration fix in 0.9.2 will help you at least a bit, otherwise we'll work on getting #1482 implemented. Sorry for the headache. -r _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
