Hrm. Most likely sitch is that there's a race condition where Heka is trying to
find it's new location in the log stream but it's still operating with the
prior (i.e. no longer valid) set of file names. That would explain both why it
didn't find the new location in the stream (b/c that has a file name that
didn't exist prior) and why the journal is pointing at a file that no longer
exists.
I'll open an issue for this.
-r
On 02/03/2016 11:05 AM, Dai Shi wrote:
Hello,
I've only been using heka for a couple months, and this is my first time
posting to this mailing list. I've been running into the following issue
with logstreamer input, and was hoping someone on here might point me in
the right direction.
Every so often, when logrotate rotates logs, the logstreamer input
appears to start tailing the oldest log, which gets deleted during the
rotate, and then it will just get stuck there and not make any progress.
Here is my logstreamer config:
[MongoTailer]
type = "LogstreamerInput"
log_directory = "/data/log/mongo"
journal_directory = "/data/appdata/heka"
# expect files named mongod.<rset>.log[-YYYYMMDD]
file_match =
'mongod\.(?P<Rset>\S+)\.log-?((?P<Year>\d{4})(?P<Month>\d{2})(?P<Day>\d{2}))?'
differentiator = ["all.", "Rset"]
priority = ["Year", "Month", "Day"]
# make the log ending in .log the most recent
[MongoTailer.translation.Year]
missing = 9999
Running heka-logstreamer on the config shows what I expect:
Logstream name: [all.venues0]
Files: 8 (printing oldest to newest)
/data/log/mongo/mongod.venues0.log-20160121
/data/log/mongo/mongod.venues0.log-20160122
/data/log/mongo/mongod.venues0.log-20160123
/data/log/mongo/mongod.venues0.log-20160124
/data/log/mongo/mongod.venues0.log-20160125
/data/log/mongo/mongod.venues0.log-20160126
/data/log/mongo/mongod.venues0.log-20160127
/data/log/mongo/mongod.venues0.log
During normal operation, looking at the journal file also shows what I
expect:
{"seek":79749937,"file_name":"/data/log/mongo/mongod.venues0.log","last_hash":"be40779a9aaa7371ee8a53fc448ba10d7c084fe3"}
But, sometimes when logrotate does rotation, the journal file will then
show:
{"seek":55436986,"file_name":"/data/log/mongo/mongod.venues0.log-20160120","last_hash":"ad99fe64c5f6cba6a2a197dfa14cc10c583cf712"}
The file it appears to tail no longer exists, and it will just sit there
until I bounce hekad. I'm not sure why it would go from tailing the
newest file to tailing the oldest file. After bouncing it will go back
through all the old logs again.
Any ideas what could be happening?
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
