A lot of people use an execSource attached to tail -F
It's not 100 reliable but it's a lot less work than writing up a custom
program
On 04/03/2012 02:06 PM, Mohit Anchlia wrote:
May be I got it wrong. I thought it's pretty common use case to send
continuous feed of logs as data is written to the logs. How do other
people design with Flume NG when using logs like apache, web server or
other kinds of logs?
On Mon, Apr 2, 2012 at 8:34 PM, Juhani Connolly
<juhani_conno...@cyberagent.co.jp
<mailto:juhani_conno...@cyberagent.co.jp>> wrote:
Easiest way is to make some kind of daemon process that regularly
stat the file, checking for changes, and stores a position within
the file up to which has been read/sent.
Some things you might want to watch out for:
- Shrinking file generally means a log roll
- Change in inode means the old file has been moved and a new one
has been made with the same name. You might want to follow the old
one, or at least check if it has any unflushed changes left(since
the concept of inodes is platform specific, you can't do this with
all languages)
You may want to look at the TailSource code from Flume OG for some
general ideas.
Just to warn you however, that tracking all the possible things
that can happen to a file can be pretty messy.
On 04/03/2012 07:07 AM, Mohit Anchlia wrote:
I just ran avro client and saw that it streamed entire file
and exited. How can I design a client that continuously keeps
forwarding the changes to the log file?
