why don't you use directory spooling ?
On Tue, Jan 22, 2013 at 7:15 AM, Henry Ma <[email protected]> wrote: > Hi, > > When using Flume to collect log files, we want to just COPY the original > files from several servers to a central storage (unix file system), not to > roll up to a big file. Because we must record some messages of the original > file such as name, host, path, timestamp, etc. Besides, we want to > guarantee total reliability: no file miss, no file reduplicated. > > It seems that, in Source, we must put a whole file (size may be between > 100KB and 100MB) into a Flume event; and in Sink, we must write each event > to a single file. > > Is it practicable? Thanks! > > -- > Best Regards, > Henry Ma > -- Nitin Pawar
