----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/6377/#review9950 -----------------------------------------------------------
Hi Patrik, thank you very much for your patch. I would personally prefer if it would be implemented as a source (I do understand windows incompatibility at the moment). However would be great to hear other opinions. I was thinking about use cases of such functionality and I've come with an improvement idea. Do you think that you could allow user to specify header name where flume will automatically populate original file name? For example if user specifies header name to "file" and configuration directory as /directory than flume will populate header "file" with content "cool.txt" for each line of "/directory/cool.txt" file. flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java <https://reviews.apache.org/r/6377/#comment21146> Nit: Could you please change the tab characters to spaces here? Jarcec - Jarek Cecho On Aug. 4, 2012, 6:59 a.m., Patrick Wendell wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/6377/ > ----------------------------------------------------------- > > (Updated Aug. 4, 2012, 6:59 a.m.) > > > Review request for Flume. > > > Description > ------- > > This patch extends the existing avro client to support a file-based spooling > mechanism. See in-line documentation for precise details, but the basic idea > is that a user can have a spool directory where files are deposited for > ingestion into flume. Once ingested the files are clearly renamed and the > implementation guarantees at-least-once delivery semantics similar to those > achieved within flume itself, even across failures and restarts of the JVM > running the code. > > I feel vaguely uneasy about building this as part of the standlone avro > client rather than as its own source. An alternative would be to build this > as a proper source (in fact, there are some ad-hoc transaction semantics used > here which would really be a better fit for a source). Interested in hearing > feedback on that as well. The benefit of having this in the avro client is > that you don't need the flume runner scripts which are not windows compatible. > > > This addresses bug FlUME-1425. > https://issues.apache.org/jira/browse/FlUME-1425 > > > Diffs > ----- > > flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java > 4a5ecae > > flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java > PRE-CREATION > flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java > PRE-CREATION > > flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java > PRE-CREATION > > flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/6377/diff/ > > > Testing > ------- > > Extensive unit tests and I also built and played with this using a stub flume > agent. If you look at the JIRA I have a configuration file for an agent that > will print out Avro events to the command line - that's helpful when testing > this. > > > Thanks, > > Patrick Wendell > >
