> On Aug. 7, 2012, 7:09 a.m., Jarek Cecho wrote: > > Hi Patrik, > > thank you very much for your patch. I would personally prefer if it would > > be implemented as a source (I do understand windows incompatibility at the > > moment). However would be great to hear other opinions. > > > > I was thinking about use cases of such functionality and I've come with an > > improvement idea. Do you think that you could allow user to specify header > > name where flume will automatically populate original file name? For > > example if user specifies header name to "file" and configuration directory > > as /directory than flume will populate header "file" with content > > "cool.txt" for each line of "/directory/cool.txt" file.
Let me re-implement this as a source, and maybe we can find a way to also include it in the avro client. The filename suggestion is also a good idea. I'll plan to add this in the next rev of the patch. We could also add that as a feature and do in a second patch depending on how much review bandwidth you have. - Patrick ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/6377/#review9950 ----------------------------------------------------------- On Aug. 4, 2012, 6:59 a.m., Patrick Wendell wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/6377/ > ----------------------------------------------------------- > > (Updated Aug. 4, 2012, 6:59 a.m.) > > > Review request for Flume. > > > Description > ------- > > This patch extends the existing avro client to support a file-based spooling > mechanism. See in-line documentation for precise details, but the basic idea > is that a user can have a spool directory where files are deposited for > ingestion into flume. Once ingested the files are clearly renamed and the > implementation guarantees at-least-once delivery semantics similar to those > achieved within flume itself, even across failures and restarts of the JVM > running the code. > > I feel vaguely uneasy about building this as part of the standlone avro > client rather than as its own source. An alternative would be to build this > as a proper source (in fact, there are some ad-hoc transaction semantics used > here which would really be a better fit for a source). Interested in hearing > feedback on that as well. The benefit of having this in the avro client is > that you don't need the flume runner scripts which are not windows compatible. > > > This addresses bug FlUME-1425. > https://issues.apache.org/jira/browse/FlUME-1425 > > > Diffs > ----- > > flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java > 4a5ecae > > flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java > PRE-CREATION > flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java > PRE-CREATION > > flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java > PRE-CREATION > > flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/6377/diff/ > > > Testing > ------- > > Extensive unit tests and I also built and played with this using a stub flume > agent. If you look at the JIRA I have a configuration file for an agent that > will print out Avro events to the command line - that's helpful when testing > this. > > > Thanks, > > Patrick Wendell > >
