----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/6377/#review12334 -----------------------------------------------------------
This responds mostly to the lower level comments - not the high re-architecting comments. I am going to defer those to an in person discussion. flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java <https://reviews.apache.org/r/6377/#comment26109> There is another case here that can affect even non-windows users. What if they run Flume, it sends a few files (say it sends 2 out of the 10 files in the directory) then the stop it. Then they run Flume again but they don't clean up the files that were sent. The first time flume copies one of these files that was already sent, it will try to do a rename to make the exact same file, and you could argue that Flume should give an error message and move on, rather than die. What do you think? I could also see maybe adding a check at start-up to see if there are any duplicate (file, file_renamed) pairs and failing there. Alternatively, it might be reasonable to just fail here if the duplicate exists. flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java <https://reviews.apache.org/r/6377/#comment26110> changed to a log warning... might still be helpful for people to know when this is happening in case they are mis-using it. flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java <https://reviews.apache.org/r/6377/#comment26118> Done - this also makes the testing way easier. flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java <https://reviews.apache.org/r/6377/#comment26119> I'm deferring this change until I do the broader re-architecture, because this exception will likely be removed. flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java <https://reviews.apache.org/r/6377/#comment26120> Why make it static? A PoolDirectoryRunnable will only be created when there is a SpoolDirectorySource that is using it... right? - Patrick Wendell On Aug. 14, 2012, 10:02 p.m., Patrick Wendell wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/6377/ > ----------------------------------------------------------- > > (Updated Aug. 14, 2012, 10:02 p.m.) > > > Review request for Flume. > > > Description > ------- > > This patch adds a spooling directory based source. The idea is that a user > can have a spool directory where files are deposited for ingestion into > flume. Once ingested, the files are clearly renamed and the implementation > guarantees at-least-once delivery semantics similar to those achieved within > flume itself, even across failures and restarts of the JVM running the code. > > This helps fill the gap for people who want a way to get reliable delivery of > events into flume, but don't want to directly write their application against > the flume API. They can simply drop log files off in a spooldir and let flume > ingest asynchronously (using some shell scripts or other automated process). > > Unlike the prior iteration, this patch implements a first-class source. It > also extends the avro client to support spooling in a similar manner. > > > This addresses bug FlUME-1425. > https://issues.apache.org/jira/browse/FlUME-1425 > > > Diffs > ----- > > > flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java > da804d7 > > flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java > abbbf1c > flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java > 4a5ecae > > flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java > PRE-CREATION > flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java > PRE-CREATION > > flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java > PRE-CREATION > > flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java > PRE-CREATION > > flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java > PRE-CREATION > > flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java > PRE-CREATION > flume-ng-doc/sphinx/FlumeUserGuide.rst 45dd7cc > > Diff: https://reviews.apache.org/r/6377/diff/ > > > Testing > ------- > > Extensive unit tests and I also built and played with this using a stub flume > agent. If you look at the JIRA I have a configuration file for an agent that > will print out Avro events to the command line - that's helpful when testing > this. > > > Thanks, > > Patrick Wendell > >
