-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6377/
-----------------------------------------------------------
(Updated Aug. 11, 2012, 5:32 a.m.)
Review request for Flume.
Changes
-------
This patch substantially changes the implementation. It:
1) Does this as a source in addition a feature of the avro client, by factoring
out functionality into a shared class.
2) Adds the header functionality suggested by Jarcek.
3) Adds relevant documentation in the user guide and several unit tests.
It's probably easier to just review this patch rather than diff from the
earlier version, given the scope and extent of the changes.
Description
-------
This patch extends the existing avro client to support a file-based spooling
mechanism. See in-line documentation for precise details, but the basic idea is
that a user can have a spool directory where files are deposited for ingestion
into flume. Once ingested the files are clearly renamed and the implementation
guarantees at-least-once delivery semantics similar to those achieved within
flume itself, even across failures and restarts of the JVM running the code.
I feel vaguely uneasy about building this as part of the standlone avro client
rather than as its own source. An alternative would be to build this as a
proper source (in fact, there are some ad-hoc transaction semantics used here
which would really be a better fit for a source). Interested in hearing
feedback on that as well. The benefit of having this in the avro client is that
you don't need the flume runner scripts which are not windows compatible.
This addresses bug FlUME-1425.
https://issues.apache.org/jira/browse/FlUME-1425
Diffs (updated)
-----
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java
da804d7
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java
abbbf1c
flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java
4a5ecae
flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java
PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java
PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java
PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java
PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java
PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java
PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java
PRE-CREATION
flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java
PRE-CREATION
flume-ng-doc/sphinx/FlumeUserGuide.rst 45dd7cc
Diff: https://reviews.apache.org/r/6377/diff/
Testing
-------
Extensive unit tests and I also built and played with this using a stub flume
agent. If you look at the JIRA I have a configuration file for an agent that
will print out Avro events to the command line - that's helpful when testing
this.
Thanks,
Patrick Wendell
