-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17867/
-----------------------------------------------------------
(Updated Feb. 8, 2014, 12:58 p.m.)
Review request for Flume and Hari Shreedharan.
Changes
-------
Linked with JIRA ID
Bugs: FLUME-2309
https://issues.apache.org/jira/browse/FLUME-2309
Repository: flume-git
Description
-------
This patch provides.
* A consume order feature in the Spooling directory source, which will allow
the users to explicitly state in which order; oldest, youngest or randomly
files should be consumed from the spooling directory.
* Fixes the old implementation of selecting the file from spooling directory.
Previously, each file to be consumed was selected by sorting, which might
become extremly time consuming if there are many files (of the order of 10K or
more). The new implementation instead do a linear scan in case when the consume
order is oldest or youngest.
* Updates the Flume user guide accordingly.
Diffs
-----
flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java
a88ed6e
flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java
f42ed2d
flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java
83522c0
flume-ng-core/src/test/java/org/apache/flume/client/avro/TestReliableSpoolingFileEventReader.java
9d708c1
flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java
503ab4d
flume-ng-doc/sphinx/FlumeUserGuide.rst d120a74
Diff: https://reviews.apache.org/r/17867/diff/
Testing
-------
Tested without specifying a consume order, by default oldest was used.
Tested with specifying oldest, youngest and random consume order.
Tested with large number of files, random order is fastest.
Thanks,
Muhammad Ehsan ul Haque
