----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17867/ -----------------------------------------------------------
(Updated Feb. 8, 2014, 1:43 a.m.) Review request for Flume and Hari Shreedharan. Repository: flume-git Description ------- This patch provides. * A consume order feature in the Spooling directory source, which will allow the users to explicitly state in which order; oldest, youngest or randomly files should be consumed from the spooling directory. * Fixes the old implementation of selecting the file from spooling directory. Previously, each file to be consumed was selected by sorting, which might become extremly time consuming if there are many files (of the order of 10K or more). The new implementation instead do a linear scan in case when the consume order is oldest or youngest. * Updates the Flume user guide accordingly. Diffs (updated) ----- flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java a88ed6e flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java f42ed2d flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java 83522c0 flume-ng-core/src/test/java/org/apache/flume/client/avro/TestReliableSpoolingFileEventReader.java 9d708c1 flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java 503ab4d flume-ng-doc/sphinx/FlumeUserGuide.rst d120a74 Diff: https://reviews.apache.org/r/17867/diff/ Testing ------- Tested without specifying a consume order, by default oldest was used. Tested with specifying oldest, youngest and random consume order. Tested with large number of files, random order is fastest. Thanks, Muhammad Ehsan ul Haque
