> On June 21, 2016, 7:02 p.m., Hari Shreedharan wrote: > > Since Flume 1.7+ will now only support Java 7+, why don't we consider using > > Java 7's new DirectoryStream API, rather than the much more expensive > > listFiles API? This probably will result in a huge performance boost plus > > far simpler code structure. It will require some rewrite of the code, but > > since we have tests and we expect current behavior, I would suggest doing > > that.
According to the current parent pom.xml, I think java 7 is currently not supported in Flume trunk. The DirectoryStream API patch is done. It will be submitted as soon as java target version is bumped in pom.xml. If it's okay with you, I'd like to have this review shipped. I'll be creating a new ticket that depends on the java 7 bump and switches listFiles to DirectoryStream API. See also: https://issues.apache.org/jira/browse/FLUME-2935 - Balázs Donát ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49025/#review138910 ----------------------------------------------------------- On June 21, 2016, 3:32 p.m., Balázs Donát Bessenyei wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49025/ > ----------------------------------------------------------- > > (Updated June 21, 2016, 3:32 p.m.) > > > Review request for Flume, Denes Arvay, Mike Percy, and Attila Simon. > > > Repository: flume-git > > > Description > ------- > > SpoolrDir currently monitors a directory and can not handle sub-directories. > This JIRA is to make SpoolDir able to walk down a source directory and > monitor new files. > > > Diffs > ----- > > > flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java > d54f415 > > flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java > 3fe947d > > flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySourceConfigurationConstants.java > 5053697 > > flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java > fe530ff > flume-ng-doc/sphinx/FlumeUserGuide.rst 74d2887 > > Diff: https://reviews.apache.org/r/49025/diff/ > > > Testing > ------- > > Ran tests before the patch: > # mvn clean install -DskipTests -Drat.skip=true; mvn -pl flume-ng-core > -Drat.skip=true test > Tests run: 378, Failures: 0, Errors: 0, Skipped: 2 > > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 07:55 min > [INFO] Finished at: 2016-06-21T16:13:46+02:00 > [INFO] Final Memory: 35M/510M > [INFO] > ------------------------------------------------------------------------ > > After patch: > # mvn clean install -DskipTests -Drat.skip=true; mvn -pl flume-ng-core > -Drat.skip=true test > Tests run: 380, Failures: 0, Errors: 0, Skipped: 2 > > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 06:18 min > [INFO] Finished at: 2016-06-21T17:04:17+02:00 > [INFO] Final Memory: 35M/511M > [INFO] > ------------------------------------------------------------------------ > > Patch also includes docs > > > Thanks, > > Balázs Donát Bessenyei > >
