----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7659/#review12949 -----------------------------------------------------------
Just for posterity, I caught up to mike on IRC, and we hashed out an approach of using Futures and adding an onIdleTimeout hook so that the eventsink can clear out the idle sfWriters which are just eating up memory - Juhani Connolly On Oct. 19, 2012, 6:01 a.m., Juhani Connolly wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/7659/ > ----------------------------------------------------------- > > (Updated Oct. 19, 2012, 6:01 a.m.) > > > Review request for Flume and Mike Percy. > > > Description > ------- > > Added lastWrite to BucketWriter to verify when it was last updated > > Added a thread to HDFSEventSink which verifies the last update of each active > bucketWriter and closes them after the configurable timeout > hdfs.closeIdleTimeout has passed. > > > This addresses bug FLUME-1660. > https://issues.apache.org/jira/browse/FLUME-1660 > > > Diffs > ----- > > flume-ng-doc/sphinx/FlumeUserGuide.rst 29ead84 > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java > bce8e11 > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java > a6d624b > > Diff: https://reviews.apache.org/r/7659/diff/ > > > Testing > ------- > > Local machine testing was performed and the correct closing of files was > confirmed, as well as the correct behavior of the configuration setting > including disabling the feature(by using the default value for > hdfs.closeIdleTimeout of 0) > > > There is one unrelated test failure which I'm not sure of(if anyone knows > what's causing this, please let me know) > > Failed tests: testInOut(org.apache.flume.test.agent.TestFileChannel): > Expected FILE_ROLL sink's dir to have only 1 child, but found 0 children. > expected:<1> but was:<0> > > All other tests pass. > > > Thanks, > > Juhani Connolly > >
