-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7659/
-----------------------------------------------------------
(Updated Oct. 31, 2012, 10:49 a.m.)
Review request for Flume and Mike Percy.
Changes
-------
Implemented the described suggestions:
- Added a callback interface to HDFSEventSink
- uses a scheduled future to close stuff once the bucket writer has been idle
too long.
- onIdleTimeout callback will now remove the bucketWriter from sfWriters,
cleaning up that leak(though this is only the case of idleTimeout is non-0)
- Added hdfs.idleTimeout configuration setting to hdfs sink, default is 0.
Perhaps we should consider enabling it by default with a reasonably long
default timeout?
Tested on local hdfs, confirmed that the config option is working and that
files are indeed getting closed. I also verified in the debugger that the
sfWriter entries were actually getting deleted.
Description
-------
Added lastWrite to BucketWriter to verify when it was last updated
Added a thread to HDFSEventSink which verifies the last update of each active
bucketWriter and closes them after the configurable timeout
hdfs.closeIdleTimeout has passed.
This addresses bug FLUME-1660.
https://issues.apache.org/jira/browse/FLUME-1660
Diffs (updated)
-----
flume-ng-doc/sphinx/FlumeUserGuide.rst c1303e0
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
9f2c763
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
e369604
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java
6a8072e
Diff: https://reviews.apache.org/r/7659/diff/
Testing
-------
Local machine testing was performed and the correct closing of files was
confirmed, as well as the correct behavior of the configuration setting
including disabling the feature(by using the default value for
hdfs.closeIdleTimeout of 0)
There is one unrelated test failure which I'm not sure of(if anyone knows
what's causing this, please let me know)
Failed tests: testInOut(org.apache.flume.test.agent.TestFileChannel):
Expected FILE_ROLL sink's dir to have only 1 child, but found 0 children.
expected:<1> but was:<0>
All other tests pass.
Thanks,
Juhani Connolly
