[
https://issues.apache.org/jira/browse/FLUME-1238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Percy updated FLUME-1238:
------------------------------
Attachment: FLUME-1238-1.patch
Patch is attached.
Description of changes:
- Now actively closes HDFS files when the rollInterval occurs
- Adds a configurable-size scheduled thread pool per HDFS sink for scheduling
the file rolling
- Moves knowledge of the principal and doAs() down one level into the
BucketWriter
- Required a little bit of refactoring of the public methods in
BucketWriter to support this in a multi-threaded fashion
- Updated unit test to verify that files are actively rolled
- Updated user guide with docs for the configuration variable
As well as:
- Minor refactoring in BucketWriter to make more variables final and thus
codify/enforce more invariants
- Minor refactoring in HDFSEventSink to remove now-unused ProxyCallable
- Minor refactoring in HDFSEventSink to remove unneeded parameters and methods
related to call timeouts
- Also added convenience functions to HDFSEventSink for time-limited append,
flush, and close to make the code more readable
> Support active rolling of files created by HDFS Event Sink
> ----------------------------------------------------------
>
> Key: FLUME-1238
> URL: https://issues.apache.org/jira/browse/FLUME-1238
> Project: Flume
> Issue Type: Improvement
> Reporter: Arvind Prabhakar
> Assignee: Mike Percy
> Attachments: FLUME-1238-1.patch
>
>
> The HDFS Event Sink uses lazy rolling for closing files that are being
> written to. This results in many files being open for longer than their
> expected roll-interval if they are not actively written to and can even last
> in open state until the sink shuts down.
> It will be preferable to have these files roll proactively rather than
> waiting for a write to trigger the roll.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
