[
https://issues.apache.org/jira/browse/FLUME-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479604#comment-13479604
]
Juhani Connolly commented on FLUME-1660:
----------------------------------------
We use roll interval, however it is not a fix applicable in all situations, and
it is clumsy. For example, if people use size based rolling along with
filenames that are named by header variables, they may or may not be open for
an arbitrary amount of time.
Closing idle files on a timer seems to me a far more elegant solution than
forcing users to guess an arbitrary time after which to roll or to limit open
files(which close in order of oldest, even if the oldest is still active)
I don't really think having the extra option hurts so long as documentation is
good. I'm setting the default setting to inactive
> Close "idle" hdfs handles
> -------------------------
>
> Key: FLUME-1660
> URL: https://issues.apache.org/jira/browse/FLUME-1660
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.4.0
> Reporter: Juhani Connolly
> Assignee: Juhani Connolly
> Priority: Minor
> Fix For: v1.4.0
>
>
> When hdfs paths are date dependent, many handles can get left open for a long
> time without anything happening to them.
> Idea here is to watch the last update of each bucketWriter, and track idle
> writers, closing them once they pass a configured timeout
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
