[
https://issues.apache.org/jira/browse/FLUME-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Juhani Connolly updated FLUME-1864:
-----------------------------------
Attachment: FLUME-1864
> Allow hdfs idle callback to clean up closed bucket writers
> ----------------------------------------------------------
>
> Key: FLUME-1864
> URL: https://issues.apache.org/jira/browse/FLUME-1864
> Project: Flume
> Issue Type: Improvement
> Reporter: Juhani Connolly
> Attachments: FLUME-1864
>
>
> In the original implementation of the idle file closing behaviour the
> callback was cancelled on close(). This makes sense assuming everything else
> behaves in a desirable manner.
> On the other hand, rollInterval will close a file and leave the bucketWriter
> in the writer map. This allows for incrementally named files to be created as
> the same path is reopened. However in some situations(primarily with time
> bucketed data), this leaves a lot of abandoned bucket writers that may be
> closed by rollInterval and thus never removed by idle.
> In FLUME-1850 a couple of approaches were suggested, and I originally
> intended to fix this by using the callback from rollInterval to remove the
> writer from the map. However this would break incremental naming.
> Until(if we ever do) we change the rolling logic it leaves the more viable
> option to be allowing the idle timer to persist after a close. In this way it
> can be used to release resources that have become unnecessary but already
> closed by rollInterval, also reducing hard to understand interactions between
> configuration variables by one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
