Juhani Connolly created FLUME-1864:
--------------------------------------
Summary: Allow hdfs idle callback to clean up closed bucket writers
Key: FLUME-1864
URL: https://issues.apache.org/jira/browse/FLUME-1864
Project: Flume
Issue Type: Improvement
Reporter: Juhani Connolly
In the original implementation of the idle file closing behaviour the callback
was cancelled on close(). This makes sense assuming everything else behaves in
a desirable manner.
On the other hand, rollInterval will close a file and leave the bucketWriter in
the writer map. This allows for incrementally named files to be created as the
same path is reopened. However in some situations(primarily with time bucketed
data), this leaves a lot of abandoned bucket writers that may be closed by
rollInterval and thus never removed by idle.
In FLUME-1850 a couple of approaches were suggested, and I originally intended
to fix this by using the callback from rollInterval to remove the writer from
the map. However this would break incremental naming.
Until(if we ever do) we change the rolling logic it leaves the more viable
option to be allowing the idle timer to persist after a close. In this way it
can be used to release resources that have become unnecessary but already
closed by rollInterval, also reducing hard to understand interactions between
configuration variables by one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
