[
https://issues.apache.org/jira/browse/FLUME-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13589001#comment-13589001
]
Hudson commented on FLUME-1864:
-------------------------------
Integrated in flume-trunk #362 (See
[https://builds.apache.org/job/flume-trunk/362/])
FLUME-1864. Allow hdfs idle callback to clean up closed bucket writers.
(Revision ce05be585f1cb91a3a00ccdfdeaaf5800ada6c35)
Result = SUCCESS
hshreedharan :
http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ce05be585f1cb91a3a00ccdfdeaaf5800ada6c35
Files :
*
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java
> Allow hdfs idle callback to clean up closed bucket writers
> ----------------------------------------------------------
>
> Key: FLUME-1864
> URL: https://issues.apache.org/jira/browse/FLUME-1864
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.4.0
> Reporter: Juhani Connolly
> Assignee: Juhani Connolly
> Fix For: v1.4.0
>
> Attachments: FLUME-1864
>
>
> In the original implementation of the idle file closing behaviour the
> callback was cancelled on close(). This makes sense assuming everything else
> behaves in a desirable manner.
> On the other hand, rollInterval will close a file and leave the bucketWriter
> in the writer map. This allows for incrementally named files to be created as
> the same path is reopened. However in some situations(primarily with time
> bucketed data), this leaves a lot of abandoned bucket writers that may be
> closed by rollInterval and thus never removed by idle.
> In FLUME-1850 a couple of approaches were suggested, and I originally
> intended to fix this by using the callback from rollInterval to remove the
> writer from the map. However this would break incremental naming.
> Until(if we ever do) we change the rolling logic it leaves the more viable
> option to be allowing the idle timer to persist after a close. In this way it
> can be used to release resources that have become unnecessary but already
> closed by rollInterval, also reducing hard to understand interactions between
> configuration variables by one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
