[
https://issues.apache.org/jira/browse/FLUME-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13555985#comment-13555985
]
Connor Woodson commented on FLUME-1856:
---------------------------------------
Alright well ignore this...doing some digging I realized I was supposed to
specify the timeout in seconds, not ms, and that the bucket will close itself
(10 hours is a tad too long of a timeout...). But that leads to a following
JIRA to put some specificity into the User guide.
https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L358
> HDFS sink idleTimeout does not close removed BucketWriters
> ----------------------------------------------------------
>
> Key: FLUME-1856
> URL: https://issues.apache.org/jira/browse/FLUME-1856
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.4.0, v1.3.1
> Reporter: Connor Woodson
>
> When a BucketWriter idles out, it is removed from the bucket list but it is
> never closed. Optimal solution will be to override LinkedHashMap.remove
> similar to the removeEldestEntry code to close the bucket.
> The effects of this not happening is that if you rely on idleTimeout for
> closing buckets (such as setting no roll period and attempting to roll via
> timestamped path), as the buckets never get closed the data never goes to s3
> and so the heap gets very full.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
