[
https://issues.apache.org/jira/browse/FLUME-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13886693#comment-13886693
]
chenshangan commented on FLUME-2306:
------------------------------------
[~hshreedharan]
It makes sense. My philosophy is that we don't need to create a new
BucketWriter every time when a file is closed as we may reuse the bucketwriter
and only roll files in the bucket. So will it be better that idleTimeout deals
with bucket but not the file in the bucket ?
> onIdleCallback is not canceled when stop hdfs sink
> --------------------------------------------------
>
> Key: FLUME-2306
> URL: https://issues.apache.org/jira/browse/FLUME-2306
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.4.0
> Reporter: chenshangan
>
> the hdfs sink cached 5000 open files by default and it cost quite a lot of
> memory in total when using lzo CompressedStream. We should open the
> idleTimeout feature to resolve it. But there seems to be a bug with this
> feature. When stopping flume, HDFSWriter does not cancel the idle scheduler,
> which might cause flume not to stop. So I extend the current close() method
> in HDFSWriter as follows, and use it in HDFSEventSink when stop the sink
> component :
> /**
> * when stop flume, all schedulers should be canceled
> * @param cancelIdleCallback
> * @throws IOException
> * @throws InterruptedException
> */
> public void close(boolean cancelIdleCallback) throws IOException,
> InterruptedException{
> close();
> if(cancelIdleCallback){
> if (idleFuture != null && !idleFuture.isDone()) {
> idleFuture.cancel(false); // do not cancel myself if running!
> idleFuture = null;
> }
> }
> }
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
