[
https://issues.apache.org/jira/browse/FLUME-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13885124#comment-13885124
]
chenshangan edited comment on FLUME-2306 at 1/29/14 8:18 AM:
-------------------------------------------------------------
[~hshreedharan]
I think you've introduce another bug. if you cancel idleFuture in
BucketWriter#close, the callback might be canceled prematurely and the
idleTimeout feature will lose effect.
So I suggest add another close() which take a parameter to determine whether
the idleFuture should be canceled and call it in HDFSEventSink#stop
e.g.
agent.sinks.sink_hdfs_s.hdfs.rollInterval = 1200
agent.sinks.sink_hdfs_s.hdfs.idleTimeout = 1800
when rollFuture execute, the file will be closed and thus idleFuture will be
canceled, so idleCallback will never be executed.
was (Author: [email protected]):
[~hshreedharan]
I think you've introduce another bug. if you cancel idleFuture in
BucketWriter#close, the callback might be canceled prematurely and the
idleTimeout feature will lose effect.
So I suggest add another close() which take a parameter to determine whether
the idleFuture should be canceled and call it in HDFSEventSink#stop
> onIdleCallback is not canceled when stop hdfs sink
> --------------------------------------------------
>
> Key: FLUME-2306
> URL: https://issues.apache.org/jira/browse/FLUME-2306
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.4.0
> Reporter: chenshangan
>
> the hdfs sink cached 5000 open files by default and it cost quite a lot of
> memory in total when using lzo CompressedStream. We should open the
> idleTimeout feature to resolve it. But there seems to be a bug with this
> feature. When stopping flume, HDFSWriter does not cancel the idle scheduler,
> which might cause flume not to stop. So I extend the current close() method
> in HDFSWriter as follows, and use it in HDFSEventSink when stop the sink
> component :
> /**
> * when stop flume, all schedulers should be canceled
> * @param cancelIdleCallback
> * @throws IOException
> * @throws InterruptedException
> */
> public void close(boolean cancelIdleCallback) throws IOException,
> InterruptedException{
> close();
> if(cancelIdleCallback){
> if (idleFuture != null && !idleFuture.isDone()) {
> idleFuture.cancel(false); // do not cancel myself if running!
> idleFuture = null;
> }
> }
> }
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
