chenshangan created FLUME-2306:
----------------------------------
Summary: onIdleCallback is not canceled when stop hdfs sink
Key: FLUME-2306
URL: https://issues.apache.org/jira/browse/FLUME-2306
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Affects Versions: v1.4.0
Reporter: chenshangan
Fix For: notrack
the hdfs sink cached 5000 open files by default and it cost quite a lot of
memory in total when using lzo CompressedStream. We should open the idleTimeout
feature to resolve it. But there seems to be a bug with this feature. When
stopping flume, HDFSWriter does not cancel the idle scheduler, which might
cause flume not to stop. So I extend the current close() method in HDFSWriter
as follows, and use it in HDFSEventSink when stop the sink component :
/**
* when stop flume, all schedulers should be canceled
* @param cancelIdleCallback
* @throws IOException
* @throws InterruptedException
*/
public void close(boolean cancelIdleCallback) throws IOException,
InterruptedException{
close();
if(cancelIdleCallback){
if (idleFuture != null && !idleFuture.isDone()) {
idleFuture.cancel(false); // do not cancel myself if running!
idleFuture = null;
}
}
}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
