ashok Kumar created FLUME-2419:
----------------------------------
Summary: Bucket writer was closed due to idling exception
Key: FLUME-2419
URL: https://issues.apache.org/jira/browse/FLUME-2419
Project: Flume
Issue Type: Bug
Affects Versions: v1.4.0
Reporter: ashok Kumar
Priority: Minor
We have flume collecting the daily log and sinks into hdfs with the
configuration,
activity.sinks.eventhdfssink_1.hdfs.rollSize = 66060288
activity.sinks.eventhdfssink_1.hdfs.rollCount = 0
activity.sinks.eventhdfssink_1.hdfs.rollInterval = 0
activity.sinks.eventhdfssink_1.hdfs.idleTimeout = 180
We have the default configuration for maxopenfile and the writing data went
into endless loop with the following exception:
"java.io.IOException: This bucket writer was closed due to idling and this
handle is thus no longer valid"
I looked over the source code (BucketWriter.java) and I establish the
exceptions occurs only at the time the file is closed and also due to idle time
out.
if (!isOpen) {
if(idleClosed) {
throw new IOException("This bucket writer was closed due to idling and
this handle " +
"is thus no longer valid");
}
open();
}
However the idlecallback has been called out during the flush() from
BucketWriter.java,
if(onIdleCallback != null)
onIdleCallback.run(onIdleCallbackPath);
the idlecallback from HDFSEventSink.java,
WriterCallback idleCallback = null;
if(idleTimeout != 0) {
idleCallback = new WriterCallback() {
@Override
public void run(String bucketPath) {
sfWriters.remove(bucketPath);
}
};
Here, we remove the entry from sfWriters map, thus forcing a recreation of the
writer during subsequent calls. Thus, I do not see a case where we can have the
file handle closed as well as idleClosed be true without removing the necessary
references. FWIW, we are seeing logs that indicate that onIdleCallback has been
triggered. Are there any other cases in which this exception can hit?
If it is of help, we saw this exception when our application (wrongly) started
pumping old events (older than 2 weeks). Is there a possibility of writer
handles created earlier being closed, but not being visible to the
BucketWriter.java code?
Any pointers to potential configuration issues is also appreciated.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
