Brock Noland created FLUME-1610:
-----------------------------------
Summary: HDDSEventSink and bucket writer have a race condition
Key: FLUME-1610
URL: https://issues.apache.org/jira/browse/FLUME-1610
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Reporter: Brock Noland
Attachments: FLUME-1610-0.patch
I have seen a scenerio where an exception was thrown during
HDFSEventSink.process when the flush on the bucket writer was called but the
BucketWriter was already closed.
Assumptions:
1) In HDFSEventSink.process when done, we flush all buckets written to once
channel returns null or batch size is exceeded
2) The BucketWriter.flush method does not check the isOpen flag.
3) Our time roll interval code assumes the next call to the bucket writer will
be append as such the isOpen flag will be checked and the underlying writer
re-opened.
As such, I think what is happening is this:
1) In HDFSEventSink.process the bucket writer is written to
2) In BucketWriter the time based roll trips
3) In HDFSEventSink.process the channel returns null or batch size is exceeded
4) In HDFSEventSink.process bucket writer flush is called throwing the
exception logged above.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
