[ https://issues.apache.org/jira/browse/FLUME-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001534#comment-16001534 ]
ASF GitHub Bot commented on FLUME-3085: --------------------------------------- Github user asfgit closed the pull request at: https://github.com/apache/flume/pull/129 > HDFS Sink can skip flushing some BucketWriters, might lead to data loss > ----------------------------------------------------------------------- > > Key: FLUME-3085 > URL: https://issues.apache.org/jira/browse/FLUME-3085 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources > Affects Versions: 1.7.0 > Reporter: Denes Arvay > Assignee: Denes Arvay > Priority: Critical > Fix For: 1.8.0 > > > The {{HDFSEventSink.process()}} is already prepared for a rare race > condition, namely when the BucketWriter acquired in [line > 389|https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L389] > gets closed by an other thread (e.g. because the {{idleTimeout}} or the > {{rollInterval}}) before the {{append()}} is called in [line > 406|https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L406]. > If this is the case the {{BucketWriter.append()}} call throws a > {{BucketClosedException}} and the sink creates a new {{BucketWriter}} > instance and appends to it. > But this newly created instance won't be added to the {{writers}} list, which > means that it won't be flushed after the processing loop finished: > https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L429 > This has multiple consequences: > - unflushed data might get lost > - the {{BucketWriter}}'s {{idleAction}} won't be scheduled > (https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L450), > which means that it won't be closed nor renamed if the idle timeout is the > only trigger for closing the file. -- This message was sent by Atlassian JIRA (v6.3.15#6346)