----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9675/#review17224 -----------------------------------------------------------
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java <https://reviews.apache.org/r/9675/#comment36594> will not continue rolling files in this directory? flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java <https://reviews.apache.org/r/9675/#comment36592> Sorry, I think I missed this one earlier. At this point doRotate is always going to be true. Was this logic meant to check if shouldRotate() returned true because of under replication? If it did, don't rotate if it is under replicated? If it is, then you need to set doRotate to false in the if block above. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java <https://reviews.apache.org/r/9675/#comment36593> One more thing I missed here. If this a previous file owned by this bucket writer was under replicated, then isUnderReplicated is never reset to false, causing every following file to be rotated prematurely. It should either be done here or in the if(shouldRotate()) {..} block. - Hari Shreedharan On March 1, 2013, 1:53 a.m., Mike Percy wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/9675/ > ----------------------------------------------------------- > > (Updated March 1, 2013, 1:53 a.m.) > > > Review request for Flume. > > > Description > ------- > > This is based on https://issues.apache.org/jira/browse/HBASE-2234 ... > essentially if datanodes are dying en masse then we want to close our files > as we are writing so that we get new pipeline assignments from the namenode. > > > This addresses bug FLUME-1916. > https://issues.apache.org/jira/browse/FLUME-1916 > > > Diffs > ----- > > flume-ng-doc/sphinx/FlumeUserGuide.rst 5ac903e > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java > PRE-CREATION > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java > cdc37f6 > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java > 1f3521e > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java > 4ea4327 > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java > 3bd25f4 > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java > abca21f > > flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java > 5e8628b > > flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java > 99b6150 > > flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java > 2e71069 > > Diff: https://reviews.apache.org/r/9675/diff/ > > > Testing > ------- > > Added unit test. > > > Thanks, > > Mike Percy > >
