> On March 1, 2013, 4:47 a.m., Hari Shreedharan wrote: > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java, > > line 430 > > <https://reviews.apache.org/r/9675/diff/2/?file=263764#file263764line430> > > > > will not continue rolling files in this directory?
Fixed > On March 1, 2013, 4:47 a.m., Hari Shreedharan wrote: > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java, > > line 440 > > <https://reviews.apache.org/r/9675/diff/2/?file=263764#file263764line440> > > > > Sorry, I think I missed this one earlier. At this point doRotate is > > always going to be true. Was this logic meant to check if shouldRotate() > > returned true because of under replication? If it did, don't rotate if it > > is under replicated? If it is, then you need to set doRotate to false in > > the if block above. Wow, good catch! Fixed. > On March 1, 2013, 4:47 a.m., Hari Shreedharan wrote: > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java, > > line 484 > > <https://reviews.apache.org/r/9675/diff/2/?file=263764#file263764line484> > > > > One more thing I missed here. If this a previous file owned by this > > bucket writer was under replicated, then isUnderReplicated is never reset > > to false, causing every following file to be rotated prematurely. It should > > either be done here or in the if(shouldRotate()) {..} block. Added to open() call - Mike ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9675/#review17224 ----------------------------------------------------------- On March 1, 2013, 5:44 a.m., Mike Percy wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/9675/ > ----------------------------------------------------------- > > (Updated March 1, 2013, 5:44 a.m.) > > > Review request for Flume. > > > Description > ------- > > This is based on https://issues.apache.org/jira/browse/HBASE-2234 ... > essentially if datanodes are dying en masse then we want to close our files > as we are writing so that we get new pipeline assignments from the namenode. > > > This addresses bug FLUME-1916. > https://issues.apache.org/jira/browse/FLUME-1916 > > > Diffs > ----- > > flume-ng-doc/sphinx/FlumeUserGuide.rst 5ac903e > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java > PRE-CREATION > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java > cdc37f6 > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java > 1f3521e > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java > 4ea4327 > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java > 3bd25f4 > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java > abca21f > > flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java > 5e8628b > > flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java > 99b6150 > > flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java > 2e71069 > > Diff: https://reviews.apache.org/r/9675/diff/ > > > Testing > ------- > > Added unit test. > > > Thanks, > > Mike Percy > >
