> On March 1, 2013, 5:58 a.m., Hari Shreedharan wrote: > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java, > > lines 429-446 > > <https://reviews.apache.org/r/9675/diff/3/?file=263788#file263788line429> > > > > I get that since consecutiveUnderReplCount is reset the counter gets > > reset on open, but you need to also set isUnderReplicated to false, > > otherwise once that is set to true, it is never reset to false and the > > consecutiveUnderReplCount can get incremented for every append following > > one rotation due to under replication.
Ugh, you are right. I tried to add a unit test for bringing back the DN but I couldn't get the NN to recognize it, so here is a fix without a unit test. I will try to add a working unit test for this condition in a follow up patch. - Mike ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9675/#review17229 ----------------------------------------------------------- On March 1, 2013, 6:38 a.m., Mike Percy wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/9675/ > ----------------------------------------------------------- > > (Updated March 1, 2013, 6:38 a.m.) > > > Review request for Flume. > > > Description > ------- > > This is based on https://issues.apache.org/jira/browse/HBASE-2234 ... > essentially if datanodes are dying en masse then we want to close our files > as we are writing so that we get new pipeline assignments from the namenode. > > > This addresses bug FLUME-1916. > https://issues.apache.org/jira/browse/FLUME-1916 > > > Diffs > ----- > > flume-ng-doc/sphinx/FlumeUserGuide.rst 5ac903e > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java > PRE-CREATION > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java > cdc37f6 > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSCompressedDataStream.java > 1f3521e > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSDataStream.java > 4ea4327 > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java > 3bd25f4 > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java > abca21f > > flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java > 5e8628b > > flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java > 99b6150 > > flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java > 2e71069 > > Diff: https://reviews.apache.org/r/9675/diff/ > > > Testing > ------- > > Added unit test. > > > Thanks, > > Mike Percy > >
