----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/13640/#review26240 -----------------------------------------------------------
Thanks for the patch Ted. But I don't think this really works since the HDFSCompressedDataStream class which is used for compressed writes does not seem to have support for getLength(). flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java <https://reviews.apache.org/r/13640/#comment51272> This should really be rollCompressedSize <= currentLength + lengthOfEventSerialized (this can be approximated by using the size of the body - this will cause some variation around the boundary, but the upper bound for that variation would be the total compressed size of that one event - which would not be really massive, and we should just document that this rollCompressedSize is approximate). - Hari Shreedharan On Aug. 19, 2013, 1:20 a.m., Ted Malaska wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/13640/ > ----------------------------------------------------------- > > (Updated Aug. 19, 2013, 1:20 a.m.) > > > Review request for Flume. > > > Bugs: 2128 > https://issues.apache.org/jira/browse/2128 > > > Repository: flume-git > > > Description > ------- > > 1) Added support for hdfs.rollCompressedSize > 2) Added unit test > 3) Updated doc > > > Diffs > ----- > > flume-ng-doc/sphinx/FlumeUserGuide.rst c614991 > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/AbstractHDFSWriter.java > da0466d > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java > 65f4d2c > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java > f0a6e4b > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSSequenceFile.java > 5fe9f1b > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWriter.java > 44a984a > > flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MockHDFSWriter.java > ec49b97 > > flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestBucketWriter.java > f741e03 > > Diff: https://reviews.apache.org/r/13640/diff/ > > > Testing > ------- > > > Thanks, > > Ted Malaska > >
