----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5378/#review8416 -----------------------------------------------------------
fileType = DataStream + codec = gzip (for example) is an invalid config. It's better to fail fast and loudly than allow the user to run with an invalid config which could cause confusion and be easy to miss in the log, especially if they have a large deployment with many sources and sinks. Please throw an IllegalArgumentException instead. - Mike Percy On June 19, 2012, 2:11 p.m., Leslin (Hong Xiang Lin) wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/5378/ > ----------------------------------------------------------- > > (Updated June 19, 2012, 2:11 p.m.) > > > Review request for Flume. > > > Description > ------- > > 1). if user set hdfs.codeC when hdfs.fileType = DataStream, sink will use > DataStream to output file, which is no compress extension like .snappy. > Warning message is added to show the codeC will be ignored. > 2). Pre-check will make sure that codec is required when fileType is set > CompressedStream. > > > This addresses bug FLUME-1200. > https://issues.apache.org/jira/browse/FLUME-1200 > > > Diffs > ----- > > flume-ng-doc/sphinx/FlumeUserGuide.rst e61443b > > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java > fc06754 > > Diff: https://reviews.apache.org/r/5378/diff/ > > > Testing > ------- > > 1. compressStream without codec --> there will be exception > agent.sinks.k1.hdfs.fileType = CompressedStream > #agent.sinks.k1.hdfs.codeC = DefaultCodec > 12/06/17 22:46:35 INFO sink.DefaultSinkFactory: Creating instance of sink k1 > typeHDFS > 12/06/17 22:46:35 ERROR properties.PropertiesFileConfigurationProvider: > Failed to load configuration data. Exception follows. > java.lang.NullPointerException: It's essential to set compress codec when > fileType is: CompressedStream > at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) > at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:221) > 2. Works fine, output file with .deflate extension. > agent.sinks.k1.hdfs.fileType = CompressedStream > agent.sinks.k1.hdfs.codeC = DefaultCodec > 3. Works fine, output file without compress extension. > agent.sinks.k1.hdfs.fileType = CompressedStream > agent.sinks.k1.hdfs.codeC = snappyCodec > 4. There is warning, output file without compress extension. > agent.sinks.k1.hdfs.fileType = DataStream > #agent.sinks.k1.hdfs.codeC = snappyCodec > 12/06/17 23:08:44 INFO snappy.LoadSnappy: Snappy native library loaded > 12/06/17 23:08:44 WARN hdfs.HDFSEventSink: CodeC: snappyCodec is ignored as > fileType: DataStream is uncompressed. To change fileType if want output > compressed. > 12/06/17 23:08:44 INFO hdfs.HDFSEventSink: Hadoop Security enabled: false > 5. works fine, output file with .snappy extension > agent.sinks.k1.hdfs.fileType = SequenceFile > agent.sinks.k1.hdfs.codeC = snappyCodec > 6. Works fine, output file without .snappy extension > agent.sinks.k1.hdfs.fileType = SequenceFile > #agent.sinks.k1.hdfs.codeC = snappyCodec > > > Thanks, > > Leslin (Hong Xiang Lin) > >
