-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6918/
-----------------------------------------------------------
Review request for Flume.
Description
-------
This patch allows users to customise the format of HDFS SequenceFiles by
providing a custom FlumeFormatter implementation.
Currently, the user can set hdfs.writeFormat to either "Text" or "Writable",
corresponding to HDFSTextFormatter and HDFSWritableFormatter respectively.
With this patch, hdfs.writeFormat can also be set to the full class name of a
Builder implementation, e.g.:
agent_foo.sinks.hdfs-sink.writeFormat=com.mycompany.flume.MyCustomFormatter$Builder
They can also pass custom configuration params to the builder, e.g.:
agent_foo.sinks.hdfs-sink.writeFormat.ignoreHeaders=foo,bar
These params will be passed to the Builder's build() method as a Context object.
I've tried to be as consistent as possible with the design of
EventSerializerFactory:
* Use an enum for the different formatter types, rather than static strings.
* Use a Builder, rather than constructing a FlumeFormatter directly.
This addresses bug FLUME-1100.
https://issues.apache.org/jira/browse/FLUME-1100
Diffs
-----
flume-ng-core/src/main/java/org/apache/flume/sink/FlumeFormatter.java 05d60b7
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
9a76ecb
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java
7b47942
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterType.java
PRE-CREATION
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSTextFormatter.java
5839dbb
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSWritableFormatter.java
9f03389
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/MyCustomFormatter.java
PRE-CREATION
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSFormatterFactory.java
PRE-CREATION
Diff: https://reviews.apache.org/r/6918/diff/
Testing
-------
Unit tests included in patch.
Using a patched build of Flume in an internal project (not in production).
Thanks,
Chris Birchall
