----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52598/ -----------------------------------------------------------
Review request for Flume. Repository: flume-git Description ------- This feature is useful for anyone who needs greater control of which partitions are being written to - normally in a situation where multiple Flume agents are being deployed in order to horizontally scale, or alternatively if there is a scenario where there is a skew in data that might lead to one or more partitions hotspotting. We also have the ability to specify custom partitions on to the Kafka Producer itself using the kafka.* configuration properties. The Kafka Producer provides the ability to set the partition ID using the following constructor (https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html#ProducerRecord%28java.lang.String,%20java.lang.Integer,%20K,%20V%29 ), this is just a matter of providing the option to use this constructor. This is specified in one of two ways: either via the staticPartition configuration property, which means that every message goes to the specified partition, or via the partitionHeader configuration property, which directs the implementation to retrieve the partitionId from one of the event headers. Diffs ----- flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java 66b553a flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannelConfiguration.java 3ab807b flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java 57c0b28 flume-ng-doc/sphinx/FlumeUserGuide.rst ab71d38 flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java 89bdd84 flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkConstants.java 1bf380c flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java 76eca37 flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/util/KafkaConsumer.java d5dfbd6 Diff: https://reviews.apache.org/r/52598/diff/ Testing ------- Unit testing done for both Kafka Channel and Kafka Sink. Thanks, Tristan Stevens
