----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52598/ -----------------------------------------------------------
(Updated Oct. 10, 2016, 8:49 a.m.) Review request for Flume and Grant Henke. Changes ------- Fixed compile problems (thanks Mike) and also checkstyle. Added some more Javadocs for test utils. Repository: flume-git Description ------- This feature is useful for anyone who needs greater control of which partitions are being written to - normally in a situation where multiple Flume agents are being deployed in order to horizontally scale, or alternatively if there is a scenario where there is a skew in data that might lead to one or more partitions hotspotting. We also have the ability to specify custom partitions on to the Kafka Producer itself using the kafka.* configuration properties. The Kafka Producer provides the ability to set the partition ID using the following constructor (https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html#ProducerRecord%28java.lang.String,%20java.lang.Integer,%20K,%20V%29 ), this is just a matter of providing the option to use this constructor. This is specified in one of two ways: either via the staticPartition configuration property, which means that every message goes to the specified partition, or via the partitionHeader configuration property, which directs the implementation to retrieve the partitionId from one of the event headers. Diffs (updated) ----- flume-ng-channels/flume-kafka-channel/pom.xml c1cc844 flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java 66b553a flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannelConfiguration.java 3ab807b flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java 57c0b28 flume-ng-doc/sphinx/FlumeUserGuide.rst ab71d38 flume-ng-sinks/flume-ng-kafka-sink/pom.xml 195c921 flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java 89bdd84 flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkConstants.java 1bf380c flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java 76eca37 flume-ng-sources/flume-kafka-source/pom.xml c89ea1a flume-shared/flume-shared-kafka-test/pom.xml PRE-CREATION flume-shared/flume-shared-kafka-test/src/main/java/org/apache/flume/shared/kafka/test/KafkaPartitionTestUtil.java PRE-CREATION flume-shared/flume-shared-kafka-test/src/main/java/org/apache/flume/shared/kafka/test/PartitionOption.java PRE-CREATION flume-shared/flume-shared-kafka-test/src/main/java/org/apache/flume/shared/kafka/test/PartitionTestScenario.java PRE-CREATION flume-shared/pom.xml PRE-CREATION pom.xml 2332a29 Diff: https://reviews.apache.org/r/52598/diff/ Testing ------- Unit testing done for both Kafka Channel and Kafka Sink. Thanks, Tristan Stevens
