----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/52598/#review152027 -----------------------------------------------------------
Ship it! Ship It! - Mike Percy On Oct. 10, 2016, 1:53 a.m., Tristan Stevens wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/52598/ > ----------------------------------------------------------- > > (Updated Oct. 10, 2016, 1:53 a.m.) > > > Review request for Flume and Grant Henke. > > > Repository: flume-git > > > Description > ------- > > This feature is useful for anyone who needs greater control of which > partitions are being written to - normally in a situation where multiple > Flume agents are being deployed in order to horizontally scale, or > alternatively if there is a scenario where there is a skew in data that might > lead to one or more partitions hotspotting. > We also have the ability to specify custom partitions on to the Kafka > Producer itself using the kafka.* configuration properties. > > The Kafka Producer provides the ability to set the partition ID using the > following constructor > (https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html#ProducerRecord%28java.lang.String,%20java.lang.Integer,%20K,%20V%29 > ), this is just a matter of providing the option to use this constructor. > > This is specified in one of two ways: either via the staticPartition > configuration property, which means that every message goes to the specified > partition, or via the partitionHeader configuration property, which directs > the implementation to retrieve the partitionId from one of the event headers. > > > Diffs > ----- > > flume-ng-channels/flume-kafka-channel/pom.xml c1cc844 > > flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java > 66b553a > > flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannelConfiguration.java > 3ab807b > > flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java > 57c0b28 > flume-ng-doc/sphinx/FlumeUserGuide.rst ab71d38 > flume-ng-sinks/flume-ng-kafka-sink/pom.xml 195c921 > > flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java > 89bdd84 > > flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkConstants.java > 1bf380c > > flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java > 76eca37 > flume-ng-sources/flume-kafka-source/pom.xml c89ea1a > flume-shared/flume-shared-kafka-test/pom.xml PRE-CREATION > > flume-shared/flume-shared-kafka-test/src/main/java/org/apache/flume/shared/kafka/test/KafkaPartitionTestUtil.java > PRE-CREATION > > flume-shared/flume-shared-kafka-test/src/main/java/org/apache/flume/shared/kafka/test/PartitionOption.java > PRE-CREATION > > flume-shared/flume-shared-kafka-test/src/main/java/org/apache/flume/shared/kafka/test/PartitionTestScenario.java > PRE-CREATION > flume-shared/pom.xml PRE-CREATION > pom.xml 2332a29 > > Diff: https://reviews.apache.org/r/52598/diff/ > > > Testing > ------- > > Unit testing done for both Kafka Channel and Kafka Sink. > > > Thanks, > > Tristan Stevens > >
