Github user srdo commented on a diff in the pull request:
https://github.com/apache/storm/pull/1808#discussion_r91347833
--- Diff: docs/storm-kafka-client.md ---
@@ -1,90 +1,222 @@
-#Storm Kafka Spout with New Kafka Consumer API
+#Storm Kafka integration using the kafka-client jar
-Apache Storm Spout implementation to consume data from Apache Kafka
versions 0.10 onwards (please see [Apache Kafka Version Compatibility]
(#compatibility)).
+##Compatability
-The Apache Storm Spout allows clients to consume data from Kafka starting
at offsets as defined by the offset strategy specified in
`FirstPollOffsetStrategy`.
-In case of failure, the Kafka Spout will re-start consuming messages from
the offset that matches the chosen `FirstPollOffsetStrategy`.
+Apache Kafka versions 0.10 onwards (please see [Apache Kafka Version
Compatibility] (#compatibility)).
-The Kafka Spout implementation allows you to specify the stream
(`KafkaSpoutStream`) associated with each topic or topic wildcard.
`KafkaSpoutStream` represents the stream and output fields. For named topics
use `KafkaSpoutStreamsNamedTopics`, and for topic wildcards use
`KafkaSpoutStreamsWildcardTopics`.
+##Writing to Kafka as part of your topology
+You can create an instance of org.apache.storm.kafka.bolt.KafkaBolt and
attach it as a component to your topology or if you
+are using trident you can use org.apache.storm.kafka.trident.TridentState,
org.apache.storm.kafka.trident.TridentStateFactory and
+org.apache.storm.kafka.trident.TridentKafkaUpdater.
-The `KafkaSpoutTuplesBuilder` wraps all the logic that builds `Tuple`s
from `ConsumerRecord`s. The logic is provided by the user through implementing
the appropriate number of `KafkaSpoutTupleBuilder` instances. For named topics
use `KafkaSpoutTuplesBuilderNamedTopics`, and for topic wildcard use
`KafkaSpoutTuplesBuilderWildcardTopics`.
+You need to provide implementations for the following 2 interfaces
-Multiple topics and topic wildcards can use the same
`KafkaSpoutTupleBuilder` implementation, as long as the logic to build `Tuple`s
from `ConsumerRecord`s is identical.
+###TupleToKafkaMapper and TridentTupleToKafkaMapper
+These interfaces have 2 methods defined:
+```java
+ K getKeyFromTuple(Tuple/TridentTuple tuple);
+ V getMessageFromTuple(Tuple/TridentTuple tuple);
+```
+
+As the name suggests, these methods are called to map a tuple to a Kafka
key and a Kafka message. If you just want one field
+as key and one field as value, then you can use the provided
FieldNameBasedTupleToKafkaMapper.java
+implementation. In the KafkaBolt, the implementation always looks for a
field with field name "key" and "message" if you
+use the default constructor to construct FieldNameBasedTupleToKafkaMapper
for backward compatibility
+reasons. Alternatively you could also specify a different key and message
field by using the non default constructor.
+In the TridentKafkaState you must specify what is the field name for key
and message as there is no default constructor.
+These should be specified while constructing and instance of
FieldNameBasedTupleToKafkaMapper.
+
+###KafkaTopicSelector and trident KafkaTopicSelector
+This interface has only one method
+```java
+public interface KafkaTopicSelector {
+ String getTopics(Tuple/TridentTuple tuple);
+}
+```
+The implementation of this interface should return the topic to which the
tuple's key/message mapping needs to be published
+You can return a null and the message will be ignored. If you have one
static topic name then you can use
+DefaultTopicSelector.java and set the name of the topic in the constructor.
+`FieldNameTopicSelector` and `FieldIndexTopicSelector` use to support
decided which topic should to push message from tuple.
--- End diff --
Phrasing is a little awkward here. Maybe something like "... can be used to
decide which topic a message should go to based on a field in the tuple"
instead?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
