update storm-kafka README.md Update storm-kafka README.md with info about retry settings. Also update the sentence on what SpoutConfig provides over KafkaConfig.
Project: http://git-wip-us.apache.org/repos/asf/storm/repo Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/77c2df28 Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/77c2df28 Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/77c2df28 Branch: refs/heads/master Commit: 77c2df28517f9b9acd87724116d9b7b3573be4fc Parents: 8c78c13 Author: Rick Kilgore <[email protected]> Authored: Thu Sep 11 17:29:05 2014 -0700 Committer: Rick Kilgore <[email protected]> Committed: Thu Sep 11 17:29:05 2014 -0700 ---------------------------------------------------------------------- external/storm-kafka/README.md | 22 +++++++++++++++----- .../src/jvm/storm/kafka/SpoutConfig.java | 9 ++++++-- 2 files changed, 24 insertions(+), 7 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/storm/blob/77c2df28/external/storm-kafka/README.md ---------------------------------------------------------------------- diff --git a/external/storm-kafka/README.md b/external/storm-kafka/README.md index 73cf334..283a950 100644 --- a/external/storm-kafka/README.md +++ b/external/storm-kafka/README.md @@ -51,17 +51,29 @@ The optional ClientId is used as a part of the zookeeper path where the spout's There are 2 extensions of KafkaConfig currently in use. -Spoutconfig is an extension of KafkaConfig that supports 2 additional fields, zkroot and id. The Zkroot will be used -as root to store your consumer's offset. The id should uniquely identify your spout. +Spoutconfig is an extension of KafkaConfig that supports additional fields with ZooKeeper connection info and for controlling +behavior specific to KafkaSpout. The Zkroot will be used as root to store your consumer's offset. The id should uniquely +identify your spout. ```java public SpoutConfig(BrokerHosts hosts, String topic, String zkRoot, String id); +public SpoutConfig(BrokerHosts hosts, String topic, String id); +``` +In addition to these parameters, SpoutConfig contains the following fields that control how KafkaSpout behaves: +```java + // setting for how often to save the current kafka offset to ZooKeeper + public long stateUpdateIntervalMs = 2000; + + // Exponential back-off retry settings. These are used when retrying messages after a bolt + // calls OutputCollector.fail(). + // Note: be sure to set backtype.storm.Config.MESSAGE_TIMEOUT_SECS appropriately to prevent + // resubmitting the message while still retrying. + public long retryInitialDelayMs = 0; + public double retryDelayMultiplier = 1.0; + public long retryDelayMaxMs = 60 * 1000; ``` Core KafkaSpout only accepts an instance of SpoutConfig. TridentKafkaConfig is another extension of KafkaConfig. -```java -public SpoutConfig(BrokerHosts hosts, String topic, String id); -``` TridentKafkaEmitter only accepts TridentKafkaConfig. The KafkaConfig class also has bunch of public variables that controls your application's behavior. Here are defaults: http://git-wip-us.apache.org/repos/asf/storm/blob/77c2df28/external/storm-kafka/src/jvm/storm/kafka/SpoutConfig.java ---------------------------------------------------------------------- diff --git a/external/storm-kafka/src/jvm/storm/kafka/SpoutConfig.java b/external/storm-kafka/src/jvm/storm/kafka/SpoutConfig.java index fbc887f..1c40ddf 100644 --- a/external/storm-kafka/src/jvm/storm/kafka/SpoutConfig.java +++ b/external/storm-kafka/src/jvm/storm/kafka/SpoutConfig.java @@ -26,10 +26,15 @@ public class SpoutConfig extends KafkaConfig implements Serializable { public Integer zkPort = null; public String zkRoot = null; public String id = null; + + // setting for how often to save the current kafka offset to ZooKeeper public long stateUpdateIntervalMs = 2000; - // Exponential back-off retry settings - note: be sure to set backtype.storm.Config.MESSAGE_TIMEOUT_SECS - // appropriately to prevent resubmitting the message while still retrying + // Exponential back-off retry settings. These are used when retrying messages after a bolt + // calls OutputCollector.fail(). + // + // Note: be sure to set backtype.storm.Config.MESSAGE_TIMEOUT_SECS appropriately to prevent + // resubmitting the message while still retrying. public long retryInitialDelayMs = 0; public double retryDelayMultiplier = 1.0; public long retryDelayMaxMs = 60 * 1000;
