[
https://issues.apache.org/jira/browse/STORM-1015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943885#comment-14943885
]
ASF GitHub Bot commented on STORM-1015:
---------------------------------------
Github user choang commented on a diff in the pull request:
https://github.com/apache/storm/pull/705#discussion_r41186043
--- Diff: external/storm-kafka/src/jvm/storm/kafka/StateStore.java ---
@@ -0,0 +1,43 @@
+package storm.kafka;
+
+import java.util.Map;
+
+/**
+ * Abstraction of a partition state storage.
+ *
+ * The partition state usually is kept in Json format in the store and in
Map format in runtime memory. An example
+ * is shown below:
+ *
+ * Json:
+ * {
+ * "broker": {
+ * "host": "kafka.sample.net",
+ * "port": 9092
+ * },
+ * "offset": 4285,
+ * "partition": 1,
+ * "topic": "testTopic",
+ * "topology": {
+ * "id": "fce905ff-25e0 -409e-bc3a-d855f 787d13b",
+ * "name": "Test Topology"
+ * }
+ * }
+ *
+ * Memory:
+ * Map<Object, Object> data = (Map<Object, Object>) ImmutableMap.builder()
+ * .put("topology", ImmutableMap.of(
+ * "id", "fce905ff-25e0 -409e-bc3a-d855f 787d13b",
+ * "name", "Test Topology"))
+ * .put("offset", 4285)
+ * .put("partition", 1)
+ * .put("broker", ImmutableMap.of(
+ * "host", "kafka.sample.net",
+ * "port", 9092))
+ * .put("topic", "testTopic").build();
+ */
+public interface StateStore {
--- End diff --
extends <tt>Closeable</tt>
> Store Kafka offsets with Kafka's consumer offset management api
> ---------------------------------------------------------------
>
> Key: STORM-1015
> URL: https://issues.apache.org/jira/browse/STORM-1015
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-kafka
> Affects Versions: 0.11.0
> Reporter: Hang Sun
> Priority: Minor
> Labels: consumer, kafka, offset
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> Current Kafka spout stores the offsets (and some other states) inside ZK with
> its proprietary format. This does not work well with other Kafka offset
> monitoring tools such as Burrow, KafkaOffsetMonitor etc. In addition, the
> performance does not scale well compared with offsets managed by Kafka's
> built-in offset management api. I have added a new option for Kafka to store
> the same data using Kafka's built-in offset management capability. The change
> is completely backward compatible with the current ZK storage option. The
> feature can be turned on by a single configuration option. Hope this will
> help people who wants to explore the option of using Kafka's built-in offset
> management api.
> References:
> https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetCommit/FetchAPI
> -thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
