[
https://issues.apache.org/jira/browse/STORM-1015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14901484#comment-14901484
]
ASF GitHub Bot commented on STORM-1015:
---------------------------------------
Github user erikdw commented on the pull request:
https://github.com/apache/storm/pull/705#issuecomment-142121623
@hsun-cnnxty : I don't see the reference links in the Description on the
PR's Conversation view? Maybe I'm looking in the wrong place? Maybe they
should be in comments in the code too?
I'm guessing the following are the links you meant to put:
*
https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
*
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
I just want it to be very clear that this change is for changing from the
kafka spout's consumer offsets being stored in ZooKeeper, to instead being
stored directly in Kafka. We should also be clear about the version of Kafka
required for such support (0.8.2+). I know the current version of the
storm-kafka pom.xml (as of this change) is already referencing 0.8.2.1, but I
feel like it should be called out as an explicit requirement in the commit.
> Store Kafka offsets with Kafka's consumer offset management api
> ---------------------------------------------------------------
>
> Key: STORM-1015
> URL: https://issues.apache.org/jira/browse/STORM-1015
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-kafka
> Affects Versions: 0.11.0
> Reporter: Hang Sun
> Priority: Minor
> Labels: consumer, kafka, offset
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> Current Kafka spout stores the offsets (and some other states) inside ZK with
> its proprietary format. This does not work well with other Kafka offset
> monitoring tools such as Burrow, KafkaOffsetMonitor etc. In addition, the
> performance does not scale well compared with offsets managed by Kafka's
> built-in offset management api. I have added a new option for Kafka to store
> the same data using Kafka's built-in offset management capability. The change
> is completely backward compatible with the current ZK storage option. The
> feature can be turned on by a single configuration option. Hope this will
> help people who wants to explore the option of using Kafka's built-in offset
> management api.
> References:
> https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetCommit/FetchAPI
> -thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
