[
https://issues.apache.org/jira/browse/KAFKA-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240166#comment-13240166
]
Jun Rao commented on KAFKA-249:
-------------------------------
Thanks for the patch. Some comments:
1. For future extension, I am thinking that we should probably unifying
KafkaMessageStream and KafkaMessageAndTopicStream to sth like
KafkaMessageMetadataStream. The stream gives a iterator of Message and its
associated meta data. For now, the meta data can be just topic. In the future,
it may include things like partition id and offset.
2. ZookeeperConsumerConnector:
2.1 updateFetcher: no need to pass in messagStreams
2.2 ZKRebalancerListener: It seems that kafkaMessageStream can be immutable.
2.3 createMessageStreamByFilter: topicsStreamsMap is empty when passed to
ZKRebalanceListener. This means that the queue is not cleared during rebalance.
2.4 consumeWildCardTopics: I find it hard to read the code in this method. Is
there a real benefit to use implicit conversion here, instead of explicit
conversion? It's not clear to me where the conversion is used. The 2-level
tuple makes it hard to figure out what the referred fields represent. Is the
code relying on groupedTopicThreadIds being sorted by (topic, threadid)? If so,
where is that enforced.
3. KafkaServerStartable: Should we remove the embedded consumer now?
4. Utils, UtilsTest: unused import
> Separate out Kafka mirroring into a stand-alone app
> ---------------------------------------------------
>
> Key: KAFKA-249
> URL: https://issues.apache.org/jira/browse/KAFKA-249
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Reporter: Joel Koshy
> Assignee: Joel Koshy
> Fix For: 0.7.1
>
> Attachments: KAFKA-249.v1.patch, KAFKA-249.v2.patch
>
>
> I would like to discuss on this jira, the feasibility/benefits of separating
> out Kafka's mirroring feature from the broker into a stand-alone app, as it
> currently has a couple of limitations and issues.
> For example, we recently had to deal with Kafka mirrors that were in fact
> idle due to the fact that mirror threads were not created at start-up due to
> a rebalancing exception, but the Kafka broker itself did not shutdown. This
> has since been fixed, but is indicative of (avoidable) problems in embedding
> non-broker specific features in the broker.
> Logically, it seems to make sense to separate it out to achieve better
> division of labor. Furthermore, enhancements to mirroring may be less
> clunky to implement and use with a stand-alone app. For example to support
> custom partitioning on the target cluster, or to mirror from multiple
> clusters we would probably need to be able to pass in multiple embedded
> consumer/embedded producer configs, which would be less ugly if the
> mirroring process were a stand-alone app. Also, if we break it out, it
> would be convenient to use as a "consumption engine" for the console
> consumer which will make it easier to add on features such as wildcards in
> topic consumption, since it contains a ZooKeeper topic discovery component.
> Any suggestions and/or objections to this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
