[
https://issues.apache.org/jira/browse/KAFKA-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240616#comment-13240616
]
Neha Narkhede commented on KAFKA-249:
-------------------------------------
This is a cool feature. Thanks for the patch !
A couple of questions -
1. It seems awkward that there is a MessageStream trait and the only API it
exposes is clear(). Any reason it doesn't expose the iterator() API ? From a
user's perspective, one might think, since it is a stream, it would expose
stream specific APIs too. It will be good to add docs to that API to explain
exactly what it is meant for.
2. I like the idea of having just one KafkaMessageMetadataStream.
3. There is some critical code that is duplicated in the
ZookeeperConsumerConnector. consume() and consumeWildcardTopics() have some
code in common. It would be great if this can be refactored to share the logic
of registering session expiration listeners, registering watches on consumer
group changes and topic partition changes.
4. Could you merge all the logic that wraps the wildcard handling in one API ?
Right now, it is distributed between createMessageStreamsByFilter and
consumeWildcardTopics. It will be great if there is one API that will pre
process the wild cards, create the relevant queues and then call a common
consume() that has the logic described in item 5 above.
5. There are several new class variables called wildcard* in
ZookeeperConsumerConnector. I'm thinking they can just be variables local to
createMessageStreamsByFilter ?
6. There is a MessageAndTopic class, that seems to act as a container to hold
message and other metadata, but only exposes one API to get the message. Topic
is exposed by making it a public val. Would it make sense to either make it a
case class or provide consistent APIs for all fields it holds ?
7. Since now we seem to have more than one iterators for the consumer, would it
make sense to rename ConsumerIterator to MessageIterator, and
TopicalConsumerIterator to MessageAndMetadataIterator ?
8. rat fails on this patch. There are some files without the Apache header
> Separate out Kafka mirroring into a stand-alone app
> ---------------------------------------------------
>
> Key: KAFKA-249
> URL: https://issues.apache.org/jira/browse/KAFKA-249
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Reporter: Joel Koshy
> Assignee: Joel Koshy
> Fix For: 0.7.1
>
> Attachments: KAFKA-249.v1.patch, KAFKA-249.v2.patch
>
>
> I would like to discuss on this jira, the feasibility/benefits of separating
> out Kafka's mirroring feature from the broker into a stand-alone app, as it
> currently has a couple of limitations and issues.
> For example, we recently had to deal with Kafka mirrors that were in fact
> idle due to the fact that mirror threads were not created at start-up due to
> a rebalancing exception, but the Kafka broker itself did not shutdown. This
> has since been fixed, but is indicative of (avoidable) problems in embedding
> non-broker specific features in the broker.
> Logically, it seems to make sense to separate it out to achieve better
> division of labor. Furthermore, enhancements to mirroring may be less
> clunky to implement and use with a stand-alone app. For example to support
> custom partitioning on the target cluster, or to mirror from multiple
> clusters we would probably need to be able to pass in multiple embedded
> consumer/embedded producer configs, which would be less ugly if the
> mirroring process were a stand-alone app. Also, if we break it out, it
> would be convenient to use as a "consumption engine" for the console
> consumer which will make it easier to add on features such as wildcards in
> topic consumption, since it contains a ZooKeeper topic discovery component.
> Any suggestions and/or objections to this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
