[
https://issues.apache.org/jira/browse/STORM-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stig Rohde Døssing updated STORM-2691:
--------------------------------------
Description:
The Trident Kafka spout uses the KafkaTridentSpoutTopicPartitionRegistry enum
to pass existing topic partitions from the spout to the coordinator. This only
works when those components happen to be in the same JVM, because the
coordinator gets the topic information from the KafkaConsumer started by
KafkaTridentSpoutEmitter.
The coordinator runs in the TridentSpoutCoordinator bolt here
https://github.com/apache/storm/blob/4c8a986f519cdf3e63bed47e9c4f723e4867267a/storm-client/src/jvm/org/apache/storm/trident/topology/TridentTopologyBuilder.java#L162,
while the spout instances (emitters) run in TridentSpoutExecutors here
https://github.com/apache/storm/blob/4c8a986f519cdf3e63bed47e9c4f723e4867267a/storm-client/src/jvm/org/apache/storm/trident/topology/TridentTopologyBuilder.java#L176.
We should replace the registry enum with writes to Zookeeper or something
similar.
Edit: The fix for this is likely to be a broader change where we split the
Subscription API into a few parts so the assignment process can be split across
the coordinator and emitter instead of the emitter doing everything.
was:
The Trident Kafka spout uses the KafkaTridentSpoutTopicPartitionRegistry enum
to pass existing topic partitions from the spout to the coordinator. This only
works when those components happen to be in the same JVM, because the
coordinator gets the topic information from the KafkaConsumer started by
KafkaTridentSpoutEmitter.
The coordinator runs in the TridentSpoutCoordinator bolt here
https://github.com/apache/storm/blob/4c8a986f519cdf3e63bed47e9c4f723e4867267a/storm-client/src/jvm/org/apache/storm/trident/topology/TridentTopologyBuilder.java#L162,
while the spout instances (emitters) run in TridentSpoutExecutors here
https://github.com/apache/storm/blob/4c8a986f519cdf3e63bed47e9c4f723e4867267a/storm-client/src/jvm/org/apache/storm/trident/topology/TridentTopologyBuilder.java#L176.
We should replace the registry enum with writes to Zookeeper or something
similar.
> storm-kafka-client Trident spout implements the Trident interface incorrectly
> -----------------------------------------------------------------------------
>
> Key: STORM-2691
> URL: https://issues.apache.org/jira/browse/STORM-2691
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka-client
> Affects Versions: 2.0.0, 1.1.1
> Reporter: Stig Rohde Døssing
> Assignee: Stig Rohde Døssing
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The Trident Kafka spout uses the KafkaTridentSpoutTopicPartitionRegistry enum
> to pass existing topic partitions from the spout to the coordinator. This
> only works when those components happen to be in the same JVM, because the
> coordinator gets the topic information from the KafkaConsumer started by
> KafkaTridentSpoutEmitter.
> The coordinator runs in the TridentSpoutCoordinator bolt here
> https://github.com/apache/storm/blob/4c8a986f519cdf3e63bed47e9c4f723e4867267a/storm-client/src/jvm/org/apache/storm/trident/topology/TridentTopologyBuilder.java#L162,
> while the spout instances (emitters) run in TridentSpoutExecutors here
> https://github.com/apache/storm/blob/4c8a986f519cdf3e63bed47e9c4f723e4867267a/storm-client/src/jvm/org/apache/storm/trident/topology/TridentTopologyBuilder.java#L176.
> We should replace the registry enum with writes to Zookeeper or something
> similar.
> Edit: The fix for this is likely to be a broader change where we split the
> Subscription API into a few parts so the assignment process can be split
> across the coordinator and emitter instead of the emitter doing everything.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
