I have a question abot Kafka Spout (
https://github.com/apache/storm/tree/master/external/storm-kafka) that
need your help.
I build Kafka Spout and set to use 7 worker to receive Kafka messages.
builder.setSpout(Utils.TOPIC_DEFAULT, new KafkaSpout(spoutConfig_Default),
7);
And finally I find that all messages go to only one worker, which is shown
as the diagram below.
Actually my Kafka topic has multiple partitions.
$ bin/kafka-topics.sh --topic general_dev --describe --zookeeper
localhost:2181
Topic:general_dev PartitionCount:7 ReplicationFactor:1
Configs:
Topic: general_dev Partition: 0 Leader: 1 Replicas: 1
Isr: 1
Topic: general_dev Partition: 1 Leader: 2 Replicas: 2
Isr: 2
Topic: general_dev Partition: 2 Leader: 3 Replicas: 3
Isr: 3
Topic: general_dev Partition: 3 Leader: 4 Replicas: 4
Isr: 4
Topic: general_dev Partition: 4 Leader: 5 Replicas: 5
Isr: 5
Topic: general_dev Partition: 5 Leader: 6 Replicas: 6
Isr: 6
Topic: general_dev Partition: 6 Leader: 7 Replicas: 7
Isr: 7
1. Does the Kafka Spout support multiple worker and how to do it?
2. If my topic has multiple partitions, and I want each partition to be
consumed by one worker to achieve the max performance.
If the current KafkaSpout cannot do this, could you please give some
suggestions on how to make it?
--
Regards,
Sai
