Kumar Gaurav created STORM-418:
----------------------------------
Summary: using storm kafka spout doesn't scale when increasing
topic partition
Key: STORM-418
URL: https://issues.apache.org/jira/browse/STORM-418
Project: Apache Storm (Incubating)
Issue Type: Bug
Affects Versions: 0.9.2-incubating
Environment: EC2 c3.8x large instances for supervisor and
kafka-0.8.1.1 running Amazon Linux 64 bit version OS.
Kafka is using 4 EBS drives of 4000 IOPS of size 150 GB each.
Reporter: Kumar Gaurav
We are running a topology of kafka spout + single bolt on a single supervisor
machine.
Bolt functionality is limited to log parsed data from spout only.
When Trying to consume same type of data by different topics using varying
topic partitions topology throughput remain same in order of 70000 msg/sec.
However, when multiple spout instances are used in same topology , we were able
to increase topology throughput to more than 100,000 msg/sec.
We have tried to increase kafka topic partition from 8 to 200, however
topology throughput remains same for sinlge spout instance.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
