How is the kafka topic you are reading from partitioned? By default, kafka will write to a single random partition at a time for 10 minutes before switching to another. So if you are looking at live data, you would only see data in one partition at a time unless you use a different partitioning scheme.
See the Kafka FAQ for details on this https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified ? On Thu, Dec 4, 2014 at 1:51 PM, Harsha <[email protected]> wrote: > can you post your storm UI executors page image. If there are 16 > executors but only 1 seems to have fetching data. Can you please check on > your kafka producer if its distributing your data among all of your > partitions. > > > On Thu, Dec 4, 2014, at 12:32 PM, Huy Le Van wrote: > > > Could someone help me please? > > Best regards, > Huy, Le Van > > On Thursday, Dec 4, 2014 at 3:35 p.m., Huy Le Van < > [email protected]>, wrote: > > > Hi, > > I’m trying to tune Kafka Trident (Transactional) and seeing that the > ‘spout0’ bolt uses only one executor. The problem is exactly as described > in > https://groups.google.com/forum/#!msg/storm-user/bI7976v9R5g/fulzpnPmzkEJ > However, my Kafka topic has 16 partitions and I already set > parallelismHint of TransactionalTridentKafkaSpout to 16. What am I doing > wrong here? Please advise. > > Many thanks, > Huy, Le Van > > > > > >
