It doesnt' look like your kafka producer is distributing data across the partitions. Whats your producer looks like . Are you sending any key with each message or using null. If you are using null than what Andrew is saying might be the problem. I would recommend using random UUID as a key to send messages to your partition.
On Thu, Dec 4, 2014, at 03:12 PM, Huy Le Van wrote: > > Hi Harsha, I’ve attached 2 images below. You can see that I assigned > 16 executors, only one seemed to work. The other screenshot is the > partition table. > > Hi Andrew, That’s an interesting. I’m quite new to Kafka. May you take > a look at the second screenshot to see if the data was distributed > evenly? Let’s say it was written to one partition at a time (yes, this > is the case where I used only one producer), would it be rebalanced > afterward? > > > > > > > Best regards, Huy, Le Van > > On Thursday, Dec 4, 2014 at 10:00 p.m., Andrew Neilson > <[email protected]>, wrote: >> How is the kafka topic you are reading from partitioned? By default, >> kafka will write to a single random partition at a time for 10 >> minutes before switching to another. So if you are looking at live >> data, you would only see data in one partition at a time unless you >> use a different partitioning scheme. >> >> See the Kafka FAQ for details on this >> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified? >> >> >> On Thu, Dec 4, 2014 at 1:51 PM, Harsha <[email protected]> wrote: >> >>> __ >>> can you post your storm UI executors page image. If there are 16 >>> executors but only 1 seems to have fetching data. Can you please >>> check on your kafka producer if its distributing your data among all >>> of your partitions. >>> >>> >>> On Thu, Dec 4, 2014, at 12:32 PM, Huy Le Van wrote: >>>> >>>> Could someone help me please? >>>> >>>> Best regards, Huy, Le Van >>>> >>>> On Thursday, Dec 4, 2014 at 3:35 p.m., Huy Le Van >>>> <[email protected]>, wrote: >>>>> >>>>> Hi, >>>>> >>>>> I’m trying to tune Kafka Trident (Transactional) and seeing that >>>>> the ‘spout0’ bolt uses only one executor. The problem is exactly >>>>> as described in >>>>> https://groups.google.com/forum/#!msg/storm-user/bI7976v9R5g/fulzpnPmzkEJ >>>>> However, my Kafka topic has 16 partitions and I already set >>>>> parallelismHint of TransactionalTridentKafkaSpout to 16. What am I >>>>> doing wrong here? Please advise. >>>>> >>>>> Many thanks, Huy, Le Van >>>>> >>>> >>> >> > > Email had 2 attachments: > * storm01.png 165k (image/png) > * storm02.png 476k (image/png)
