Priyank, Already seen this. My question was more on practical limitations on parallelism than doing it right.
Thanks anyways. Regards Kashyap On Jan 5, 2016 4:47 PM, "Priyank Shah" <ps...@hortonworks.com> wrote: > Hi Kashyap, > > Please check out > http://www.slideshare.net/ptgoetz/scaling-apache-storm-strata-hadoopworld-2014 > > > Priyank > > > From: Kashyap Mhaisekar > Reply-To: "user@storm.apache.org" > Date: Tuesday, January 5, 2016 at 1:21 PM > To: "user@storm.apache.org" > Subject: Storm bolt instances - limits > > Hi, > I had a question on the no. of instances of bolts that can be defined on a > topology. Here is my use case - > 1. 32 GB machine > 2. 8 Processors > 3. 12 Workers for each box (6700 to 6711) > 4. Each worker has been defined with 2G heap and 64mb of XSS. Ulimit for > storm is at 1024 > > My topology has 5 bolts and a spout and If the combined parallelism is > above 1000, i see OOM errrors indicating - java.lang.OutOfMemoryError: > unable to create new native thread > > When this error occurs, topology is getting initialized and has not > started processing. > > Is there a guideline from resource perspective as to how many instances > can be created? > > Thanks > Kashyap >
