i think the worker has to be changed using "worker.childopts" , we are currently changing this via /opt/storm/conf/storm.yaml
i am not sure if there is something we have to do for Open JDK , you might want to look that up a bit :) On Sun, Jan 10, 2016 at 11:32 AM, Kashyap Mhaisekar <[email protected]> wrote: > Some interesting observations - > > I can create only have parallelism around 415 without getting the > java.lang.OutOfMemoryError > : unable to create new native Thread This is despite - > 1. Heap being 2g and above > 2. system memory and swap were still under control (Under 30% > utilization). CPU was under 10% > 3. ulimit -a shows open files and no. of processers at 65336 > 4. XSS has no impact -> Tried 104K (default of 64bit Open JDK 7) to all > numbers high and low -> has no affect > > Question now I have is - Who creates the supervisors? Is it the worker > thread or the supervisor thread or nimbus? I think it is the worker thread. > If I need to change the JVM params to see what is teh max parallelism hint > I can give, do I change this for supervisor child opts, worker child opts? > Or storm itself in general? > > Thanks > Kashyap > > On Thu, Jan 7, 2016 at 11:58 PM, Anishek Agarwal <[email protected]> > wrote: > >> More stack size per thread will definitely require more memory for sure, >> its just that i didnt spend too much time investigating the cause. and for >> oracle default is 228k so we cant go lower than that. >> >> Please do let us know your observations / inferences from the test. >> >> thanks >> anishek >> >> On Thu, Jan 7, 2016 at 6:43 PM, Kashyap Mhaisekar <[email protected]> >> wrote: >> >>> That's one area that is confusing... Xss indicates stack size at each >>> thread level. Increasing it means that each thread effectively has more >>> memory for stack and hence overall requirements for memory increase. Am not >>> really sure how increasing stack size allows for more thread creation. Will >>> test this part and confirm back >>> >>> Thanks >>> Kashyap >>> On Jan 7, 2016 02:58, "Anishek Agarwal" <[email protected]> wrote: >>> >>>> Yeh I have encountered the same error, and as you said its because >>>> there is a limit to the number of threads with a specific stack size, that >>>> can be created for a given heap, there are a few discussions on >>>> Stackoverflow for the same , I also tried to verify this via a standalone >>>> program but it was behaving very non intuitively on the local machine, for >>>> ex it was stopping after creating 980 threads always with the same stack >>>> size of 256k and Heap size varying from default to 1 GB. >>>> >>>> we are using oracle jdk and hence in our case the the stack size >>>> cannont below 228k we have it as 256k with about 4 GB of -Xmx its working >>>> fine with increased number of threads. >>>> >>>> On Wed, Jan 6, 2016 at 7:14 PM, Kashyap Mhaisekar <[email protected]> >>>> wrote: >>>> >>>>> Thanks Anishek. >>>>> I read that OOM due to lack of native threads is because of stack size >>>>> or because of OS running out of threads. >>>>> >>>>> Did you specifically encounter this error in your use case? >>>>> >>>>> Thanks >>>>> Kashyap >>>>> On Jan 6, 2016 02:32, "Anishek Agarwal" <[email protected]> wrote: >>>>> >>>>>> Hey Kashyap, >>>>>> >>>>>> There seems to be a lot of threads that are created per bolt thread >>>>>> within storm for processing. for ex we have apporx 100 parallelism per >>>>>> worker(with all bolts in a topology) and we had to specify the -Xmx to 4 >>>>>> GB >>>>>> -- internally looked like the process was having abour 3.5-4K threads. >>>>>> The >>>>>> number of instances that can be created will depend on the heap size per >>>>>> worker gets so larger it is higher number of threads, from what you have >>>>>> published it seems it would be better to have less workers per box say >>>>>> 4-8 >>>>>> and higher max heap per worker so you can create more threads per worker. >>>>>> >>>>>> anishek >>>>>> >>>>>> On Wed, Jan 6, 2016 at 6:33 AM, Kashyap Mhaisekar < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Priyank, >>>>>>> Already seen this. My question was more on practical limitations on >>>>>>> parallelism than doing it right. >>>>>>> >>>>>>> Thanks anyways. >>>>>>> >>>>>>> Regards >>>>>>> Kashyap >>>>>>> On Jan 5, 2016 4:47 PM, "Priyank Shah" <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Kashyap, >>>>>>>> >>>>>>>> Please check out >>>>>>>> http://www.slideshare.net/ptgoetz/scaling-apache-storm-strata-hadoopworld-2014 >>>>>>>> >>>>>>>> >>>>>>>> Priyank >>>>>>>> >>>>>>>> >>>>>>>> From: Kashyap Mhaisekar >>>>>>>> Reply-To: "[email protected]" >>>>>>>> Date: Tuesday, January 5, 2016 at 1:21 PM >>>>>>>> To: "[email protected]" >>>>>>>> Subject: Storm bolt instances - limits >>>>>>>> >>>>>>>> Hi, >>>>>>>> I had a question on the no. of instances of bolts that can be >>>>>>>> defined on a topology. Here is my use case - >>>>>>>> 1. 32 GB machine >>>>>>>> 2. 8 Processors >>>>>>>> 3. 12 Workers for each box (6700 to 6711) >>>>>>>> 4. Each worker has been defined with 2G heap and 64mb of XSS. >>>>>>>> Ulimit for storm is at 1024 >>>>>>>> >>>>>>>> My topology has 5 bolts and a spout and If the combined parallelism >>>>>>>> is above 1000, i see OOM errrors indicating - >>>>>>>> java.lang.OutOfMemoryError: >>>>>>>> unable to create new native thread >>>>>>>> >>>>>>>> When this error occurs, topology is getting initialized and has not >>>>>>>> started processing. >>>>>>>> >>>>>>>> Is there a guideline from resource perspective as to how many >>>>>>>> instances can be created? >>>>>>>> >>>>>>>> Thanks >>>>>>>> Kashyap >>>>>>>> >>>>>>> >>>>>> >>>> >> >
