Some interesting observations - I can create only have parallelism around 415 without getting the java.lang.OutOfMemoryError : unable to create new native Thread This is despite - 1. Heap being 2g and above 2. system memory and swap were still under control (Under 30% utilization). CPU was under 10% 3. ulimit -a shows open files and no. of processers at 65336 4. XSS has no impact -> Tried 104K (default of 64bit Open JDK 7) to all numbers high and low -> has no affect
Question now I have is - Who creates the supervisors? Is it the worker thread or the supervisor thread or nimbus? I think it is the worker thread. If I need to change the JVM params to see what is teh max parallelism hint I can give, do I change this for supervisor child opts, worker child opts? Or storm itself in general? Thanks Kashyap On Thu, Jan 7, 2016 at 11:58 PM, Anishek Agarwal <[email protected]> wrote: > More stack size per thread will definitely require more memory for sure, > its just that i didnt spend too much time investigating the cause. and for > oracle default is 228k so we cant go lower than that. > > Please do let us know your observations / inferences from the test. > > thanks > anishek > > On Thu, Jan 7, 2016 at 6:43 PM, Kashyap Mhaisekar <[email protected]> > wrote: > >> That's one area that is confusing... Xss indicates stack size at each >> thread level. Increasing it means that each thread effectively has more >> memory for stack and hence overall requirements for memory increase. Am not >> really sure how increasing stack size allows for more thread creation. Will >> test this part and confirm back >> >> Thanks >> Kashyap >> On Jan 7, 2016 02:58, "Anishek Agarwal" <[email protected]> wrote: >> >>> Yeh I have encountered the same error, and as you said its because there >>> is a limit to the number of threads with a specific stack size, that can be >>> created for a given heap, there are a few discussions on Stackoverflow for >>> the same , I also tried to verify this via a standalone program but it was >>> behaving very non intuitively on the local machine, for ex it was stopping >>> after creating 980 threads always with the same stack size of 256k and Heap >>> size varying from default to 1 GB. >>> >>> we are using oracle jdk and hence in our case the the stack size cannont >>> below 228k we have it as 256k with about 4 GB of -Xmx its working fine with >>> increased number of threads. >>> >>> On Wed, Jan 6, 2016 at 7:14 PM, Kashyap Mhaisekar <[email protected]> >>> wrote: >>> >>>> Thanks Anishek. >>>> I read that OOM due to lack of native threads is because of stack size >>>> or because of OS running out of threads. >>>> >>>> Did you specifically encounter this error in your use case? >>>> >>>> Thanks >>>> Kashyap >>>> On Jan 6, 2016 02:32, "Anishek Agarwal" <[email protected]> wrote: >>>> >>>>> Hey Kashyap, >>>>> >>>>> There seems to be a lot of threads that are created per bolt thread >>>>> within storm for processing. for ex we have apporx 100 parallelism per >>>>> worker(with all bolts in a topology) and we had to specify the -Xmx to 4 >>>>> GB >>>>> -- internally looked like the process was having abour 3.5-4K threads. The >>>>> number of instances that can be created will depend on the heap size per >>>>> worker gets so larger it is higher number of threads, from what you have >>>>> published it seems it would be better to have less workers per box say 4-8 >>>>> and higher max heap per worker so you can create more threads per worker. >>>>> >>>>> anishek >>>>> >>>>> On Wed, Jan 6, 2016 at 6:33 AM, Kashyap Mhaisekar <[email protected] >>>>> > wrote: >>>>> >>>>>> Priyank, >>>>>> Already seen this. My question was more on practical limitations on >>>>>> parallelism than doing it right. >>>>>> >>>>>> Thanks anyways. >>>>>> >>>>>> Regards >>>>>> Kashyap >>>>>> On Jan 5, 2016 4:47 PM, "Priyank Shah" <[email protected]> wrote: >>>>>> >>>>>>> Hi Kashyap, >>>>>>> >>>>>>> Please check out >>>>>>> http://www.slideshare.net/ptgoetz/scaling-apache-storm-strata-hadoopworld-2014 >>>>>>> >>>>>>> >>>>>>> Priyank >>>>>>> >>>>>>> >>>>>>> From: Kashyap Mhaisekar >>>>>>> Reply-To: "[email protected]" >>>>>>> Date: Tuesday, January 5, 2016 at 1:21 PM >>>>>>> To: "[email protected]" >>>>>>> Subject: Storm bolt instances - limits >>>>>>> >>>>>>> Hi, >>>>>>> I had a question on the no. of instances of bolts that can be >>>>>>> defined on a topology. Here is my use case - >>>>>>> 1. 32 GB machine >>>>>>> 2. 8 Processors >>>>>>> 3. 12 Workers for each box (6700 to 6711) >>>>>>> 4. Each worker has been defined with 2G heap and 64mb of XSS. Ulimit >>>>>>> for storm is at 1024 >>>>>>> >>>>>>> My topology has 5 bolts and a spout and If the combined parallelism >>>>>>> is above 1000, i see OOM errrors indicating - >>>>>>> java.lang.OutOfMemoryError: >>>>>>> unable to create new native thread >>>>>>> >>>>>>> When this error occurs, topology is getting initialized and has not >>>>>>> started processing. >>>>>>> >>>>>>> Is there a guideline from resource perspective as to how many >>>>>>> instances can be created? >>>>>>> >>>>>>> Thanks >>>>>>> Kashyap >>>>>>> >>>>>> >>>>> >>> >
