Hi Nathan, I guess you missed the first mail of this thread.
As mentioned before I m having 3 node cluster out of which node 1 is for nimbus and ui while other 2 nodes are for worker and supervisor. The specification of machine is 4 core intel xeon processor which currently is running with more than 300% cpu utilization. At this case how can I Increase number of workers Thanks! On Feb 26, 2015 3:48 AM, "Nathan Leung" <[email protected]> wrote: > If you have only one node on the cluster then yes I would recommend to add > more nodes. If you have more than one I would recommend you increase the > worker processes in the topology configuration to use the other hardware. > On Feb 25, 2015 10:38 AM, "Vineet Mishra" <[email protected]> wrote: > >> If I am taking you right, by adding more workers you mean adding more >> nodes to the existing cluster or/and enhancing the existing configuration >> with perhaps 8 or 16 core cpu? >> >> On Wed, Feb 25, 2015 at 6:59 PM, Nathan Leung <[email protected]> wrote: >> >>> In practice it's very difficult to fully maximize cpu usage, especially >>> when each thread has to do things like wait for i/o in any way (network, >>> disk, etc). Since you're cpu limited, or very close to it, you should try >>> adding more workers. >>> On Feb 25, 2015 7:48 AM, "Vineet Mishra" <[email protected]> wrote: >>> >>>> Hi Nathan, >>>> >>>> You are right, I guess that was the case. >>>> >>>> But I am not sure why even after increasing the spouts to 20 I couldn't >>>> end up with max throughput, I hold 4 core Xeon processor Intel(R) Xeon(R) >>>> CPU E5-2680 v2 @ 2.80GHz, although I could see the high CPU utilization >>>> reaching more than 350% but is it the only reason behind the less >>>> performance gain than expected or do I need to add some configuration >>>> tuning. >>>> >>>> Thanks! >>>> >>>> On Wed, Feb 25, 2015 at 5:53 PM, Nathan Leung <[email protected]> >>>> wrote: >>>> >>>>> It looks like you configured your topology to only use one worker >>>>> process. >>>>> >>>>> On Wed, Feb 25, 2015 at 6:39 AM, Vineet Mishra <[email protected] >>>>> > wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> I am running a 3 node storm cluster with a Topology which is >>>>>> subscribed to Kafka using Kafka-Storm Spout. >>>>>> >>>>>> The Cluster is hosting Nimbus on a node and left with 2 other nodes >>>>>> for the workers and supervisor to function. >>>>>> >>>>>> While running the Topology I could see that all the spouts(20) and >>>>>> bolts(5) tasks are running on the same machine leaving the other node >>>>>> instance without any CPU utilization. >>>>>> >>>>>> [1-1] 31m 7s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 2264180 2264180 >>>>>> 11.519 2262620 0 [2-2] 31m 7s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 841540 841540 >>>>>> 19.717 840940 0 [3-3] 31m 7s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1080080 1080080 >>>>>> 18.105 1079240 0 [4-4] 31m 7s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1113900 1113900 >>>>>> 17.966 1105320 0 [5-5] 31m 7s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 854220 854220 >>>>>> 19.342 856240 0 [6-6] 31m 7s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1249640 1249640 >>>>>> 17.071 1251320 0 [7-7] 31m 7s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 2266520 2266520 >>>>>> 11.63 2267260 0 [8-8] 31m 7s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 629760 629760 >>>>>> 20.8 629120 0 [9-9] 31m 7s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 854360 854360 >>>>>> 19.513 851800 0 [10-10] 31m 7s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1417380 1417380 >>>>>> 15.78 1418200 0 [11-11] 31m 7s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 858620 858620 >>>>>> 20.897 858180 0 [12-12] 31m 7s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1242980 1242980 >>>>>> 16.766 1241100 0 [13-13] 31m 7s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 352300 352300 >>>>>> 19.975 349420 0 [14-14] 31m 7s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 627920 627920 >>>>>> 20.329 625440 0 [15-15] 31m 7s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1061880 1061880 >>>>>> 18.263 1061300 0 [16-16] 31m 7s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 706480 706480 >>>>>> 20.297 702720 0 [17-17] 31m 7s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1518800 1518800 >>>>>> 15.154 1514640 0 [18-18] 31m 7s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 622640 622640 >>>>>> 20.743 626240 0 [19-19] 31m 7s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 3376960 3376960 >>>>>> 8.361 3377060 0 [20-20] 31m 7s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1044940 1044940 >>>>>> 18.569 1043760 0 >>>>>> >>>>>> Same is the case for bolts, >>>>>> >>>>>> [22-22]} 32m 13s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0 0.002 0.05 >>>>>> 1376560 0.042 1376540 0 [23-23]} 32m 13s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0 0.001 0.021 >>>>>> 347180 0.02 347180 0 [24-24]} 32m 13s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0 0 0.023 34600 >>>>>> 0.023 34600 0 [25-25]} 32m 13s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0 0 0.017 8380 >>>>>> 0.014 8380 0 [26-26]} 32m 13s ip-20-0-0-75 6703 >>>>>> <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0 0.082 0.039 >>>>>> 22217940 0.041 22217920 0 >>>>>> >>>>>> ip-20-0-0-75 is only utilized while their is another host which is >>>>>> not utilized at all. Please find the supervisor summary below >>>>>> >>>>>> 8e550ab9-9e0d-48a7-8899-0963d5f00a41 ip-20-0-0-78 1d 5h 7m 32s 4 0 >>>>>> 9d98d0b4-1bb4-42e9-9a72-a67b82c64b2c ip-20-0-0-75 5d 23h 57m 53s 4 1 >>>>>> This is not the regular behavior of Storm as I have seen both the >>>>>> nodes operating while running the jobs, but how we can comment on the >>>>>> stability of the storm with this kind off behavior. >>>>>> >>>>>> Urgent Issue, Request for expert intervention. >>>>>> >>>>>> Thanks! >>>>>> >>>>> >>>>> >>>> >>
