Hi All, I am running a 3 node storm cluster with a Topology which is subscribed to Kafka using Kafka-Storm Spout.
The Cluster is hosting Nimbus on a node and left with 2 other nodes for the workers and supervisor to function. While running the Topology I could see that all the spouts(20) and bolts(5) tasks are running on the same machine leaving the other node instance without any CPU utilization. [1-1] 31m 7s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 2264180 2264180 11.519 2262620 0 [2-2] 31m 7s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 841540 841540 19.717 840940 0 [3-3] 31m 7s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1080080 1080080 18.105 1079240 0 [4-4] 31m 7s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1113900 1113900 17.966 1105320 0 [5-5] 31m 7s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 854220 854220 19.342 856240 0 [6-6] 31m 7s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1249640 1249640 17.071 1251320 0 [7-7] 31m 7s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 2266520 2266520 11.63 2267260 0 [8-8] 31m 7s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 629760 629760 20.8 629120 0 [9-9] 31m 7s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 854360 854360 19.513 851800 0 [10-10] 31m 7s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1417380 1417380 15.78 1418200 0 [11-11] 31m 7s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 858620 858620 20.897 858180 0 [12-12] 31m 7s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1242980 1242980 16.766 1241100 0 [13-13] 31m 7s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 352300 352300 19.975 349420 0 [14-14] 31m 7s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 627920 627920 20.329 625440 0 [15-15] 31m 7s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1061880 1061880 18.263 1061300 0 [16-16] 31m 7s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 706480 706480 20.297 702720 0 [17-17] 31m 7s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1518800 1518800 15.154 1514640 0 [18-18] 31m 7s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 622640 622640 20.743 626240 0 [19-19] 31m 7s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 3376960 3376960 8.361 3377060 0 [20-20] 31m 7s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 1044940 1044940 18.569 1043760 0 Same is the case for bolts, [22-22]} 32m 13s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0 0.002 0.05 1376560 0.042 1376540 0 [23-23]} 32m 13s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0 0.001 0.021 347180 0.02 347180 0 [24-24]} 32m 13s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0 0 0.023 34600 0.023 34600 0 [25-25]} 32m 13s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0 0 0.017 8380 0.014 8380 0 [26-26]} 32m 13s ip-20-0-0-75 6703 <http://ip-20-0-0-75:8000/log?file=worker-6703.log> 0 0 0.082 0.039 22217940 0.041 22217920 0 ip-20-0-0-75 is only utilized while their is another host which is not utilized at all. Please find the supervisor summary below 8e550ab9-9e0d-48a7-8899-0963d5f00a41 ip-20-0-0-78 1d 5h 7m 32s 4 0 9d98d0b4-1bb4-42e9-9a72-a67b82c64b2c ip-20-0-0-75 5d 23h 57m 53s 4 1 This is not the regular behavior of Storm as I have seen both the nodes operating while running the jobs, but how we can comment on the stability of the storm with this kind off behavior. Urgent Issue, Request for expert intervention. Thanks!
