Version Info: "org.apache.storm" % "storm-core" % "1.2.1" "org.apache.storm" % "storm-kafka-client" % "1.2.1"
I have a storm Topology with 3 bolts(A,B,C), Where the middle bolt takes around 450ms mean time and other two bolts takes less than 1ms. which is why I run B with parallelism hint of 1200 to 1500, but it takes more than 25 minutes to start. It takes few seconds when I run B with parallelism hint of 700, but I still feel there will be performance improvement with a higher parallelism on B, but *how to reduce the start up time of topology. * I see only following lines in worker-artifacts log, during this time: 2018-05-20 19:31:38.132 o.a.s.d.executor main [INFO] Loading executor mySlowBolt:[481 481] 2018-05-20 19:31:38.133 o.a.s.d.executor main [INFO] Loaded executor tasks mySlowBolt:[481 481] 2018-05-20 19:31:38.135 o.a.s.d.executor main [INFO] Finished loading executor mySlowBolt:[481 481] 2018-05-20 19:31:38.254 o.a.s.d.executor main [INFO] Loading executor mySlowBolt:[361 361] 2018-05-20 19:31:38.255 o.a.s.d.executor main [INFO] Loaded executor tasks mySlowBolt:[361 361] 2018-05-20 19:31:38.256 o.a.s.d.executor main [INFO] Finished loading executor mySlowBolt:[361 361] 2018-05-20 19:31:38.373 o.a.s.d.executor main [INFO] Loading executor mySlowBolt:[927 927] 2018-05-20 19:31:38.374 o.a.s.d.executor main [INFO] Loaded executor tasks mySlowBolt:[927 927] 2018-05-20 19:31:38.375 o.a.s.d.executor main [INFO] Finished loading executor mySlowBolt:[927 927] 2018-05-20 19:31:39.122 o.a.s.d.executor main [INFO] Loading executor mySlowBolt:[499 499] 2018-05-20 19:31:39.124 o.a.s.d.executor main [INFO] Loaded executor tasks mySlowBolt:[499 499] 2018-05-20 19:31:39.125 o.a.s.d.executor main [INFO] Finished loading executor mySlowBolt:[499 499] 2018-05-20 19:31:39.215 o.a.s.d.executor main [INFO] Loading executor mySlowBolt:[857 857] 2018-05-20 19:31:39.216 o.a.s.d.executor main [INFO] Loaded executor tasks mySlowBolt:[857 857] 2018-05-20 19:31:39.217 o.a.s.d.executor main [INFO] Finished loading executor mySlowBolt:[857 857] 2018-05-20 19:31:39.326 o.a.s.d.executor main [INFO] Loading executor mySlowBolt:[757 757] 2018-05-20 19:31:39.327 o.a.s.d.executor main [INFO] Loaded executor tasks mySlowBolt:[757 757] 2018-05-20 19:31:39.328 o.a.s.d.executor main [INFO] Finished loading executor mySlowBolt:[757 757] 2018-05-20 19:31:39.442 o.a.s.d.executor main [INFO] Loading executor mySlowBolt:[285 285] 2018-05-20 19:31:39.442 o.a.s.d.executor main [INFO] Loaded executor tasks mySlowBolt:[285 285] 2018-05-20 19:31:39.444 o.a.s.d.executor main [INFO] Finished loading executor mySlowBolt:[285 285] 2018-05-20 19:31:39.809 o.a.s.d.executor main [INFO] Loading executor mySlowBolt:[81 81] 2018-05-20 19:31:39.810 o.a.s.d.executor main [INFO] Loaded executor tasks mySlowBolt:[81 81] 2018-05-20 19:31:39.811 o.a.s.d.executor main [INFO] Finished loading executor mySlowBolt:[81 81] 2018-05-20 19:31:39.964 o.a.s.d.executor main [INFO] Loading executor mySlowBolt:[85 85] 2018-05-20 19:31:39.965 o.a.s.d.executor main [INFO] Loaded executor tasks mySlowBolt:[85 85] 2018-05-20 19:31:39.966 o.a.s.d.executor main [INFO] Finished loading executor mySlowBolt:[85 85] This, as I understand, is basically loading of the executor, which happens as many times as my parallelism hint, till 20 - 25 minutes. Earlier I had issue running topology with this parallelism, but after changing few configs <https://stackoverflow.com/a/50435082/1610034> I was able to resolve it, so it seems this might also be happening due to some config which I am not yet aware. is there some config, which adds delay between two executors loading? *Few more details*: I am running topology on two machines with ample RAM and CPU, with one worker on each machine, as I have found only running two workers give better performance compare to increasing workers with same number of executors. I dont have any code in prepare method of these bolts, still I see only 7-8 bolts loading per sec. SO link <https://stackoverflow.com/q/50436973/1610034>.
