Hi, I have a topology that’s bottlenecked right now by a terminal bolt that’s writing small batches to an endpoint. I’ve increased the number of executors several times so that it’s no longer bottlenecked there, but I still notice when there’s a traffic spike that despite capacity hovering around 1.0, probably half of the executors are idle. Can anyone give insight as to why this might be? I’ve read the docs on storm parallelism and can’t understand why this is happening. FWIW, all of the non-fieldsGrouping bolts are using localOrShuffleGrouping - perhaps this has something to do with it? I have a feeling that this is the core of the problem, but it’s not clear to me why exactly you wouldn’t use localOrShuffle over Shuffle.
Thanks, Luke
