Storm topology with old kafka spout connected to local kafka show great performance, which I'm satisfied with. But when I connect to external kafka, which is located on separate cluster, spout performance drops significantly. And the same topology works 10 times slower. I've already tried to increase this spout setting: - bufferSizeBytes up to 67108864 - fetchSizeBytes up to 67108864 - minFetchBytes up to 67108864 - maxSpoutPending was doubled of old value
All experiments gave me a slight increase in performance, but still far from the values obtained when connecting to the local kafka. I thought my network throughput is low, but when I've increased kafka spout parallelism, storm throughput increased proportionally.
