How to scale or possibly auto-scale a spark streaming application consuming from kafka and using kafka direct streams. We are using spark 1.6.3, cannot move to 2.x unless there is a strong reason.
Scenario: Kafka topic with 10 partitions Standalone cluster running on kubernetes with 1 master and 2 workers What we would like to know? Increase the number of partitions (say from 10 to 15) Add additional worker node without restarting the streaming application and start consuming off the additional partitions. Is this possible? i.e. start additional workers in standalone cluster to auto-scale an existing spark streaming application that is already running or we have to stop and resubmit the streaming app? Best Regards, Pranav Shukla