How to Scale Streaming Application to Multiple Workers

[Artemis User]

Hi,

We have a streaming application that read microbatch csv files and 
involves the foreachBatch call.  Each microbatch can be processed 
independently.  I noticed that only one worker node is being utilized.  
Is there anyway or any explicit method to distribute the batch work load 
to multiple workers?  I would think Spark would execute foreachBatch 
method on different workers since each batch can be treated as atomic?

Thanks!

ND


---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org