Hi Amit On Mon, Oct 7, 2019 at 18:33 Amit Sharma <resolve...@gmail.com> wrote:
> Can you please help me understand this. I believe driver programs runs on > master node If we are running 4 spark job and driver memory config is 4g then total 16 > 6b would be used of master node. This depends on what master/deploy mode you're using: if it's "local" master and "client mode" then yes tasks execute in the same JVM as the driver. In this case though, the driver JVM uses whatever much space is allocated for the driver regardless of how many threads you have. So if we will run more jobs then we need more memory on master. Please > correct me if I am wrong. > This depends on your application, but in general more threads will require more memory. > > Thanks > Amit > -- It's dark in this basement.
