HI I think you are mixing terminologies here. Loosely speaking, Master manages worker machines. Each worker machine can run one or more processes. A process can be a driver or executor. You submit applications to the master. Each application will have driver and executors. Master will decide where to put each of them. In cluster mode, master will distribute the drivers across the cluster. In client mode, master will try to run the driver processes within master's own process. You can launch multiple master processes as well and use them for a set of applications - this happens when you use YARN. I am not sure how Mesos or K8 works in that score though.
HTH... Ayan On Tue, Oct 8, 2019 at 12:11 PM Andrew Melo <andrew.m...@gmail.com> wrote: > Hi > > On Mon, Oct 7, 2019 at 19:20 Amit Sharma <resolve...@gmail.com> wrote: > >> Thanks Andrew but I am asking specific to driver memory not about >> executors memory. We have just one master and if each jobs driver.memory=4g >> and master nodes total memory is 16gb then we can not execute more than 4 >> jobs at a time. > > > I understand that. I think there's a misunderstanding with the > terminology, though. Are you running multiple separate spark instances on a > single machine or one instance with multiple jobs inside. > > >> >> On Monday, October 7, 2019, Andrew Melo <andrew.m...@gmail.com> wrote: >> >>> Hi Amit >>> >>> On Mon, Oct 7, 2019 at 18:33 Amit Sharma <resolve...@gmail.com> wrote: >>> >>>> Can you please help me understand this. I believe driver programs runs >>>> on master node >>> >>> If we are running 4 spark job and driver memory config is 4g then total >>>> 16 6b would be used of master node. >>> >>> >>> This depends on what master/deploy mode you're using: if it's "local" >>> master and "client mode" then yes tasks execute in the same JVM as the >>> driver. In this case though, the driver JVM uses whatever much space is >>> allocated for the driver regardless of how many threads you have. >>> >>> >>> So if we will run more jobs then we need more memory on master. Please >>>> correct me if I am wrong. >>>> >>> >>> This depends on your application, but in general more threads will >>> require more memory. >>> >>> >>> >>>> >>>> Thanks >>>> Amit >>>> >>> -- >>> It's dark in this basement. >>> >> -- > It's dark in this basement. > -- Best Regards, Ayan Guha
