Re: Architecture of Spark Connect
By default for now, yes. One Spark Connect server handles multiple Spark Sessions. To multiplex or run multiple Drivers, you need some work such as gateway. On Thu, 14 Dec 2023 at 12:03, Kezhi Xiong wrote: > Hi, > > My understanding is there is only one driver/spark context for all user > sessions. When you run the bin/start-connect-server script, you are > submitting one long standing spark job / application. Every time a new user > request comes in, a new user session is created under that. Please correct > me if I am wrong. > > Kezhi > > On Thu, Dec 14, 2023 at 10:35 AM Nikhil Goyal wrote: > >> [ External sender. Exercise caution. ] >> >> If multiple applications are running, we would need multiple spark >> connect servers? If so, is the user responsible for creating these servers >> or they are just created on the fly when the user requests a new spark >> session? >> >> On Thu, Dec 14, 2023 at 10:28 AM Nikhil Goyal >> wrote: >> >>> Hi folks, >>> I am trying to understand one question. Does Spark Connect create a new >>> driver in the backend for every user or there are a fixed number of drivers >>> running to which requests are sent to? >>> >>> Thanks >>> Nikhil >>> >>
Re: Architecture of Spark Connect
Hi, My understanding is there is only one driver/spark context for all user sessions. When you run the bin/start-connect-server script, you are submitting one long standing spark job / application. Every time a new user request comes in, a new user session is created under that. Please correct me if I am wrong. Kezhi On Thu, Dec 14, 2023 at 10:35 AM Nikhil Goyal wrote: > [ External sender. Exercise caution. ] > > If multiple applications are running, we would need multiple spark connect > servers? If so, is the user responsible for creating these servers or they > are just created on the fly when the user requests a new spark session? > > On Thu, Dec 14, 2023 at 10:28 AM Nikhil Goyal wrote: > >> Hi folks, >> I am trying to understand one question. Does Spark Connect create a new >> driver in the backend for every user or there are a fixed number of drivers >> running to which requests are sent to? >> >> Thanks >> Nikhil >> >
Re: Architecture of Spark Connect
If multiple applications are running, we would need multiple spark connect servers? If so, is the user responsible for creating these servers or they are just created on the fly when the user requests a new spark session? On Thu, Dec 14, 2023 at 10:28 AM Nikhil Goyal wrote: > Hi folks, > I am trying to understand one question. Does Spark Connect create a new > driver in the backend for every user or there are a fixed number of drivers > running to which requests are sent to? > > Thanks > Nikhil >
Architecture of Spark Connect
Hi folks, I am trying to understand one question. Does Spark Connect create a new driver in the backend for every user or there are a fixed number of drivers running to which requests are sent to? Thanks Nikhil
