Hi, My understanding is there is only one driver/spark context for all user sessions. When you run the bin/start-connect-server script, you are submitting one long standing spark job / application. Every time a new user request comes in, a new user session is created under that. Please correct me if I am wrong.
Kezhi On Thu, Dec 14, 2023 at 10:35 AM Nikhil Goyal <nownik...@gmail.com> wrote: > [ External sender. Exercise caution. ] > > If multiple applications are running, we would need multiple spark connect > servers? If so, is the user responsible for creating these servers or they > are just created on the fly when the user requests a new spark session? > > On Thu, Dec 14, 2023 at 10:28 AM Nikhil Goyal <nownik...@gmail.com> wrote: > >> Hi folks, >> I am trying to understand one question. Does Spark Connect create a new >> driver in the backend for every user or there are a fixed number of drivers >> running to which requests are sent to? >> >> Thanks >> Nikhil >> >
