Ignore my previous emails. I rushed to read them :-). On Tue, 23 Jun 2026 at 07:59, Hyukjin Kwon <[email protected]> wrote:
> > but I think it doesn't work with the interpreter way Nicholas mentioned. > > Oh oops sorry I forgot that I added that support as well. > Anyway it's still related to > https://lists.apache.org/thread/j382to15zgy8mr2pvrtcod9c02zj1org > > On Tue, 23 Jun 2026 at 07:57, Hyukjin Kwon <[email protected]> wrote: > >> Spark already supports `bin/pyspark --remote local` now but I think it >> doesn't work with the interpreter way Nicholas mentioned. >> This is probably related to >> https://lists.apache.org/thread/j382to15zgy8mr2pvrtcod9c02zj1org as well. >> >> >> On Tue, 23 Jun 2026 at 05:05, Tian Gao via dev <[email protected]> >> wrote: >> >>> Yes I think that's totally fine. >>> >>> I even support `.remote("local").getOrCreate()` as long as `"local"` has >>> no other ambiguity (is it used now? if not I'm fine to have this as a >>> special value). What I don't support is that we change the behavior for >>> something we are supporting now. For example, >>> .remote("sc://localhost:15002").getOrCreate() should not magically create >>> the server if there's no existing server. >>> >>> Tian >>> >>> On Mon, Jun 22, 2026 at 12:01 PM Nicholas Chammas < >>> [email protected]> wrote: >>> >>>> >>>> > On Jun 22, 2026, at 2:45 PM, Tian Gao <[email protected]> >>>> wrote: >>>> > >>>> > All I'm saying is that this is a tradeoff, it has benefits in some >>>> cases, but it's not a free lunch. I support having this as a feature when >>>> users know what they are doing, aka explicit consent. I'm against making >>>> this an implicit default to replace the existing behavior. >>>> >>>> OK, so are you in support of the weaker solution I outlined then? That >>>> is, we add a `spark connect` CLI with some accompanying user guide >>>> documentation, and make any existing Connect server discoverable by >>>> `.remote(“local”)`. The user still has to explicitly start up a persistent >>>> server. >>>> >>>> I think down the line we want to consider making this part of the >>>> default local workflow, but we can revisit that after this first step. >>> >>>
