[
https://issues.apache.org/jira/browse/SPARK-61008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61008:
------------------------------------
Assignee: Hyukjin Kwon
> Get or create the sessions of SparkSession.builder of the Python client as
> PySpark does
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-61008
> URL: https://issues.apache.org/jira/browse/SPARK-61008
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark 4.3.0 Connect, {{SparkSession.builder.getOrCreate()}} returns the
> active session of the thread, or else the default session, and creates one
> only when there is neither; it applies the options of the builder to the
> session that it returns. Its URL is that of {{remote()}}, or else of
> {{SPARK_REMOTE}}, and it raises {{CONNECT_URL_NOT_SET}} without either. The
> first session created is the default session, and the active session is
> thread-local: {{create()}} sets the default and the active session only when
> they are not set, and {{stop()}} unsets them.
> The Rust-backed Python client creates a new session at each
> {{getOrCreate()}}, so the configurations and temporary views of a session are
> not those of the next one: after
> {{spark.conf.set("spark.sql.shuffle.partitions", "7")}} and
> {{createOrReplaceTempView("v")}}, a second {{getOrCreate()}} returns a
> session whose configuration is {{200}} and that has no view {{v}}. It raises
> {{ValueError}} without {{remote()}}, even with {{SPARK_REMOTE}} set, and
> keeps one active session for all threads, which {{create()}} replaces. Get or
> create sessions as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
