Ruifeng Zheng created SPARK-44750:
-------------------------------------
Summary: SparkSession.Builder should respect the options
Key: SPARK-44750
URL: https://issues.apache.org/jira/browse/SPARK-44750
Project: Spark
Issue Type: Improvement
Components: Connect, PySpark
Affects Versions: 3.5.0, 4.0.0
Reporter: Ruifeng Zheng
In connect session builder, we use {{config}} method to set options.
However, the options are actually ignored.
{code}
def create(self) -> "SparkSession":
has_channel_builder = self._channel_builder is not None
has_spark_remote = "spark.remote" in self._options
if has_channel_builder and has_spark_remote:
raise ValueError(
"Only one of connection string or channelBuilder "
"can be used to create a new SparkSession."
)
if not has_channel_builder and not has_spark_remote:
raise ValueError(
"Needs either connection string or channelBuilder to create
a new SparkSession."
)
if has_channel_builder:
assert self._channel_builder is not None
session = SparkSession(connection=self._channel_builder)
else:
spark_remote = to_str(self._options.get("spark.remote"))
assert spark_remote is not None
session = SparkSession(connection=spark_remote)
SparkSession._set_default_and_active_session(session)
return session
{code}
we should respect the options by invoking {{session.conf.set}} after creation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
