This is an automated email from the ASF dual-hosted git repository.
ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ed03173f33a [SPARK-45024][PYTHON][CONNECT] Filter out some
configurations in Session Creation
ed03173f33a is described below
commit ed03173f33aad0cf8b18096a4fb2470059410751
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Thu Aug 31 12:36:35 2023 +0800
[SPARK-45024][PYTHON][CONNECT] Filter out some configurations in Session
Creation
### What changes were proposed in this pull request?
https://github.com/apache/spark/pull/42694 filtered out static
configurations in local mode
This filter out some configurations in both local mode and non-local mode,
since the configurations are actually set after session creation, so
configurations like `spark.remote` always take no effect.
### Why are the changes needed?
avoid unnecessary RPCs and warnings
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
CI
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #42741 from zhengruifeng/filter_out_some_config.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/sql/connect/session.py | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/python/pyspark/sql/connect/session.py
b/python/pyspark/sql/connect/session.py
index c326f94d80c..934386ce954 100644
--- a/python/pyspark/sql/connect/session.py
+++ b/python/pyspark/sql/connect/session.py
@@ -180,10 +180,16 @@ class SparkSession:
def _apply_options(self, session: "SparkSession") -> None:
with self._lock:
for k, v in self._options.items():
- try:
- session.conf.set(k, v)
- except Exception as e:
- warnings.warn(str(e))
+ # the options are applied after session creation,
+ # so following options always take no effect
+ if k not in [
+ "spark.remote",
+ "spark.master",
+ ]:
+ try:
+ session.conf.set(k, v)
+ except Exception as e:
+ warnings.warn(str(e))
def create(self) -> "SparkSession":
has_channel_builder = self._channel_builder is not None
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
