[spark] branch master updated: [SPARK-45024][PYTHON][CONNECT] Filter out some configurations in Session Creation

[ruifengz]

This is an automated email from the ASF dual-hosted git repository.

ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new ed03173f33a [SPARK-45024][PYTHON][CONNECT] Filter out some 
configurations in Session Creation
ed03173f33a is described below

commit ed03173f33aad0cf8b18096a4fb2470059410751
Author: Ruifeng Zheng <ruife...@apache.org>
AuthorDate: Thu Aug 31 12:36:35 2023 +0800

    [SPARK-45024][PYTHON][CONNECT] Filter out some configurations in Session 
Creation
    
    ### What changes were proposed in this pull request?
    https://github.com/apache/spark/pull/42694 filtered out static 
configurations in local mode
    
    This filter out some configurations in both local mode and non-local mode, 
since the configurations are actually set after session creation, so 
configurations like `spark.remote` always take no effect.
    
    ### Why are the changes needed?
    avoid unnecessary RPCs and warnings
    
    ### Does this PR introduce _any_ user-facing change?
    no
    
    ### How was this patch tested?
    CI
    
    ### Was this patch authored or co-authored using generative AI tooling?
    no
    
    Closes #42741 from zhengruifeng/filter_out_some_config.
    
    Authored-by: Ruifeng Zheng <ruife...@apache.org>
    Signed-off-by: Ruifeng Zheng <ruife...@apache.org>
---
 python/pyspark/sql/connect/session.py | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/python/pyspark/sql/connect/session.py 
b/python/pyspark/sql/connect/session.py
index c326f94d80c..934386ce954 100644
--- a/python/pyspark/sql/connect/session.py
+++ b/python/pyspark/sql/connect/session.py
@@ -180,10 +180,16 @@ class SparkSession:
         def _apply_options(self, session: "SparkSession") -> None:
             with self._lock:
                 for k, v in self._options.items():
-                    try:
-                        session.conf.set(k, v)
-                    except Exception as e:
-                        warnings.warn(str(e))
+                    # the options are applied after session creation,
+                    # so following options always take no effect
+                    if k not in [
+                        "spark.remote",
+                        "spark.master",
+                    ]:
+                        try:
+                            session.conf.set(k, v)
+                        except Exception as e:
+                            warnings.warn(str(e))
 
         def create(self) -> "SparkSession":
             has_channel_builder = self._channel_builder is not None


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org