This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new d1207c5  [SPARK-60742][CONNECT] Build SQL function exports in linear 
time
d1207c5 is described below

commit d1207c5517942ccbbc45f859a3563b35e82080ca
Author: DB Tsai <[email protected]>
AuthorDate: Fri Nov 6 17:13:21 2026 +0900

    [SPARK-60742][CONNECT] Build SQL function exports in linear time
    
    ### What changes were proposed in this pull request?
    
    The loop that builds `pyspark.sql.functions.__all__` at import now checks 
membership against a temporary set instead of the growing list, so it is linear 
instead of quadratic in the number of exports (about 570). `__all__` keeps its 
content and order, and `builtin.__all__` stays a separate list equal to it.
    
    The loop also no longer leaves its variables `name` and `obj` behind as 
attributes of the module; PySpark has neither `functions.name` nor 
`functions.obj`.
    
    ### Why are the changes needed?
    
    [SPARK-60742](https://issues.apache.org/jira/browse/SPARK-60742): the 
export loop compared each name with every name exported before it. The cost is 
small in absolute terms (milliseconds at import), so this is a cleanup rather 
than a measurable speedup.
    
    ### Does this PR introduce _any_ user-facing change?
    
    `pyspark.sql.functions.name` and `pyspark.sql.functions.obj`, leftovers of 
the export loop, no longer exist, as in PySpark. The exported functions, their 
order, star imports and the `builtin` and `partitioning` submodules are 
unchanged.
    
    ### How was this patch tested?
    
    - A new offline test imports `pyspark.sql.functions` in a fresh process and 
checks `__all__` (order, uniqueness, callables, classes), star imports of 
`functions` and `functions.builtin`, the `builtin` and `partitioning` 
submodules, aliases, signatures, the absence of the loop variables, and the 
same after `importlib.reload`.
    - Against the current master, merged into this branch, and a local Spark 
4.3.0 server: the lib and every integration test of `apache-spark-connect` and 
`apache-spark-connect-core`, e2e included (771 passed), the build of all tests 
with `wasm-udf`, the offline Python suites (721 passed, 9 skipped), the live 
drop-in and Arrow suites (76 passed) and the live cases (10 passed). `cargo fmt 
--all --check` passes.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6); the merge with master and the review 
follow-ups by Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #142 from dbtsai/codex/spark-60742-linear-function-exports.
    
    Lead-authored-by: DB Tsai <[email protected]>
    Co-authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 python/pyspark/sql/functions.py     |  9 ++++--
 python/tests/test_dropin_offline.py | 59 +++++++++++++++++++++++++++++++++++++
 2 files changed, 66 insertions(+), 2 deletions(-)

diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py
index 916d56c..94a5e36 100644
--- a/python/pyspark/sql/functions.py
+++ b/python/pyspark/sql/functions.py
@@ -717,12 +717,17 @@ __all__ = [
 
 # Add all other function names (dynamically)
 import inspect
-for name, obj in list(globals().items()):
+_export_globals = list(globals().items())
+_exported_names = set(__all__)
+for name, obj in _export_globals:
     if (callable(obj) and
         not name.startswith('_') and
-        name not in __all__ and
+        name not in _exported_names and
         name not in ['inspect', 'importlib', '_gen_module', 'spec', 'os']):
         __all__.append(name)
+        _exported_names.add(name)
+# Leave no loop variables behind, as PySpark has no `functions.name` or 
`functions.obj`.
+del _export_globals, _exported_names, name, obj
 
 # Expose the `builtin` submodule at ``pyspark.sql.functions.builtin``. 
Upstream v4.2.0 splits
 # functions into a package whose ``builtin`` submodule holds the actual 
functions and whose
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 8ee2728..8421806 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -430,6 +430,65 @@ def test_functions_udtf_and_arrow_decorators():
 
 # --------------------------------------------------------------------------- 
functions / columns
 
+def test_functions_exports_fresh_process():
+    import subprocess
+    import sys
+    import textwrap
+
+    result = subprocess.run([sys.executable, "-c", textwrap.dedent('''
+        import importlib
+        import inspect
+        import sys
+        from pyspark.sql import functions as F
+        assert not hasattr(F, "name") and not hasattr(F, "obj")
+
+        initial = ["col", "lit", "expr", "sum", "count", "avg", "max", "min",
+                   "when", "udf", "pandas_udf"]
+
+        def check_exports():
+            assert type(F.__all__) is list
+            assert F.__all__[:len(initial)] == initial
+            assert len(F.__all__) == len(set(F.__all__))
+            assert all(callable(getattr(F, name)) for name in F.__all__)
+            assert not any(name.startswith("_") for name in F.__all__)
+            for name in ("OrderingColumn", "PartitioningColumn", 
"SelectedColumn"):
+                assert name in F.__all__
+                assert inspect.isclass(getattr(F, name))
+            namespace = {}
+            exec("from pyspark.sql.functions import *", namespace)
+            assert [name for name in namespace if name != "__builtins__"] == 
F.__all__
+            assert all(namespace[name] is getattr(F, name) for name in 
F.__all__)
+            builtin = importlib.import_module("pyspark.sql.functions.builtin")
+            assert builtin is F.builtin is sys.modules[builtin.__name__]
+            assert type(builtin.__all__) is list
+            assert builtin.__all__ == F.__all__
+            assert builtin.__all__ is not F.__all__
+            assert all(getattr(builtin, name) is getattr(F, name) for name in 
F.__all__)
+            builtin_namespace = {}
+            exec("from pyspark.sql.functions.builtin import *", 
builtin_namespace)
+            assert [name for name in builtin_namespace if name != 
"__builtins__"] == F.__all__
+            assert all(builtin_namespace[name] is getattr(F, name) for name in 
F.__all__)
+            part = 
importlib.import_module("pyspark.sql.functions.partitioning")
+            assert part is F.partitioning is sys.modules[part.__name__]
+            assert part.__all__ == ["years", "months", "days", "hours", 
"bucket"]
+            assert all(getattr(part, name) is getattr(F, name) for name in 
part.__all__)
+            assert F.random is F.rand
+            assert F.column is not F.col
+            assert list(inspect.signature(F.first).parameters) == ["col", 
"ignorenulls"]
+            assert list(inspect.signature(F.split).parameters) == ["str", 
"pattern", "limit"]
+            return F.__all__.copy()
+
+        expected = check_exports()
+        old_all = F.__all__
+        old_all.append("caller_added_name")
+        F.builtin.__all__.append("builtin_only_name")
+        importlib.reload(F)
+        assert F.__all__ is not old_all
+        assert check_exports() == expected
+    ''')], capture_output=True, text=True)
+    assert result.returncode == 0, result.stdout + result.stderr
+
+
 def test_functions_build_columns():
     from pyspark.sql import functions as F
     c = F.col("id")


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to