This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new d1207c5 [SPARK-60742][CONNECT] Build SQL function exports in linear
time
d1207c5 is described below
commit d1207c5517942ccbbc45f859a3563b35e82080ca
Author: DB Tsai <[email protected]>
AuthorDate: Fri Nov 6 17:13:21 2026 +0900
[SPARK-60742][CONNECT] Build SQL function exports in linear time
### What changes were proposed in this pull request?
The loop that builds `pyspark.sql.functions.__all__` at import now checks
membership against a temporary set instead of the growing list, so it is linear
instead of quadratic in the number of exports (about 570). `__all__` keeps its
content and order, and `builtin.__all__` stays a separate list equal to it.
The loop also no longer leaves its variables `name` and `obj` behind as
attributes of the module; PySpark has neither `functions.name` nor
`functions.obj`.
### Why are the changes needed?
[SPARK-60742](https://issues.apache.org/jira/browse/SPARK-60742): the
export loop compared each name with every name exported before it. The cost is
small in absolute terms (milliseconds at import), so this is a cleanup rather
than a measurable speedup.
### Does this PR introduce _any_ user-facing change?
`pyspark.sql.functions.name` and `pyspark.sql.functions.obj`, leftovers of
the export loop, no longer exist, as in PySpark. The exported functions, their
order, star imports and the `builtin` and `partitioning` submodules are
unchanged.
### How was this patch tested?
- A new offline test imports `pyspark.sql.functions` in a fresh process and
checks `__all__` (order, uniqueness, callables, classes), star imports of
`functions` and `functions.builtin`, the `builtin` and `partitioning`
submodules, aliases, signatures, the absence of the loop variables, and the
same after `importlib.reload`.
- Against the current master, merged into this branch, and a local Spark
4.3.0 server: the lib and every integration test of `apache-spark-connect` and
`apache-spark-connect-core`, e2e included (771 passed), the build of all tests
with `wasm-udf`, the offline Python suites (721 passed, 9 skipped), the live
drop-in and Arrow suites (76 passed) and the live cases (10 passed). `cargo fmt
--all --check` passes.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6); the merge with master and the review
follow-ups by Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #142 from dbtsai/codex/spark-60742-linear-function-exports.
Lead-authored-by: DB Tsai <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/sql/functions.py | 9 ++++--
python/tests/test_dropin_offline.py | 59 +++++++++++++++++++++++++++++++++++++
2 files changed, 66 insertions(+), 2 deletions(-)
diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py
index 916d56c..94a5e36 100644
--- a/python/pyspark/sql/functions.py
+++ b/python/pyspark/sql/functions.py
@@ -717,12 +717,17 @@ __all__ = [
# Add all other function names (dynamically)
import inspect
-for name, obj in list(globals().items()):
+_export_globals = list(globals().items())
+_exported_names = set(__all__)
+for name, obj in _export_globals:
if (callable(obj) and
not name.startswith('_') and
- name not in __all__ and
+ name not in _exported_names and
name not in ['inspect', 'importlib', '_gen_module', 'spec', 'os']):
__all__.append(name)
+ _exported_names.add(name)
+# Leave no loop variables behind, as PySpark has no `functions.name` or
`functions.obj`.
+del _export_globals, _exported_names, name, obj
# Expose the `builtin` submodule at ``pyspark.sql.functions.builtin``.
Upstream v4.2.0 splits
# functions into a package whose ``builtin`` submodule holds the actual
functions and whose
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 8ee2728..8421806 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -430,6 +430,65 @@ def test_functions_udtf_and_arrow_decorators():
# ---------------------------------------------------------------------------
functions / columns
+def test_functions_exports_fresh_process():
+ import subprocess
+ import sys
+ import textwrap
+
+ result = subprocess.run([sys.executable, "-c", textwrap.dedent('''
+ import importlib
+ import inspect
+ import sys
+ from pyspark.sql import functions as F
+ assert not hasattr(F, "name") and not hasattr(F, "obj")
+
+ initial = ["col", "lit", "expr", "sum", "count", "avg", "max", "min",
+ "when", "udf", "pandas_udf"]
+
+ def check_exports():
+ assert type(F.__all__) is list
+ assert F.__all__[:len(initial)] == initial
+ assert len(F.__all__) == len(set(F.__all__))
+ assert all(callable(getattr(F, name)) for name in F.__all__)
+ assert not any(name.startswith("_") for name in F.__all__)
+ for name in ("OrderingColumn", "PartitioningColumn",
"SelectedColumn"):
+ assert name in F.__all__
+ assert inspect.isclass(getattr(F, name))
+ namespace = {}
+ exec("from pyspark.sql.functions import *", namespace)
+ assert [name for name in namespace if name != "__builtins__"] ==
F.__all__
+ assert all(namespace[name] is getattr(F, name) for name in
F.__all__)
+ builtin = importlib.import_module("pyspark.sql.functions.builtin")
+ assert builtin is F.builtin is sys.modules[builtin.__name__]
+ assert type(builtin.__all__) is list
+ assert builtin.__all__ == F.__all__
+ assert builtin.__all__ is not F.__all__
+ assert all(getattr(builtin, name) is getattr(F, name) for name in
F.__all__)
+ builtin_namespace = {}
+ exec("from pyspark.sql.functions.builtin import *",
builtin_namespace)
+ assert [name for name in builtin_namespace if name !=
"__builtins__"] == F.__all__
+ assert all(builtin_namespace[name] is getattr(F, name) for name in
F.__all__)
+ part =
importlib.import_module("pyspark.sql.functions.partitioning")
+ assert part is F.partitioning is sys.modules[part.__name__]
+ assert part.__all__ == ["years", "months", "days", "hours",
"bucket"]
+ assert all(getattr(part, name) is getattr(F, name) for name in
part.__all__)
+ assert F.random is F.rand
+ assert F.column is not F.col
+ assert list(inspect.signature(F.first).parameters) == ["col",
"ignorenulls"]
+ assert list(inspect.signature(F.split).parameters) == ["str",
"pattern", "limit"]
+ return F.__all__.copy()
+
+ expected = check_exports()
+ old_all = F.__all__
+ old_all.append("caller_added_name")
+ F.builtin.__all__.append("builtin_only_name")
+ importlib.reload(F)
+ assert F.__all__ is not old_all
+ assert check_exports() == expected
+ ''')], capture_output=True, text=True)
+ assert result.returncode == 0, result.stdout + result.stderr
+
+
def test_functions_build_columns():
from pyspark.sql import functions as F
c = F.col("id")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]