This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 08b93473 [SPARK-61203][CONNECT][PYTHON] Make pyspark.sql.functions an
attribute of pyspark.sql once pyspark.sql is imported, as PySpark does
08b93473 is described below
commit 08b93473df0e73d955445bb7b4a54a28de325f46
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 10:02:42 2026 +0900
[SPARK-61203][CONNECT][PYTHON] Make pyspark.sql.functions an attribute of
pyspark.sql once pyspark.sql is imported, as PySpark does
### What changes were proposed in this pull request?
[SPARK-61203](https://issues.apache.org/jira/browse/SPARK-61203): Make
`pyspark.sql.functions` an attribute of `pyspark.sql` once `pyspark.sql` is
imported, as PySpark does.
- `python/pyspark/sql/__init__.py` imports the `functions` submodule with
the other submodules that it already imports (`conf`, `geo_utils`, `profiler`,
...). After `import pyspark.sql`, `pyspark.sql.functions` is an attribute, and
`import pyspark; pyspark.sql.functions` works too.
- Official PySpark 4.3.0 has the same attribute, because its session,
dataframe and column modules import the functions package at module level. The
sorted set of names of `dir(pyspark.sql)` after `import pyspark.sql` is equal
to PySpark's: `functions` was the only name missing.
- The import cost does not grow. `python -X importtime -c "import
pyspark.sql"` took 145-199 ms cumulative over five runs with the change and
188-238 ms without it; the machine was loaded, so the runs are noisy, and the
change adds no measurable time.
### Why are the changes needed?
`import pyspark.sql` followed by `pyspark.sql.functions.col("a")` raised
`AttributeError: module 'pyspark.sql' has no attribute 'functions'` in the
drop-in, until some code imported `pyspark.sql.functions` explicitly. Code that
uses the attribute, as it does with PySpark, failed.
### Does this PR introduce _any_ user-facing change?
Yes. `pyspark.sql.functions` is an attribute of `pyspark.sql` after `import
pyspark.sql`, as in PySpark.
### How was this patch tested?
- An offline test
`test_sql_functions_is_an_attribute_once_pyspark_sql_is_imported` runs in a new
process, which checks `"functions" in dir(pyspark.sql)`,
`pyspark.sql.functions.__name__`, and that `col("a")` gives `Column<'a'>`. It
fails without this change (AttributeError) and passes with it. Official PySpark
4.3.0 gives the same results for the same script.
- The sorted `dir(pyspark.sql)` after `import pyspark.sql` was compared
with official PySpark 4.3.0: the two lists are equal.
- The lib tests of both crates (618 passed), the build of all tests, also
with `wasm-udf`, and the offline Python suites (2457 passed, 8 skipped);
rustfmt passes, and clippy reports no diagnostics in the changed lines. CI runs
the e2e and live suites.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #389 from
HyukjinKwon/SPARK-61203-r2-make-pyspark-sql-functions-an-attribute.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/pyspark/sql/__init__.py | 1 +
python/tests/test_dropin_offline.py | 18 ++++++++++++++++++
2 files changed, 19 insertions(+)
diff --git a/python/pyspark/sql/__init__.py b/python/pyspark/sql/__init__.py
index 9c11a7a4..815219da 100644
--- a/python/pyspark/sql/__init__.py
+++ b/python/pyspark/sql/__init__.py
@@ -96,6 +96,7 @@ if Geometry is not None:
# attributes of pyspark.sql.
from pyspark.sql import ( # noqa: E402,F401
conf,
+ functions,
geo_utils,
profiler,
sql_formatter,
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index e3008f0c..97706933 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -698,6 +698,24 @@ def test_datatypes_usable_in_sets_and_dicts():
assert len({fa, fa2, fb}) == 2
+def test_sql_functions_is_an_attribute_once_pyspark_sql_is_imported():
+ # As PySpark's, whose modules import pyspark.sql.functions, `functions` is
an attribute of
+ # pyspark.sql after `import pyspark.sql` alone, and of pyspark.sql through
pyspark. A new
+ # process keeps the modules that this test process has imported from
hiding a missing one.
+ import subprocess
+ import sys
+ import textwrap
+
+ result = subprocess.run([sys.executable, "-c", textwrap.dedent("""
+ import pyspark
+ assert "functions" in dir(pyspark.sql), sorted(dir(pyspark.sql))
+ assert pyspark.sql.functions.__name__ == "pyspark.sql.functions"
+ print(repr(pyspark.sql.functions.col("a")))
+ """)], capture_output=True, text=True)
+ assert result.returncode == 0, result.stdout + result.stderr
+ assert result.stdout.strip() == "Column<'a'>", result.stdout
+
+
def
test_data_types_have_their_attributes_type_names_and_singletons_as_pyspark():
# As PySpark's: the attributes of the parameters of the types, typeName as
a classmethod of
# the name of the class, INVALID_TYPENAME_CALL of StructField.typeName,
and the types of
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]