This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 08b93473 [SPARK-61203][CONNECT][PYTHON] Make pyspark.sql.functions an 
attribute of pyspark.sql once pyspark.sql is imported, as PySpark does
08b93473 is described below

commit 08b93473df0e73d955445bb7b4a54a28de325f46
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 10:02:42 2026 +0900

    [SPARK-61203][CONNECT][PYTHON] Make pyspark.sql.functions an attribute of 
pyspark.sql once pyspark.sql is imported, as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61203](https://issues.apache.org/jira/browse/SPARK-61203): Make 
`pyspark.sql.functions` an attribute of `pyspark.sql` once `pyspark.sql` is 
imported, as PySpark does.
    
    - `python/pyspark/sql/__init__.py` imports the `functions` submodule with 
the other submodules that it already imports (`conf`, `geo_utils`, `profiler`, 
...). After `import pyspark.sql`, `pyspark.sql.functions` is an attribute, and 
`import pyspark; pyspark.sql.functions` works too.
    - Official PySpark 4.3.0 has the same attribute, because its session, 
dataframe and column modules import the functions package at module level. The 
sorted set of names of `dir(pyspark.sql)` after `import pyspark.sql` is equal 
to PySpark's: `functions` was the only name missing.
    - The import cost does not grow. `python -X importtime -c "import 
pyspark.sql"` took 145-199 ms cumulative over five runs with the change and 
188-238 ms without it; the machine was loaded, so the runs are noisy, and the 
change adds no measurable time.
    
    ### Why are the changes needed?
    
    `import pyspark.sql` followed by `pyspark.sql.functions.col("a")` raised 
`AttributeError: module 'pyspark.sql' has no attribute 'functions'` in the 
drop-in, until some code imported `pyspark.sql.functions` explicitly. Code that 
uses the attribute, as it does with PySpark, failed.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `pyspark.sql.functions` is an attribute of `pyspark.sql` after `import 
pyspark.sql`, as in PySpark.
    
    ### How was this patch tested?
    
    - An offline test 
`test_sql_functions_is_an_attribute_once_pyspark_sql_is_imported` runs in a new 
process, which checks `"functions" in dir(pyspark.sql)`, 
`pyspark.sql.functions.__name__`, and that `col("a")` gives `Column<'a'>`. It 
fails without this change (AttributeError) and passes with it. Official PySpark 
4.3.0 gives the same results for the same script.
    - The sorted `dir(pyspark.sql)` after `import pyspark.sql` was compared 
with official PySpark 4.3.0: the two lists are equal.
    - The lib tests of both crates (618 passed), the build of all tests, also 
with `wasm-udf`, and the offline Python suites (2457 passed, 8 skipped); 
rustfmt passes, and clippy reports no diagnostics in the changed lines. CI runs 
the e2e and live suites.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #389 from 
HyukjinKwon/SPARK-61203-r2-make-pyspark-sql-functions-an-attribute.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 python/pyspark/sql/__init__.py      |  1 +
 python/tests/test_dropin_offline.py | 18 ++++++++++++++++++
 2 files changed, 19 insertions(+)

diff --git a/python/pyspark/sql/__init__.py b/python/pyspark/sql/__init__.py
index 9c11a7a4..815219da 100644
--- a/python/pyspark/sql/__init__.py
+++ b/python/pyspark/sql/__init__.py
@@ -96,6 +96,7 @@ if Geometry is not None:
 # attributes of pyspark.sql.
 from pyspark.sql import (  # noqa: E402,F401
     conf,
+    functions,
     geo_utils,
     profiler,
     sql_formatter,
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index e3008f0c..97706933 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -698,6 +698,24 @@ def test_datatypes_usable_in_sets_and_dicts():
     assert len({fa, fa2, fb}) == 2
 
 
+def test_sql_functions_is_an_attribute_once_pyspark_sql_is_imported():
+    # As PySpark's, whose modules import pyspark.sql.functions, `functions` is 
an attribute of
+    # pyspark.sql after `import pyspark.sql` alone, and of pyspark.sql through 
pyspark. A new
+    # process keeps the modules that this test process has imported from 
hiding a missing one.
+    import subprocess
+    import sys
+    import textwrap
+
+    result = subprocess.run([sys.executable, "-c", textwrap.dedent("""
+        import pyspark
+        assert "functions" in dir(pyspark.sql), sorted(dir(pyspark.sql))
+        assert pyspark.sql.functions.__name__ == "pyspark.sql.functions"
+        print(repr(pyspark.sql.functions.col("a")))
+    """)], capture_output=True, text=True)
+    assert result.returncode == 0, result.stdout + result.stderr
+    assert result.stdout.strip() == "Column<'a'>", result.stdout
+
+
 def 
test_data_types_have_their_attributes_type_names_and_singletons_as_pyspark():
     # As PySpark's: the attributes of the parameters of the types, typeName as 
a classmethod of
     # the name of the class, INVALID_TYPENAME_CALL of StructField.typeName, 
and the types of


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to