This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 0993315b [SPARK-61161][CONNECT][PYTHON] Export the deprecated 
functions of pyspark.sql.connect.functions, such as countDistinct and 
bitwiseNOT, as PySpark does
0993315b is described below

commit 0993315b818f02a392ad20d170abf8176130cade
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 13:26:29 2026 +0900

    [SPARK-61161][CONNECT][PYTHON] Export the deprecated functions of 
pyspark.sql.connect.functions, such as countDistinct and bitwiseNOT, as PySpark 
does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61161](https://issues.apache.org/jira/browse/SPARK-61161): Export 
the deprecated functions of `pyspark.sql.connect.functions`, such as 
`countDistinct` and `bitwiseNOT`, as PySpark does.
    
    - `pyspark/sql/connect/functions/__init__.py` star-imports 
`pyspark.sql.connect.functions.builtin` instead of `pyspark.sql.functions`, as 
PySpark's package does. The builtin module has no `__all__`, so its public 
names are exported.
    - `pyspark/sql/connect/functions/builtin.py` exposes `py_random`, the 
`random` module imported as in PySpark's builtin module.
    - `pyspark.sql.functions` itself is unchanged.
    
    The public functions of PySpark's `pyspark.sql.connect.functions` that were 
missing are `countDistinct`, `sumDistinct`, `approxCountDistinct`, `chr`, 
`toDegrees`, `toRadians`, `shiftLeft`, `shiftRight`, `shiftRightUnsigned`, 
`bitwiseNOT`, `random`, `uuid` and `py_random`. The Connect tests use them 
through the `CF` alias. After the change, the public names of the package match 
PySpark 4.3.0's, apart from the expression classes and modules that PySpark's 
builtin module imports, which a [...]
    
    ### Why are the changes needed?
    
    ```python
    from pyspark.sql.connect.functions import countDistinct
    # ImportError: cannot import name 'countDistinct' from 
'pyspark.sql.connect.functions'
    ```
    
    PySpark 4.3.0 imports these names from the same package.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `pyspark.sql.connect.functions` exposes the 13 functions above (and 
`py_random`), as PySpark does.
    
    ### How was this patch tested?
    
    - Offline 
`test_connect_functions_package_exports_the_deprecated_functions_as_pyspark`: 
each of the 12 functions of the package is the one of the builtin module, 
`py_random` is the `random` module, and `bitwiseNOT` builds a column. The same 
results with official PySpark 4.3.0.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, and the offline Python suites (2202 passed, 8 skipped), and 
the live tests of functions and columns (45 passed) against a local Spark 4.3.0 
server; rustfmt passes.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #349 from HyukjinKwon/SPARK-61161-connect-functions-exports.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 python/pyspark/sql/connect/functions/__init__.py |  2 +-
 python/pyspark/sql/connect/functions/builtin.py  |  1 +
 python/tests/test_dropin_offline.py              | 17 +++++++++++++++++
 3 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/python/pyspark/sql/connect/functions/__init__.py 
b/python/pyspark/sql/connect/functions/__init__.py
index 896fb504..485494ca 100644
--- a/python/pyspark/sql/connect/functions/__init__.py
+++ b/python/pyspark/sql/connect/functions/__init__.py
@@ -15,6 +15,6 @@
 # limitations under the License.
 #
 """Re-export of ``pyspark.sql.functions`` under the Connect import path."""
-from pyspark.sql.functions import *  # noqa: F401,F403
+from pyspark.sql.connect.functions.builtin import *  # noqa: F401,F403
 from pyspark.sql.functions import col, expr  # noqa: F401
 from pyspark.sql.connect.functions import partitioning  # noqa: F401
diff --git a/python/pyspark/sql/connect/functions/builtin.py 
b/python/pyspark/sql/connect/functions/builtin.py
index 44206284..13a5f6ba 100644
--- a/python/pyspark/sql/connect/functions/builtin.py
+++ b/python/pyspark/sql/connect/functions/builtin.py
@@ -15,6 +15,7 @@
 # limitations under the License.
 #
 """Re-export of ``pyspark.sql.functions`` under the Connect import path."""
+import random as py_random  # noqa: F401
 from typing import TYPE_CHECKING
 
 # Every public name of pyspark.sql.functions.builtin, also the deprecated 
functions that
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 31174616..654a8d84 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -10609,6 +10609,23 @@ def 
test_execution_info_decodes_observed_metrics_and_operation_id(mock_connect):
     assert pairs["dt"] == datetime.timedelta(microseconds=90_061_000_001)
 
 
+def 
test_connect_functions_package_exports_the_deprecated_functions_as_pyspark():
+    # As PySpark Connect's functions package, which star-imports its builtin 
module (which has
+    # no __all__): the functions that pyspark.sql.functions no longer lists 
are attributes of
+    # it too, as the Connect tests use them through their CF alias.
+    import random
+
+    import pyspark.sql.connect.functions as connect_functions
+    from pyspark.sql.connect.functions import builtin
+
+    for name in ("countDistinct", "sumDistinct", "approxCountDistinct", "chr", 
"toDegrees",
+                 "toRadians", "shiftLeft", "shiftRight", "shiftRightUnsigned", 
"bitwiseNOT",
+                 "random", "uuid"):
+        assert getattr(connect_functions, name) is getattr(builtin, name)
+    assert connect_functions.py_random is random
+    assert connect_functions.bitwiseNOT(connect_functions.col("id")) is not 
None
+
+
 def test_execution_info_unsupported_literal_raises_on_flows(mock_connect):
     from pyspark.errors import PySparkTypeError
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to