[spark] branch master updated: [SPARK-41928][CONNECT][PYTHON] Add the unsupported list for `functions`

[gurwls223]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 6d986da1c9c [SPARK-41928][CONNECT][PYTHON] Add the unsupported list 
for `functions`
6d986da1c9c is described below

commit 6d986da1c9ca60cbefe72d3cc2d0fac4daca2f45
Author: Ruifeng Zheng <ruife...@apache.org>
AuthorDate: Sat Jan 7 13:51:16 2023 +0900

    [SPARK-41928][CONNECT][PYTHON] Add the unsupported list for `functions`
    
    ### What changes were proposed in this pull request?
    Add the unsupported list for `functions` module
    
    ### Why are the changes needed?
    to explictly tell users they are not implemented
    
    ### Does this PR introduce _any_ user-facing change?
    yes
    
    ### How was this patch tested?
    added UT
    
    Closes #39438 from zhengruifeng/connect_function_unsupported.
    
    Authored-by: Ruifeng Zheng <ruife...@apache.org>
    Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
 python/pyspark/sql/connect/functions.py                   |  8 ++++++++
 python/pyspark/sql/tests/connect/test_connect_function.py | 12 ++++++++++++
 2 files changed, 20 insertions(+)

diff --git a/python/pyspark/sql/connect/functions.py 
b/python/pyspark/sql/connect/functions.py
index aa760c1627b..356855581bd 100644
--- a/python/pyspark/sql/connect/functions.py
+++ b/python/pyspark/sql/connect/functions.py
@@ -2333,6 +2333,14 @@ def unwrap_udt(col: "ColumnOrName") -> Column:
 unwrap_udt.__doc__ = pysparkfuncs.unwrap_udt.__doc__
 
 
+def udf(*args: Any, **kwargs: Any) -> None:
+    raise NotImplementedError("udf() is not implemented.")
+
+
+def pandas_udf(*args: Any, **kwargs: Any) -> None:
+    raise NotImplementedError("pandas_udf() is not implemented.")
+
+
 def _test() -> None:
     import os
     import sys
diff --git a/python/pyspark/sql/tests/connect/test_connect_function.py 
b/python/pyspark/sql/tests/connect/test_connect_function.py
index bde5ca26aa8..2ae2eb556ad 100644
--- a/python/pyspark/sql/tests/connect/test_connect_function.py
+++ b/python/pyspark/sql/tests/connect/test_connect_function.py
@@ -2155,6 +2155,18 @@ class 
SparkConnectFunctionTests(SparkConnectFuncTestCase):
             ).toPandas(),
         )
 
+    def test_unsupported_functions(self):
+        # SPARK-41928: Disable unsupported functions.
+
+        from pyspark.sql.connect import functions as CF
+
+        for f in (
+            "udf",
+            "pandas_udf",
+        ):
+            with self.assertRaises(NotImplementedError):
+                getattr(CF, f)()
+
 
 if __name__ == "__main__":
     import os


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org