This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 6959bf98 [SPARK-61204][CONNECT][PYTHON] Give None for the className of 
a builtin function without a class in Catalog.listFunctions and getFunction, as 
PySpark does
6959bf98 is described below

commit 6959bf98cf306de6315119c1a58e06a1d04172f5
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 10:03:55 2026 +0900

    [SPARK-61204][CONNECT][PYTHON] Give None for the className of a builtin 
function without a class in Catalog.listFunctions and getFunction, as PySpark 
does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61204](https://issues.apache.org/jira/browse/SPARK-61204): Give None 
for the className of a builtin function without a class in 
`Catalog.listFunctions` and `getFunction`, as PySpark does.
    
    - The server sends a null `className` for the builtins `!=`, `<>`, `case` 
and `||`. The catalog parser read every string column with `row_str`, which 
turns a null into `""`. The `className` column is now read with `row_opt_str`, 
so it is `None` when the server sends null, and the Python `Function.className` 
is `None` for those four builtins.
    - The Rust `spark_connect::catalog::Function::class_name` field changes 
from `String` to `Option<String>`. This is the only public type change.
    - Official PySpark 4.3.0 reads the same columns with `as_py()`, which is 
`None` for a null, and gives `None` for these four builtins. The other nullable 
columns (`locationUri`, `tableType`, `dataType`, the partition names) are never 
null in the server's catalog results, so they keep `row_str`. A probe of 
`listCatalogs`, `listDatabases`, `getDatabase`, `listTables`, `getTable`, 
`listColumns`, `listFunctions` and `getFunction` gives the same fields and the 
same `None`s with the drop-in  [...]
    
    ### Why are the changes needed?
    
    `spark.catalog.listFunctions()` returned `className == ""` for `!=`, `<>`, 
`case` and `||`, where PySpark returns `None`. Code that filters on 
`f.className is None`, or compares the value with PySpark's, gets different 
results.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `Function.className` is `None` (not `""`) for the builtins that have 
no class. The Rust `Function::class_name` is an `Option<String>`.
    
    ### How was this patch tested?
    
    - An offline test 
`test_catalog_function_class_name_is_none_without_a_class_as_pyspark` (the mock 
server answers `listFunctions` with a `!=` row whose className is null, and an 
`abs` row). It fails without this change (`''` for `!=`) and passes with it. 
Official PySpark 4.3.0 gives `None` for the same rows.
    - A live test with the same name, which checks `className is None` for `!=` 
and `||`, and that `getFunction("abs")` gives the same className as 
`listFunctions`. Official PySpark 4.3.0 gives the same results on a Spark 4.3.0 
server.
    - Against a local Spark 4.3.0 server, the live `test_dropin_e2e.py -k 
"catalog or function or table"` tests (33 passed).
    - The lib tests of both crates (618 passed), the build of all tests, also 
with `wasm-udf`, and the offline Python suites (2457 passed, 8 skipped); 
rustfmt passes, and clippy reports no diagnostics in the changed lines. CI runs 
the e2e and live suites.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #390 from 
HyukjinKwon/SPARK-61204-r6-give-none-for-the-classname-of-a-builtin.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/spark-connect/src/catalog.rs |  4 ++--
 python/tests/test_dropin_e2e.py     |  8 ++++++++
 python/tests/test_dropin_offline.py | 21 +++++++++++++++++++++
 3 files changed, 31 insertions(+), 2 deletions(-)

diff --git a/crates/spark-connect/src/catalog.rs 
b/crates/spark-connect/src/catalog.rs
index 3c02f90b..cf9ba5a2 100644
--- a/crates/spark-connect/src/catalog.rs
+++ b/crates/spark-connect/src/catalog.rs
@@ -55,7 +55,7 @@ pub struct Function {
     pub catalog: Option<String>,
     pub namespace: Option<Vec<String>>,
     pub description: Option<String>,
-    pub class_name: String,
+    pub class_name: Option<String>,
     pub is_temporary: bool,
 }
 
@@ -115,7 +115,7 @@ fn parse_function(r: &Row) -> Function {
         catalog: row_opt_str(r, 1),
         namespace: row_opt_namespace(r, 2),
         description: row_opt_str(r, 3),
-        class_name: row_str(r, 4),
+        class_name: row_opt_str(r, 4),
         is_temporary: row_bool(r, 5),
     }
 }
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 18ab1f3f..fc615232 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1043,6 +1043,14 @@ def test_create_dataframe_from_arrow_table(session, 
schema, expected):
     assert _rows(df) == ARROW_TABLE_ROWS
 
 
+def 
test_catalog_function_class_name_is_none_without_a_class_as_pyspark(session):
+    # As PySpark's, the className of a builtin that has no class (`!=`, `||`) 
is None.
+    functions = {f.name: f for f in session.catalog.listFunctions()}
+    assert functions["!="].className is None and functions["||"].className is 
None
+    assert 
functions["abs"].className.startswith("org.apache.spark.sql.catalyst.")
+    assert session.catalog.getFunction("abs").className == 
functions["abs"].className
+
+
 def test_spark_conf_and_task_context_are_those_of_pyspark(session):
     # As PySpark's: pyspark exports the SparkConf, whose settings the builder 
sets on the
     # session, and the TaskContext, which a UDF uses on the server.
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 97706933..11e747e1 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -7906,6 +7906,27 @@ def 
test_create_dataframe_of_a_numpy_array_as_pyspark(mock_connect):
         spark.createDataFrame(np.array([]))
     assert e.value.getCondition() == "CANNOT_INFER_EMPTY_SCHEMA"
 
+
+def 
test_catalog_function_class_name_is_none_without_a_class_as_pyspark(mock_connect):
+    import pyarrow as pa
+
+    spark, service = mock_connect
+    # As PySpark's, a function whose className is null (a builtin such as 
`!=`) has None for
+    # it, not an empty string. The mock answers with the rows of the catalog's 
Function.
+    table = pa.table({
+        "name": ["!=", "abs"], "catalog": [None, None], "namespace": [None, 
None],
+        "description": ["d1", "d2"], "className": [None, "org.a.Abs"],
+        "isTemporary": [True, True]})
+    sink = pa.BufferOutputStream()
+    with pa.ipc.new_stream(sink, table.schema) as writer:
+        writer.write_table(table)
+    # ExecutePlanResponse.arrow_batch=2: ArrowBatch.row_count=1, data=2.
+    service.execute_payload = message_field(2, varint_field(1, table.num_rows) 
+ message_field(
+        2, sink.getvalue().to_pybytes()))
+    functions = spark.catalog.listFunctions()
+    assert [(f.name, f.className) for f in functions] == [("!=", None), 
("abs", "org.a.Abs")]
+
+
 def test_create_dataframe_of_a_dataframe_raises_as_pyspark():
     from pyspark.errors import PySparkTypeError
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to