This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 6959bf98 [SPARK-61204][CONNECT][PYTHON] Give None for the className of
a builtin function without a class in Catalog.listFunctions and getFunction, as
PySpark does
6959bf98 is described below
commit 6959bf98cf306de6315119c1a58e06a1d04172f5
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 10:03:55 2026 +0900
[SPARK-61204][CONNECT][PYTHON] Give None for the className of a builtin
function without a class in Catalog.listFunctions and getFunction, as PySpark
does
### What changes were proposed in this pull request?
[SPARK-61204](https://issues.apache.org/jira/browse/SPARK-61204): Give None
for the className of a builtin function without a class in
`Catalog.listFunctions` and `getFunction`, as PySpark does.
- The server sends a null `className` for the builtins `!=`, `<>`, `case`
and `||`. The catalog parser read every string column with `row_str`, which
turns a null into `""`. The `className` column is now read with `row_opt_str`,
so it is `None` when the server sends null, and the Python `Function.className`
is `None` for those four builtins.
- The Rust `spark_connect::catalog::Function::class_name` field changes
from `String` to `Option<String>`. This is the only public type change.
- Official PySpark 4.3.0 reads the same columns with `as_py()`, which is
`None` for a null, and gives `None` for these four builtins. The other nullable
columns (`locationUri`, `tableType`, `dataType`, the partition names) are never
null in the server's catalog results, so they keep `row_str`. A probe of
`listCatalogs`, `listDatabases`, `getDatabase`, `listTables`, `getTable`,
`listColumns`, `listFunctions` and `getFunction` gives the same fields and the
same `None`s with the drop-in [...]
### Why are the changes needed?
`spark.catalog.listFunctions()` returned `className == ""` for `!=`, `<>`,
`case` and `||`, where PySpark returns `None`. Code that filters on
`f.className is None`, or compares the value with PySpark's, gets different
results.
### Does this PR introduce _any_ user-facing change?
Yes. `Function.className` is `None` (not `""`) for the builtins that have
no class. The Rust `Function::class_name` is an `Option<String>`.
### How was this patch tested?
- An offline test
`test_catalog_function_class_name_is_none_without_a_class_as_pyspark` (the mock
server answers `listFunctions` with a `!=` row whose className is null, and an
`abs` row). It fails without this change (`''` for `!=`) and passes with it.
Official PySpark 4.3.0 gives `None` for the same rows.
- A live test with the same name, which checks `className is None` for `!=`
and `||`, and that `getFunction("abs")` gives the same className as
`listFunctions`. Official PySpark 4.3.0 gives the same results on a Spark 4.3.0
server.
- Against a local Spark 4.3.0 server, the live `test_dropin_e2e.py -k
"catalog or function or table"` tests (33 passed).
- The lib tests of both crates (618 passed), the build of all tests, also
with `wasm-udf`, and the offline Python suites (2457 passed, 8 skipped);
rustfmt passes, and clippy reports no diagnostics in the changed lines. CI runs
the e2e and live suites.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #390 from
HyukjinKwon/SPARK-61204-r6-give-none-for-the-classname-of-a-builtin.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/spark-connect/src/catalog.rs | 4 ++--
python/tests/test_dropin_e2e.py | 8 ++++++++
python/tests/test_dropin_offline.py | 21 +++++++++++++++++++++
3 files changed, 31 insertions(+), 2 deletions(-)
diff --git a/crates/spark-connect/src/catalog.rs
b/crates/spark-connect/src/catalog.rs
index 3c02f90b..cf9ba5a2 100644
--- a/crates/spark-connect/src/catalog.rs
+++ b/crates/spark-connect/src/catalog.rs
@@ -55,7 +55,7 @@ pub struct Function {
pub catalog: Option<String>,
pub namespace: Option<Vec<String>>,
pub description: Option<String>,
- pub class_name: String,
+ pub class_name: Option<String>,
pub is_temporary: bool,
}
@@ -115,7 +115,7 @@ fn parse_function(r: &Row) -> Function {
catalog: row_opt_str(r, 1),
namespace: row_opt_namespace(r, 2),
description: row_opt_str(r, 3),
- class_name: row_str(r, 4),
+ class_name: row_opt_str(r, 4),
is_temporary: row_bool(r, 5),
}
}
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 18ab1f3f..fc615232 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1043,6 +1043,14 @@ def test_create_dataframe_from_arrow_table(session,
schema, expected):
assert _rows(df) == ARROW_TABLE_ROWS
+def
test_catalog_function_class_name_is_none_without_a_class_as_pyspark(session):
+ # As PySpark's, the className of a builtin that has no class (`!=`, `||`)
is None.
+ functions = {f.name: f for f in session.catalog.listFunctions()}
+ assert functions["!="].className is None and functions["||"].className is
None
+ assert
functions["abs"].className.startswith("org.apache.spark.sql.catalyst.")
+ assert session.catalog.getFunction("abs").className ==
functions["abs"].className
+
+
def test_spark_conf_and_task_context_are_those_of_pyspark(session):
# As PySpark's: pyspark exports the SparkConf, whose settings the builder
sets on the
# session, and the TaskContext, which a UDF uses on the server.
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 97706933..11e747e1 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -7906,6 +7906,27 @@ def
test_create_dataframe_of_a_numpy_array_as_pyspark(mock_connect):
spark.createDataFrame(np.array([]))
assert e.value.getCondition() == "CANNOT_INFER_EMPTY_SCHEMA"
+
+def
test_catalog_function_class_name_is_none_without_a_class_as_pyspark(mock_connect):
+ import pyarrow as pa
+
+ spark, service = mock_connect
+ # As PySpark's, a function whose className is null (a builtin such as
`!=`) has None for
+ # it, not an empty string. The mock answers with the rows of the catalog's
Function.
+ table = pa.table({
+ "name": ["!=", "abs"], "catalog": [None, None], "namespace": [None,
None],
+ "description": ["d1", "d2"], "className": [None, "org.a.Abs"],
+ "isTemporary": [True, True]})
+ sink = pa.BufferOutputStream()
+ with pa.ipc.new_stream(sink, table.schema) as writer:
+ writer.write_table(table)
+ # ExecutePlanResponse.arrow_batch=2: ArrowBatch.row_count=1, data=2.
+ service.execute_payload = message_field(2, varint_field(1, table.num_rows)
+ message_field(
+ 2, sink.getvalue().to_pybytes()))
+ functions = spark.catalog.listFunctions()
+ assert [(f.name, f.className) for f in functions] == [("!=", None),
("abs", "org.a.Abs")]
+
+
def test_create_dataframe_of_a_dataframe_raises_as_pyspark():
from pyspark.errors import PySparkTypeError
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]