This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 93f62eb [SPARK-60995][CONNECT][PYTHON] Check the arguments of when,
the partition transforms and the UDF decorators as PySpark does
93f62eb is described below
commit 93f62eb8cb1323a78d5d2a95af9104c680d4a7d5
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 18:04:19 2026 +0900
[SPARK-60995][CONNECT][PYTHON] Check the arguments of when, the partition
transforms and the UDF decorators as PySpark does
### What changes were proposed in this pull request?
[SPARK-60995](https://issues.apache.org/jira/browse/SPARK-60995): these
functions now check their arguments before any request, with PySpark's errors.
- **`when`**: `when()` and `Column.when` raise `NOT_EXPECTED_TYPE` for a
condition that is not a Column. `Column.when` raises `INVALID_WHEN_USAGE` for a
column that is not a `when()` without `otherwise()`. `Column.otherwise` raises
`PySparkTypeError` for a column that is not a `when()`, or one that already has
an `otherwise()`.
- **Partition transforms**: `pyspark.sql.functions.partitioning` holds
PySpark Connect's partition transforms. Its `bucket` raises `NOT_EXPECTED_TYPE`
for a `numBuckets` that is neither an int nor a Column. `bucket`, `years`,
`months`, `days` and `hours` of `pyspark.sql.functions` warn that they are
deprecated, and call those of `partitioning`, as in PySpark.
- **`pandas_udf` and `arrow_udf`** are PySpark's.
`pyspark/sql/pandas/functions.py` is vendored from Apache Spark 4.3.0 and
checked by the drift check. It creates the UDF with the new
`pyspark.sql.connect.udf._create_udf`. So:
- they raise `CANNOT_BE_NONE` without a return type;
- they raise `INVALID_PANDAS_UDF_TYPE` for a function type that is not a
value of `PandasUDFType` or `ArrowUDFType`;
- they warn as PySpark's do.
`UserDefinedFunction` checks its function, return type and eval type, and
the return type for the eval type (`_check_return_type`, verbatim), as PySpark
Connect's does.
- **`udtf` and `arrow_udtf`** check their handler classes as PySpark does
(`_validate_udtf_handler` and `_validate_arrow_udtf_handler`, verbatim):
`INVALID_UDTF_HANDLER_TYPE`, `INVALID_UDTF_NO_EVAL`,
`INVALID_UDTF_RETURN_TYPE`, `INVALID_UDTF_BOTH_RETURN_TYPE_AND_ANALYZE` and
`INVALID_ARROW_UDTF_WITH_ANALYZE`.
### Why are the changes needed?
Before, these calls built other expressions, or UDFs, without an error:
- `F.when("a", 1)` was `CASE WHEN a THEN 1 END`.
- `F.col("a").when(cond, 1)` started a new CASE WHEN, dropping `a`.
- `F.col("a").otherwise(0)` returned `a`.
- `F.bucket("n", "c")` took `n` as a column name.
- `pandas_udf` took `"scalar"` or `"grouped_map"` as a function type, which
PySpark does not. It ignored a `PandasUDFType`, such as `GROUPED_MAP`, and
inferred the eval type instead. Without a return type, it defaulted to
`StringType`.
- `udtf(3)` and `arrow_udtf(3)` returned a UDTF.
### Does this PR introduce _any_ user-facing change?
Yes. These functions raise the errors of PySpark 4.3.0 for these arguments.
`pandas_udf` and `arrow_udf` take PySpark's function types, and no longer
`"scalar"` and the other strings. `PandasUDFType` no longer has the `MAP_ITER`
and `COGROUPED_MAP` that PySpark does not have.
### How was this patch tested?
- A probe of 34 calls of these functions, valid and invalid, gives the same
results and warnings as official PySpark 4.3.0. Before this change, 26 of them
differed. One call differs only in that official PySpark needs a session to
parse the DDL return type of a pandas UDF, which the probe does not have.
- New offline tests check the errors of 18 of these calls, the reprs of
`when` and `otherwise`, and the warnings of the partition transforms. They also
pass with official PySpark 4.3.0, and fail before this change.
- The offline tests of `pandas_udf` and `arrow_udf` now pass PySpark's
function types and return types.
- The lib tests and every integration test, e2e included, against a local
Spark 4.3.0 server (652 passed, and 123 of the transport crate), the build of
all tests with `wasm-udf`, the offline Python suites (818 passed, 9 skipped),
and the live drop-in and Arrow suites (79 passed) and live cases (10 passed).
Clippy reports no diagnostics in the changed lines.
The eval type of `arrow_udtf`, with which every `arrow_udtf` fails, is
[SPARK-60999](https://issues.apache.org/jira/browse/SPARK-60999).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #198 from HyukjinKwon/SPARK-60995-argument-checks.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/column.rs | 51 +-
python/pyspark/sql/connect/udf.py | 6 +-
python/pyspark/sql/functions.py | 113 +++-
python/pyspark/sql/functions_generated.py | 32 --
python/pyspark/sql/pandas/functions.py | 917 +++++++++++++++++++++++++++++-
python/pyspark/sql/udf.py | 347 +++++++----
python/pyspark/sql/udtf.py | 51 ++
python/tests/test_dropin_offline.py | 137 ++++-
scripts/check_vendored_upstream.sh | 2 +
scripts/gen_fn_dispatch.py | 8 +
10 files changed, 1463 insertions(+), 201 deletions(-)
diff --git a/crates/pyspark-rs/src/column.rs b/crates/pyspark-rs/src/column.rs
index a4d1614..d60c0d6 100644
--- a/crates/pyspark-rs/src/column.rs
+++ b/crates/pyspark-rs/src/column.rs
@@ -295,15 +295,58 @@ impl PyColumn {
PyColumn::new(self.column.clone().desc_nulls_last())
}
- /// When condition for CASE statements.
+ /// A branch more of a CASE WHEN. Mirrors `Column.when`: the condition is
a Column, and
+ /// the column a `when()` without `otherwise()` (`INVALID_WHEN_USAGE`).
fn when(&self, condition: &Bound<'_, PyAny>, value: &Bound<'_, PyAny>) ->
PyResult<PyColumn> {
- let cond_col = py_obj_to_column(condition)?;
+ let py = condition.py();
+ let Ok(condition) = condition.extract::<PyRef<'_, PyColumn>>() else {
+ let arg_type = condition.get_type().name()?.to_string();
+ return Err(raise_pyspark(
+ py,
+ "PySparkTypeError",
+ "NOT_EXPECTED_TYPE",
+ &[
+ ("expected_type", "Column"),
+ ("arg_name", "condition"),
+ ("arg_type", &arg_type),
+ ],
+ ));
+ };
+ match self.column.expression().node() {
+ ExpressionNode::CaseWhen(case_when) if
case_when.else_expr.is_none() => {}
+ _ => {
+ return Err(raise_pyspark(
+ py,
+ "PySparkTypeError",
+ "INVALID_WHEN_USAGE",
+ &[],
+ ))
+ }
+ }
let val_col = py_obj_to_column(value)?;
- Ok(PyColumn::new(self.column.clone().when(cond_col, val_col)))
+ Ok(PyColumn::new(
+ self.column.clone().when(condition.column.clone(), val_col),
+ ))
}
- /// Otherwise for CASE statements.
+ /// The value of a CASE WHEN when no branch holds. Mirrors
`Column.otherwise`, which
+ /// raises PySparkTypeError for a column that is not a `when()`, or one
that has one.
fn otherwise(&self, value: &Bound<'_, PyAny>) -> PyResult<PyColumn> {
+ let message = match self.column.expression().node() {
+ ExpressionNode::CaseWhen(case_when) if
case_when.else_expr.is_none() => None,
+ ExpressionNode::CaseWhen(_) => Some(
+ "otherwise() can only be applied once on a Column previously
generated by when()",
+ ),
+ _ => Some("otherwise() can only be applied on a Column previously
generated by when()"),
+ };
+ if let Some(message) = message {
+ let py = value.py();
+ let error = py
+ .import("pyspark.errors")?
+ .getattr("PySparkTypeError")?
+ .call1((message,))?;
+ return Err(PyErr::from_value(error));
+ }
let val_col = py_obj_to_column(value)?;
Ok(PyColumn::new(self.column.clone().otherwise(val_col)))
}
diff --git a/python/pyspark/sql/connect/udf.py
b/python/pyspark/sql/connect/udf.py
index 3b7daaa..ea4c9bb 100644
--- a/python/pyspark/sql/connect/udf.py
+++ b/python/pyspark/sql/connect/udf.py
@@ -14,7 +14,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-"""UserDefinedFunction / UDFRegistration under the connect import path."""
-from pyspark.sql.udf import UDFRegistration, UserDefinedFunction
+"""UserDefinedFunction / UDFRegistration under the connect import path, and the
+``_create_udf`` with which ``pyspark.sql.pandas.functions`` creates pandas and
Arrow UDFs, as
+in PySpark Connect."""
+from pyspark.sql.udf import UDFRegistration, UserDefinedFunction, _create_udf
# noqa: F401
__all__ = ["UDFRegistration", "UserDefinedFunction"]
diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py
index 94a5e36..8933608 100644
--- a/python/pyspark/sql/functions.py
+++ b/python/pyspark/sql/functions.py
@@ -305,7 +305,8 @@ def _dict_to_options_column(options_dict):
return options_dict
# Import UDF / UDTF functions
-from pyspark.sql.udf import udf, pandas_udf, arrow_udf
+from pyspark.sql.udf import udf
+from pyspark.sql.pandas.functions import arrow_udf, pandas_udf
from pyspark.sql.udtf import (
udtf,
arrow_udtf,
@@ -361,7 +362,30 @@ def expr(str):
"""Parses the expression string into the column that it represents."""
return _pyfunc_expr(str)
count = _col_wrapper(_pyfunc_count)
-when = _pyfunc_when
+
+
+def when(condition, value):
+ """when(condition, value): a CASE WHEN of one branch.
+
+ Mirrors pyspark.sql.connect.functions.when: the condition is a Column, and
the value a
+ Column or the literal of a value.
+ """
+ if not isinstance(condition, _pyspark.Column):
+ from pyspark.errors import PySparkTypeError
+
+ raise PySparkTypeError(
+ errorClass="NOT_EXPECTED_TYPE",
+ messageParameters={
+ "expected_type": "Column",
+ "arg_name": "condition",
+ "arg_type": type(condition).__name__,
+ },
+ )
+ from pyspark.sql.utils import enum_to_value
+
+ value = enum_to_value(value)
+ return _pyfunc_when(condition, value if isinstance(value, _pyspark.Column)
else lit(value))
+
# Mixed/special functions - explicit wrappers for non-generic dispatch
def from_avro(data, json_format_schema):
@@ -665,21 +689,86 @@ def broadcast(df):
# Expose the `partitioning` submodule (bucket/days/hours/months/years) at
# ``pyspark.sql.functions.partitioning``. Since this module is a plain module
(not a
-# package), register a synthetic submodule holding the flat partition-transform
-# functions so both ``import pyspark.sql.functions.partitioning`` and
-# ``F.partitioning.bucket`` work, mirroring reference pyspark.
+# package), register a synthetic submodule holding the partition transforms,
as PySpark
+# Connect's, so both ``import pyspark.sql.functions.partitioning`` and
+# ``F.partitioning.bucket`` work. The functions of the same names here are
deprecated, as
+# in PySpark: they warn and call those of the submodule.
import sys as _sys
import types as _types
+import warnings as _warnings
+
+
+def _partition_transforms():
+ _over_columns = _gen_module._invoke_function_over_columns
+
+ def years(col):
+ return _over_columns("years", col)
+
+ def months(col):
+ return _over_columns("months", col)
+
+ def days(col):
+ return _over_columns("days", col)
+
+ def hours(col):
+ return _over_columns("hours", col)
+
+ def bucket(numBuckets, col):
+ if isinstance(numBuckets, int):
+ _numBuckets = lit(numBuckets)
+ elif isinstance(numBuckets, _pyspark.Column):
+ _numBuckets = numBuckets
+ else:
+ from pyspark.errors import PySparkTypeError
+
+ raise PySparkTypeError(
+ errorClass="NOT_EXPECTED_TYPE",
+ messageParameters={
+ "expected_type": "Column or int",
+ "arg_name": "numBuckets",
+ "arg_type": type(numBuckets).__name__,
+ },
+ )
+ return _invoke_function("bucket", _numBuckets, _column_or_name(col))
+
+ return years, months, days, hours, bucket
+
partitioning = _types.ModuleType("pyspark.sql.functions.partitioning")
partitioning.__doc__ = "Partition transform functions
(bucket/days/hours/months/years)."
-for _pname in ("years", "months", "days", "hours", "bucket"):
- _obj = globals().get(_pname)
- if _obj is not None:
- setattr(partitioning, _pname, _obj)
-partitioning.__all__ = [
- _n for _n in ("years", "months", "days", "hours", "bucket") if
hasattr(partitioning, _n)
-]
+for _transform in _partition_transforms():
+ _transform.__module__ = partitioning.__name__
+ _transform.__qualname__ = _transform.__name__
+ setattr(partitioning, _transform.__name__, _transform)
+del _transform
+partitioning.__all__ = ["years", "months", "days", "hours", "bucket"]
+
+
+def years(col):
+ _warnings.warn("Deprecated in 4.0.0, use partitioning.years instead.",
FutureWarning)
+ return partitioning.years(col)
+
+
+def months(col):
+ _warnings.warn("Deprecated in 4.0.0, use partitioning.months instead.",
FutureWarning)
+ return partitioning.months(col)
+
+
+def days(col):
+ _warnings.warn("Deprecated in 4.0.0, use partitioning.days instead.",
FutureWarning)
+ return partitioning.days(col)
+
+
+def hours(col):
+ _warnings.warn("Deprecated in 4.0.0, use partitioning.hours instead.",
FutureWarning)
+ return partitioning.hours(col)
+
+
+def bucket(numBuckets, col):
+ _warnings.warn("Deprecated in 4.0.0, use partitioning.bucket instead.",
FutureWarning)
+ return partitioning.bucket(numBuckets, col)
+
+
_sys.modules["pyspark.sql.functions.partitioning"] = partitioning
# Public classes/enums that pyspark.sql.functions re-exports (so `from
diff --git a/python/pyspark/sql/functions_generated.py
b/python/pyspark/sql/functions_generated.py
index 4d5baf0..06e857d 100644
--- a/python/pyspark/sql/functions_generated.py
+++ b/python/pyspark/sql/functions_generated.py
@@ -250,11 +250,6 @@ def btrim(str, trim=None):
return _invoke_function_over_columns("btrim", str, trim)
else:
return _invoke_function_over_columns("btrim", str)
-def bucket(numBuckets, col):
- _a = []
- _a.append(numBuckets)
- _a.append(col)
- return _dispatch("bucket", _a)
def cardinality(col):
return _invoke_function_over_columns("cardinality", col)
def cbrt(col):
@@ -396,10 +391,6 @@ def dayofweek(col):
return _invoke_function_over_columns("dayofweek", col)
def dayofyear(col):
return _invoke_function_over_columns("dayofyear", col)
-def days(col):
- _a = []
- _a.append(col)
- return _dispatch("days", _a)
def decode(col, charset):
return _invoke_function("decode", _to_col(col), lit(charset))
def degrees(col):
@@ -577,10 +568,6 @@ def hmac(key, message, algorithm=None):
return _invoke_function_over_columns("hmac", key, message, algorithm)
def hour(col):
return _invoke_function_over_columns("hour", col)
-def hours(col):
- _a = []
- _a.append(col)
- return _dispatch("hours", _a)
def hypot(col1, col2):
return _invoke_binary_math_function("hypot", col1, col2)
def ifnull(col1, col2):
@@ -960,10 +947,6 @@ def month(col):
return _invoke_function_over_columns("month", col)
def monthname(col):
return _invoke_function_over_columns("monthname", col)
-def months(col):
- _a = []
- _a.append(col)
- return _dispatch("months", _a)
def months_between(date1, date2, roundOff=True):
return _invoke_function("months_between", _to_col(date1), _to_col(date2),
lit(roundOff))
def named_struct(*cols):
@@ -1972,11 +1955,6 @@ def weekday(col):
return _invoke_function_over_columns("weekday", col)
def weekofyear(col):
return _invoke_function_over_columns("weekofyear", col)
-def when(condition, value):
- _a = []
- _a.append(condition)
- _a.append(value)
- return _dispatch("when", _a)
def width_bucket(v, min, max, numBucket):
numBucket = _enum_to_value(numBucket)
numBucket = lit(numBucket) if isinstance(numBucket, int) else numBucket
@@ -2048,10 +2026,6 @@ def xxhash64(*cols):
return _invoke_function_over_columns("xxhash64", *cols)
def year(col):
return _invoke_function_over_columns("year", col)
-def years(col):
- _a = []
- _a.append(col)
- return _dispatch("years", _a)
def zeroifnull(col):
return _invoke_function_over_columns("zeroifnull", col)
@@ -2119,7 +2093,6 @@ __all__ = [
"bool_or",
"bround",
"btrim",
- "bucket",
"cardinality",
"cbrt",
"ceil",
@@ -2177,7 +2150,6 @@ __all__ = [
"dayofmonth",
"dayofweek",
"dayofyear",
- "days",
"decode",
"degrees",
"dense_rank",
@@ -2225,7 +2197,6 @@ __all__ = [
"hll_union_agg",
"hmac",
"hour",
- "hours",
"hypot",
"ifnull",
"ilike",
@@ -2320,7 +2291,6 @@ __all__ = [
"monotonically_increasing_id",
"month",
"monthname",
- "months",
"months_between",
"named_struct",
"nanvl",
@@ -2576,7 +2546,6 @@ __all__ = [
"version",
"weekday",
"weekofyear",
- "when",
"width_bucket",
"window",
"window_time",
@@ -2591,6 +2560,5 @@ __all__ = [
"xpath_string",
"xxhash64",
"year",
- "years",
"zeroifnull",
]
diff --git a/python/pyspark/sql/pandas/functions.py
b/python/pyspark/sql/pandas/functions.py
index fec7268..b14b10b 100644
--- a/python/pyspark/sql/pandas/functions.py
+++ b/python/pyspark/sql/pandas/functions.py
@@ -14,43 +14,871 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-"""Rust-backed pandas UDF entry points under the official import path.
-Canonical home of the ``PandasUDFType`` / ``ArrowUDFType`` enums (mirroring
reference
-pyspark); ``pyspark.sql.functions`` re-exports them from here.
-"""
+import functools
import warnings
from inspect import getfullargspec, signature
from typing import get_type_hints
-from pyspark.errors import PySparkValueError
+from pyspark.util import PythonEvalType
from pyspark.sql.pandas.typehints import infer_eval_type
-from pyspark.util import PythonEvalType # (pyspark.sql.pandas.typehints reads
it here too)
+from pyspark.sql.pandas.utils import require_minimum_pandas_version,
require_minimum_pyarrow_version
+from pyspark.sql.types import DataType
+from pyspark.sql.udf import _create_udf
+from pyspark.sql.utils import is_remote
+from pyspark.errors import PySparkTypeError, PySparkValueError
class PandasUDFType:
- """Pandas UDF Types. See :func:`pyspark.sql.functions.pandas_udf`."""
+ """Pandas UDF Types. See :meth:`pyspark.sql.functions.pandas_udf`."""
- SCALAR = 200
- SCALAR_ITER = 204
- GROUPED_MAP = 201
- GROUPED_AGG = 202
- GROUPED_AGG_ITER = 217
- MAP_ITER = 205
- COGROUPED_MAP = 206
+ SCALAR = PythonEvalType.SQL_SCALAR_PANDAS_UDF
+
+ SCALAR_ITER = PythonEvalType.SQL_SCALAR_PANDAS_ITER_UDF
+
+ GROUPED_MAP = PythonEvalType.SQL_GROUPED_MAP_PANDAS_UDF
+
+ GROUPED_AGG = PythonEvalType.SQL_GROUPED_AGG_PANDAS_UDF
+
+ GROUPED_AGG_ITER = PythonEvalType.SQL_GROUPED_AGG_PANDAS_ITER_UDF
class ArrowUDFType:
- """Arrow UDF Types. See :func:`pyspark.sql.functions.pandas_udf`."""
+ """Arrow UDF Types. See :meth:`pyspark.sql.functions.arrow_udf`."""
+
+ SCALAR = PythonEvalType.SQL_SCALAR_ARROW_UDF
+
+ SCALAR_ITER = PythonEvalType.SQL_SCALAR_ARROW_ITER_UDF
+
+ GROUPED_AGG = PythonEvalType.SQL_GROUPED_AGG_ARROW_UDF
+
+ GROUPED_AGG_ITER = PythonEvalType.SQL_GROUPED_AGG_ARROW_ITER_UDF
+
+
+def arrow_udf(f=None, returnType=None, functionType=None):
+ """
+ Creates an arrow user defined function.
+
+ Arrow UDFs are user defined functions that are executed by Spark using
Arrow to transfer
+ and work with the data, which allows `pyarrow.Array` operations. An Arrow
UDF is defined
+ using the `arrow_udf` as a decorator or to wrap the function, and no
additional configuration
+ is required. An Arrow UDF behaves as a regular PySpark function API in
general.
+
+ .. versionadded:: 4.1.0
+
+ Parameters
+ ----------
+ f : function, optional
+ user-defined function. A python function if used as a standalone
function
+ returnType : :class:`pyspark.sql.types.DataType` or str, optional
+ the return type of the user-defined function. The value can be either a
+ :class:`pyspark.sql.types.DataType` object or a DDL-formatted type
string.
+ functionType : int, optional
+ an enum value in :class:`pyspark.sql.functions.ArrowUDFType`.
+ Default: SCALAR. This parameter exists for compatibility.
+ Using Python type hints is encouraged.
+
+ Examples
+ --------
+ In order to use this API, customarily the below are imported:
+
+ >>> import pyarrow as pa
+ >>> from pyspark.sql.functions import arrow_udf
+
+ `Python type hints <https://www.python.org/dev/peps/pep-0484>`_
+ detect the function types as below:
+
+ >>> from pyspark.sql.functions import ArrowUDFType
+ >>> from pyspark.sql.types import IntegerType
+ >>> @arrow_udf(IntegerType(), ArrowUDFType.SCALAR)
+ ... def slen(v: pa.Array) -> pa.Array:
+ ... return pa.compute.utf8_length(v)
+
+ Note that the type hint should use `pyarrow.Array` in all cases.
+
+ * Arrays to Arrays
+ `pyarrow.Array`, ... -> `pyarrow.Array`
+
+ The function takes one or more `pyarrow.Array` and outputs one
`pyarrow.Array`.
+ The output of the function should always be of the same length as the
input.
+
+ >>> @arrow_udf("string")
+ ... def to_upper(s: pa.Array) -> pa.Array:
+ ... return pa.compute.ascii_upper(s)
+ ...
+ >>> df = spark.createDataFrame([("John Doe",)], ("name",))
+ >>> df.select(to_upper("name")).show()
+ +--------------+
+ |to_upper(name)|
+ +--------------+
+ | JOHN DOE|
+ +--------------+
+
+ >>> @arrow_udf("first string, last string")
+ ... def split_expand(v: pa.Array) -> pa.Array:
+ ... b = pa.compute.ascii_split_whitespace(v)
+ ... s0 = pa.array([t[0] for t in b])
+ ... s1 = pa.array([t[1] for t in b])
+ ... return pa.StructArray.from_arrays([s0, s1], names=["first",
"last"])
+ ...
+ >>> df = spark.createDataFrame([("John Doe",)], ("name",))
+ >>> df.select(split_expand("name")).show()
+ +------------------+
+ |split_expand(name)|
+ +------------------+
+ | {John, Doe}|
+ +------------------+
+
+ This type of Pandas UDF can use keyword arguments:
+
+ >>> from pyspark.sql import functions as sf
+ >>> @arrow_udf(returnType=IntegerType())
+ ... def calc(a: pa.Array, b: pa.Array) -> pa.Array:
+ ... return pa.compute.add(a, pa.compute.multiply(b, 10))
+ ...
+ >>> spark.range(2).select(calc(b=sf.col("id") * 10,
a=sf.col("id"))).show()
+ +-----------------------------+
+ |calc(b => (id * 10), a => id)|
+ +-----------------------------+
+ | 0|
+ | 101|
+ +-----------------------------+
+
+ .. note:: The length of the input is not that of the whole input
column, but is the
+ length of an internal batch used for each call to the function.
+
+ * Iterator of Arrays to Iterator of Arrays
+ `Iterator[pyarrow.Array]` -> `Iterator[pyarrow.Array]`
+
+ The function takes an iterator of `pyarrow.Array` and outputs an
iterator of
+ `pyarrow.Array`. In this case, the created arrow UDF instance requires
one input
+ column when this is called as a PySpark column. The length of the
entire output from
+ the function should be the same length of the entire input; therefore,
it can
+ prefetch the data from the input iterator as long as the lengths are
the same.
+
+ It is also useful when the UDF execution
+ requires initializing some states, although internally it works
identically as
+ Arrays to Arrays case. The pseudocode below illustrates the example.
+
+ .. highlight:: python
+ .. code-block:: python
+
+ @arrow_udf("long")
+ def calculate(iterator: Iterator[pa.Array]) -> Iterator[pa.Array]:
+ # Do some expensive initialization with a state
+ state = very_expensive_initialization()
+ for x in iterator:
+ # Use that state for whole iterator.
+ yield calculate_with_state(x, state)
+
+ df.select(calculate("value")).show()
+
+ >>> import pandas as pd
+ >>> from typing import Iterator
+ >>> @arrow_udf("long")
+ ... def plus_one(iterator: Iterator[pa.Array]) -> Iterator[pa.Array]:
+ ... for v in iterator:
+ ... yield pa.compute.add(v, 1)
+ ...
+ >>> df = spark.createDataFrame(pd.DataFrame([1, 2, 3], columns=["v"]))
+ >>> df.select(plus_one(df.v)).show()
+ +-----------+
+ |plus_one(v)|
+ +-----------+
+ | 2|
+ | 3|
+ | 4|
+ +-----------+
+
+ .. note:: The length of each series is the length of a batch
internally used.
+
+ * Iterator of Multiple Arrays to Iterator of Arrays
+ `Iterator[Tuple[pyarrow.Array, ...]]` -> `Iterator[pyarrow.Array]`
+
+ The function takes an iterator of a tuple of multiple `pyarrow.Array`
and outputs an
+ iterator of `pyarrow.Array`. In this case, the created arrow UDF
instance requires
+ input columns as many as the series when this is called as a PySpark
column.
+ Otherwise, it has the same characteristics and restrictions as
Iterator of Arrays
+ to Iterator of Arrays case.
+
+ >>> from typing import Iterator, Tuple
+ >>> from pyspark.sql import functions as sf
+ >>> @arrow_udf("long")
+ ... def multiply(iterator: Iterator[Tuple[pa.Array, pa.Array]]) ->
Iterator[pa.Array]:
+ ... for v1, v2 in iterator:
+ ... yield pa.compute.multiply(v1, v2.field("v"))
+ ...
+ >>> df = spark.createDataFrame(pd.DataFrame([1, 2, 3], columns=["v"]))
+ >>> df.withColumn('output', multiply(sf.col("v"),
sf.struct(sf.col("v")))).show()
+ +---+------+
+ | v|output|
+ +---+------+
+ | 1| 1|
+ | 2| 4|
+ | 3| 9|
+ +---+------+
+
+ .. note:: The length of each series is the length of a batch
internally used.
+
+ * Arrays to Scalar
+ `pyarrow.Array`, ... -> `Any`
+
+ The function takes `pyarrow.Array` and returns a scalar value. The
returned scalar
+ can be a python primitive type, (e.g., int or float), a numpy data
type (e.g.,
+ numpy.int64 or numpy.float64), or a `pyarrow.Scalar` instance which
supports complex
+ return types.
+ `Any` should ideally be a specific scalar type accordingly.
+
+ >>> @arrow_udf("double")
+ ... def mean_udf(v: pa.Array) -> float:
+ ... return pa.compute.mean(v).as_py()
+ ...
+ >>> df = spark.createDataFrame(
+ ... [(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)], ("id",
"v"))
+ >>> df.groupby("id").agg(mean_udf(df['v'])).show()
+ +---+-----------+
+ | id|mean_udf(v)|
+ +---+-----------+
+ | 1| 1.5|
+ | 2| 6.0|
+ +---+-----------+
+
+ The retun type can also be a complex type such as struct, list, or map.
+
+ >>> @arrow_udf("struct<m1: double, m2: double>")
+ ... def min_max_udf(v: pa.Array) -> pa.Scalar:
+ ... m1 = pa.compute.min(v)
+ ... m2 = pa.compute.max(v)
+ ... t = pa.struct([pa.field("m1", pa.float64()), pa.field("m2",
pa.float64())])
+ ... return pa.scalar(value={"m1": m1.as_py(), "m2": m2.as_py()},
type=t)
+ ...
+ >>> df = spark.createDataFrame(
+ ... [(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)], ("id",
"v"))
+ >>> df.groupby("id").agg(min_max_udf(df['v'])).show()
+ +---+--------------+
+ | id|min_max_udf(v)|
+ +---+--------------+
+ | 1| {1.0, 2.0}|
+ | 2| {3.0, 10.0}|
+ +---+--------------+
+
+ This type of Pandas UDF can use keyword arguments:
+
+ >>> @arrow_udf("double")
+ ... def weighted_mean_udf(v: pa.Array, w: pa.Array) -> float:
+ ... import numpy as np
+ ... return np.average(v.to_numpy(), weights=w)
+ ...
+ >>> df = spark.createDataFrame(
+ ... [(1, 1.0, 1.0), (1, 2.0, 2.0), (2, 3.0, 1.0), (2, 5.0, 2.0),
(2, 10.0, 3.0)],
+ ... ("id", "v", "w"))
+ >>> df.groupby("id").agg(weighted_mean_udf(w=df["w"],
v=df["v"])).show()
+ +---+---------------------------------+
+ | id|weighted_mean_udf(w => w, v => v)|
+ +---+---------------------------------+
+ | 1| 1.6666666666666667|
+ | 2| 7.166666666666667|
+ +---+---------------------------------+
+
+ This UDF can also be used as window functions as below:
+
+ >>> from pyspark.sql import Window
+ >>> @arrow_udf("double")
+ ... def mean_udf(v: pa.Array) -> float:
+ ... return pa.compute.mean(v).as_py()
+ ...
+ >>> df = spark.createDataFrame(
+ ... [(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)], ("id",
"v"))
+ >>> w = Window.partitionBy('id').orderBy('v').rowsBetween(-1, 0)
+ >>> df.withColumn('mean_v', mean_udf("v").over(w)).show()
+ +---+----+------+
+ | id| v|mean_v|
+ +---+----+------+
+ | 1| 1.0| 1.0|
+ | 1| 2.0| 1.5|
+ | 2| 3.0| 3.0|
+ | 2| 5.0| 4.0|
+ | 2|10.0| 7.5|
+ +---+----+------+
+
+ .. note:: For performance reasons, the input series to window
functions are not copied.
+ Therefore, mutating the input arrays is not allowed and will cause
incorrect results.
+ For the same reason, users should also not rely on the index of
the input arrays.
+
+ * Iterator of Arrays to Scalar
+ `Iterator[pyarrow.Array]` -> `Any`
+
+ The function takes an iterator of `pyarrow.Array` and returns a scalar
value. This is
+ useful for grouped aggregations where the UDF can process all batches
for a group
+ iteratively, which is more memory-efficient than loading all data at
once. The returned
+ scalar can be a python primitive type, a numpy data type, or a
`pyarrow.Scalar` instance.
+
+ .. note:: Only a single UDF is supported per aggregation.
+
+ >>> from typing import Iterator
+ >>> @arrow_udf("double")
+ ... def arrow_mean(it: Iterator[pa.Array]) -> float:
+ ... sum_val = 0.0
+ ... cnt = 0
+ ... for v in it:
+ ... assert isinstance(v, pa.Array)
+ ... sum_val += pa.compute.sum(v).as_py()
+ ... cnt += len(v)
+ ... return sum_val / cnt
+ ...
+ >>> df = spark.createDataFrame(
+ ... [(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)], ("id",
"v"))
+ >>> df.groupby("id").agg(arrow_mean(df['v'])).show()
+ +---+-------------+
+ | id|arrow_mean(v)|
+ +---+-------------+
+ | 1| 1.5|
+ | 2| 6.0|
+ +---+-------------+
+
+ * Iterator of Multiple Arrays to Scalar
+ `Iterator[Tuple[pyarrow.Array, ...]]` -> `Any`
+
+ The function takes an iterator of a tuple of multiple `pyarrow.Array`
and returns a
+ scalar value. This is useful for grouped aggregations with multiple
input columns.
+
+ .. note:: Only a single UDF is supported per aggregation.
+
+ >>> from typing import Iterator, Tuple
+ >>> import numpy as np
+ >>> @arrow_udf("double")
+ ... def arrow_weighted_mean(it: Iterator[Tuple[pa.Array, pa.Array]])
-> float:
+ ... weighted_sum = 0.0
+ ... weight = 0.0
+ ... for v, w in it:
+ ... assert isinstance(v, pa.Array)
+ ... assert isinstance(w, pa.Array)
+ ... weighted_sum += np.dot(v, w)
+ ... weight += pa.compute.sum(w).as_py()
+ ... return weighted_sum / weight
+ ...
+ >>> df = spark.createDataFrame(
+ ... [(1, 1.0, 1.0), (1, 2.0, 2.0), (2, 3.0, 1.0), (2, 5.0, 2.0),
(2, 10.0, 3.0)],
+ ... ("id", "v", "w"))
+ >>> df.groupby("id").agg(arrow_weighted_mean(df["v"], df["w"])).show()
+ +---+-------------------------+
+ | id|arrow_weighted_mean(v, w)|
+ +---+-------------------------+
+ | 1| 1.6666666666666...|
+ | 2| 7.166666666666...|
+ +---+-------------------------+
+
+ Notes
+ -----
+ The user-defined functions do not support conditional expressions or short
circuiting
+ in boolean expressions and it ends up with being executed all internally.
If the functions
+ can fail on special rows, the workaround is to incorporate the condition
into the functions.
+
+ The user-defined functions do not take keyword arguments on the calling
side.
+
+ The data type of returned `pyarrow.Array` from the user-defined functions
should be
+ matched with defined `returnType` (see :meth:`types.to_arrow_type` and
+ :meth:`types.from_arrow_type`). When there is mismatch between them, Spark
might do
+ conversion on returned data. The conversion is not guaranteed to be
correct and results
+ should be checked for accuracy by users.
+
+ See Also
+ --------
+ pyspark.sql.GroupedData.agg
+ pyspark.sql.DataFrame.mapInArrow
+ pyspark.sql.GroupedData.applyInArrow
+ pyspark.sql.PandasCogroupedOps.applyInArrow
+ pyspark.sql.UDFRegistration.register
+ pyspark.sql.GroupedData.applyInPandas
+ """
+ require_minimum_pyarrow_version()
+
+ return vectorized_udf(f, returnType, functionType, "arrow")
+
+
+def pandas_udf(f=None, returnType=None, functionType=None):
+ """
+ Creates a pandas user defined function.
+
+ Pandas UDFs are user defined functions that are executed by Spark using
Arrow to transfer
+ data and Pandas to work with the data, which allows pandas operations. A
Pandas UDF
+ is defined using the `pandas_udf` as a decorator or to wrap the function,
and no
+ additional configuration is required. A Pandas UDF behaves as a regular
PySpark function
+ API in general.
+
+ .. versionadded:: 2.3.0
+
+ .. versionchanged:: 3.4.0
+ Supports Spark Connect.
+
+ .. versionchanged:: 4.0.0
+ Supports keyword-arguments in SCALAR and GROUPED_AGG type.
+
+ .. versionchanged:: 4.1.0
+ Supports iterator API in GROUPED_MAP type.
+
+ .. versionchanged:: 4.2.0
+ Supports iterator API in GROUPED_AGG type.
+
+ Parameters
+ ----------
+ f : function, optional
+ user-defined function. A python function if used as a standalone
function
+ returnType : :class:`pyspark.sql.types.DataType` or str, optional
+ the return type of the user-defined function. The value can be either a
+ :class:`pyspark.sql.types.DataType` object or a DDL-formatted type
string.
+ functionType : int, optional
+ an enum value in :class:`pyspark.sql.functions.PandasUDFType`.
+ Default: SCALAR. This parameter exists for compatibility.
+ Using Python type hints is encouraged.
+
+ Examples
+ --------
+ In order to use this API, customarily the below are imported:
+
+ >>> import pandas as pd
+ >>> from pyspark.sql.functions import pandas_udf
+
+ From Spark 3.0 with Python 3.6+, `Python type hints
<https://www.python.org/dev/peps/pep-0484>`_
+ detect the function types as below:
+
+ >>> from pyspark.sql.types import IntegerType
+ >>> @pandas_udf(IntegerType())
+ ... def slen(s: pd.Series) -> pd.Series:
+ ... return s.str.len()
+
+ Prior to Spark 3.0, the pandas UDF used `functionType` to decide the
execution type as below:
+
+ >>> from pyspark.sql.functions import PandasUDFType
+ >>> from pyspark.sql.types import IntegerType
+ >>> @pandas_udf(IntegerType(), PandasUDFType.SCALAR)
+ ... def slen(s):
+ ... return s.str.len()
+
+ It is preferred to specify type hints for the pandas UDF instead of
specifying pandas UDF
+ type via `functionType` which will be deprecated in the future releases.
+
+ Note that the type hint should use `pandas.Series` in all cases but there
is one variant
+ that `pandas.DataFrame` should be used for its input or output type hint
instead when the input
+ or output column is of :class:`pyspark.sql.types.StructType`. The
following example shows
+ a Pandas UDF which takes long column, string column and struct column, and
outputs a struct
+ column. It requires the function to specify the type hints of
`pandas.Series` and
+ `pandas.DataFrame` as below:
+
+ >>> @pandas_udf("col1 string, col2 long")
+ ... def func(s1: pd.Series, s2: pd.Series, s3: pd.DataFrame) ->
pd.DataFrame:
+ ... s3['col2'] = s1 + s2.str.len()
+ ... return s3
+
+
+ Create a Spark DataFrame that has three columns including a struct column.
+
+ >>> df = spark.createDataFrame(
+ ... [[1, "a string", ("a nested string",)]],
+ ... "long_col long, string_col string, struct_col struct<col1:string>")
+
+ >>> df.printSchema()
+ root
+ |-- long_col: long (nullable = true)
+ |-- string_col: string (nullable = true)
+ |-- struct_col: struct (nullable = true)
+ | |-- col1: string (nullable = true)
+
+ >>> df.select(func("long_col", "string_col", "struct_col")).printSchema()
+ root
+ |-- func(long_col, string_col, struct_col): struct (nullable = true)
+ | |-- col1: string (nullable = true)
+ | |-- col2: long (nullable = true)
+
+ In the following sections, it describes the combinations of the supported
type hints. For
+ simplicity, `pandas.DataFrame` variant is omitted.
+
+ * Series to Series
+ `pandas.Series`, ... -> `pandas.Series`
+
+ The function takes one or more `pandas.Series` and outputs one
`pandas.Series`.
+ The output of the function should always be of the same length as the
input.
+
+ >>> @pandas_udf("string")
+ ... def to_upper(s: pd.Series) -> pd.Series:
+ ... return s.str.upper()
+ ...
+ >>> df = spark.createDataFrame([("John Doe",)], ("name",))
+ >>> df.select(to_upper("name")).show()
+ +--------------+
+ |to_upper(name)|
+ +--------------+
+ | JOHN DOE|
+ +--------------+
+
+ >>> @pandas_udf("first string, last string")
+ ... def split_expand(s: pd.Series) -> pd.DataFrame:
+ ... return s.str.split(expand=True)
+ ...
+ >>> df = spark.createDataFrame([("John Doe",)], ("name",))
+ >>> df.select(split_expand("name")).show()
+ +------------------+
+ |split_expand(name)|
+ +------------------+
+ | {John, Doe}|
+ +------------------+
+
+ This type of Pandas UDF can use keyword arguments:
- SCALAR = 250
- SCALAR_ITER = 251
- GROUPED_AGG = 252
- GROUPED_AGG_ITER = 254
+ >>> from pyspark.sql import functions as sf
+ >>> @pandas_udf(returnType=IntegerType())
+ ... def calc(a: pd.Series, b: pd.Series) -> pd.Series:
+ ... return a + 10 * b
+ ...
+ >>> spark.range(2).select(calc(b=sf.col("id") * 10,
a=sf.col("id"))).show()
+ +-----------------------------+
+ |calc(b => (id * 10), a => id)|
+ +-----------------------------+
+ | 0|
+ | 101|
+ +-----------------------------+
+ .. note:: The length of the input is not that of the whole input
column, but is the
+ length of an internal batch used for each call to the function.
+ * Iterator of Series to Iterator of Series
+ `Iterator[pandas.Series]` -> `Iterator[pandas.Series]`
-# Verbatim from PySpark: validates the function of a pandas or Arrow UDF for
its eval type.
+ The function takes an iterator of `pandas.Series` and outputs an
iterator of
+ `pandas.Series`. In this case, the created pandas UDF instance
requires one input
+ column when this is called as a PySpark column. The length of the
entire output from
+ the function should be the same length of the entire input; therefore,
it can
+ prefetch the data from the input iterator as long as the lengths are
the same.
+
+ It is also useful when the UDF execution
+ requires initializing some states although internally it works
identically as
+ Series to Series case. The pseudocode below illustrates the example.
+
+ .. highlight:: python
+ .. code-block:: python
+
+ @pandas_udf("long")
+ def calculate(iterator: Iterator[pd.Series]) ->
Iterator[pd.Series]:
+ # Do some expensive initialization with a state
+ state = very_expensive_initialization()
+ for x in iterator:
+ # Use that state for whole iterator.
+ yield calculate_with_state(x, state)
+
+ df.select(calculate("value")).show()
+
+ >>> from typing import Iterator
+ >>> @pandas_udf("long")
+ ... def plus_one(iterator: Iterator[pd.Series]) -> Iterator[pd.Series]:
+ ... for s in iterator:
+ ... yield s + 1
+ ...
+ >>> df = spark.createDataFrame(pd.DataFrame([1, 2, 3], columns=["v"]))
+ >>> df.select(plus_one(df.v)).show()
+ +-----------+
+ |plus_one(v)|
+ +-----------+
+ | 2|
+ | 3|
+ | 4|
+ +-----------+
+
+ .. note:: The length of each series is the length of a batch
internally used.
+
+ * Iterator of Multiple Series to Iterator of Series
+ `Iterator[Tuple[pandas.Series, ...]]` -> `Iterator[pandas.Series]`
+
+ The function takes an iterator of a tuple of multiple `pandas.Series`
and outputs an
+ iterator of `pandas.Series`. In this case, the created pandas UDF
instance requires
+ input columns as many as the series when this is called as a PySpark
column.
+ Otherwise, it has the same characteristics and restrictions as
Iterator of Series
+ to Iterator of Series case.
+
+ >>> from typing import Iterator, Tuple
+ >>> from pyspark.sql import functions as sf
+ >>> @pandas_udf("long")
+ ... def multiply(iterator: Iterator[Tuple[pd.Series, pd.DataFrame]])
-> Iterator[pd.Series]:
+ ... for s1, df in iterator:
+ ... yield s1 * df.v
+ ...
+ >>> df = spark.createDataFrame(pd.DataFrame([1, 2, 3], columns=["v"]))
+ >>> df.withColumn('output', multiply(sf.col("v"),
sf.struct(sf.col("v")))).show()
+ +---+------+
+ | v|output|
+ +---+------+
+ | 1| 1|
+ | 2| 4|
+ | 3| 9|
+ +---+------+
+
+ .. note:: The length of each series is the length of a batch
internally used.
+
+ * Series to Scalar
+ `pandas.Series`, ... -> `Any`
+
+ The function takes `pandas.Series` and returns a scalar value. The
`returnType`
+ should be a primitive data type, and the returned scalar can be either
a python primitive
+ type, e.g., int or float or a numpy data type, e.g., numpy.int64 or
numpy.float64.
+ `Any` should ideally be a specific scalar type accordingly.
+
+ >>> @pandas_udf("double")
+ ... def mean_udf(v: pd.Series) -> float:
+ ... return v.mean()
+ ...
+ >>> df = spark.createDataFrame(
+ ... [(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)], ("id",
"v"))
+ >>> df.groupby("id").agg(mean_udf(df['v'])).show()
+ +---+-----------+
+ | id|mean_udf(v)|
+ +---+-----------+
+ | 1| 1.5|
+ | 2| 6.0|
+ +---+-----------+
+
+ This type of Pandas UDF can use keyword arguments:
+
+ >>> @pandas_udf("double")
+ ... def weighted_mean_udf(v: pd.Series, w: pd.Series) -> float:
+ ... import numpy as np
+ ... return np.average(v, weights=w)
+ ...
+ >>> df = spark.createDataFrame(
+ ... [(1, 1.0, 1.0), (1, 2.0, 2.0), (2, 3.0, 1.0), (2, 5.0, 2.0),
(2, 10.0, 3.0)],
+ ... ("id", "v", "w"))
+ >>> df.groupby("id").agg(weighted_mean_udf(w=df["w"],
v=df["v"])).show()
+ +---+---------------------------------+
+ | id|weighted_mean_udf(w => w, v => v)|
+ +---+---------------------------------+
+ | 1| 1.6666666666666667|
+ | 2| 7.166666666666667|
+ +---+---------------------------------+
+
+ This UDF can also be used as window functions as below:
+
+ >>> from pyspark.sql import Window
+ >>> @pandas_udf("double")
+ ... def mean_udf(v: pd.Series) -> float:
+ ... return v.mean()
+ ...
+ >>> df = spark.createDataFrame(
+ ... [(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)], ("id",
"v"))
+ >>> w = Window.partitionBy('id').orderBy('v').rowsBetween(-1, 0)
+ >>> df.withColumn('mean_v', mean_udf("v").over(w)).show()
+ +---+----+------+
+ | id| v|mean_v|
+ +---+----+------+
+ | 1| 1.0| 1.0|
+ | 1| 2.0| 1.5|
+ | 2| 3.0| 3.0|
+ | 2| 5.0| 4.0|
+ | 2|10.0| 7.5|
+ +---+----+------+
+
+ .. note:: For performance reasons, the input series to window
functions are not copied.
+ Therefore, mutating the input series is not allowed and will cause
incorrect results.
+ For the same reason, users should also not rely on the index of
the input series.
+
+ * Iterator of Series to Scalar
+ `Iterator[pandas.Series]` -> `Any`
+
+ The function takes an iterator of `pandas.Series` and returns a scalar
value. This is
+ useful for grouped aggregations where the UDF can process all batches
for a group
+ iteratively, which is more memory-efficient than loading all data at
once. The returned
+ scalar can be a python primitive type or a numpy data type.
+
+ .. note:: Only a single UDF is supported per aggregation.
+
+ >>> from typing import Iterator
+ >>> @pandas_udf("double")
+ ... def pandas_mean_iter(it: Iterator[pd.Series]) -> float:
+ ... sum_val = 0.0
+ ... cnt = 0
+ ... for v in it:
+ ... sum_val += v.sum()
+ ... cnt += len(v)
+ ... return sum_val / cnt
+ ...
+ >>> df = spark.createDataFrame(
+ ... [(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)], ("id",
"v"))
+ >>> df.groupby("id").agg(pandas_mean_iter(df['v'])).show()
+ +---+-------------------+
+ | id|pandas_mean_iter(v)|
+ +---+-------------------+
+ | 1| 1.5|
+ | 2| 6.0|
+ +---+-------------------+
+
+ * Iterator of Multiple Series to Scalar
+ `Iterator[Tuple[pandas.Series, ...]]` -> `Any`
+
+ The function takes an iterator of a tuple of multiple `pandas.Series`
and returns a
+ scalar value. This is useful for grouped aggregations with multiple
input columns.
+
+ .. note:: Only a single UDF is supported per aggregation.
+
+ >>> from typing import Iterator, Tuple
+ >>> import numpy as np
+ >>> @pandas_udf("double")
+ ... def pandas_weighted_mean_iter(it: Iterator[Tuple[pd.Series,
pd.Series]]) -> float:
+ ... weighted_sum = 0.0
+ ... weight = 0.0
+ ... for v, w in it:
+ ... weighted_sum += np.dot(v, w)
+ ... weight += w.sum()
+ ... return weighted_sum / weight
+ ...
+ >>> df = spark.createDataFrame(
+ ... [(1, 1.0, 1.0), (1, 2.0, 2.0), (2, 3.0, 1.0), (2, 5.0, 2.0),
(2, 10.0, 3.0)],
+ ... ("id", "v", "w"))
+ >>> df.groupby("id").agg(pandas_weighted_mean_iter(df["v"],
df["w"])).show()
+ +---+-------------------------------+
+ | id|pandas_weighted_mean_iter(v, w)|
+ +---+-------------------------------+
+ | 1| 1.6666666666666...|
+ | 2| 7.166666666666...|
+ +---+-------------------------------+
+
+ Notes
+ -----
+ The user-defined functions do not support conditional expressions or short
circuiting
+ in boolean expressions and it ends up with being executed all internally.
If the functions
+ can fail on special rows, the workaround is to incorporate the condition
into the functions.
+
+ The user-defined functions do not take keyword arguments on the calling
side.
+
+ The data type of returned `pandas.Series` from the user-defined functions
should be
+ matched with defined `returnType` (see :meth:`types.to_arrow_type` and
+ :meth:`types.from_arrow_type`). When there is mismatch between them, Spark
might do
+ conversion on returned data. The conversion is not guaranteed to be
correct and results
+ should be checked for accuracy by users.
+
+ Currently,
+ :class:`pyspark.sql.types.ArrayType` of
:class:`pyspark.sql.types.TimestampType` and
+ nested :class:`pyspark.sql.types.StructType`
+ are currently not supported as output types.
+
+ See Also
+ --------
+ pyspark.sql.GroupedData.agg
+ pyspark.sql.DataFrame.mapInPandas
+ pyspark.sql.GroupedData.applyInPandas
+ pyspark.sql.PandasCogroupedOps.applyInPandas
+ pyspark.sql.UDFRegistration.register
+ """
+
+ # The return type and input type behavior of pandas_udfs is documented in
+ # python/pyspark/sql/tests/udf_type_tests.
+ # It shows most of Pandas data and SQL type conversions in Pandas UDFs
that are not
+ # yet visible to the user.
+ # Some of behaviors are buggy and might be changed in the near future. The
table might
+ # have to be eventually documented externally.
+ # The folder python/pyspark/sql/tests/udf_type_tests contains type tests
and golden
+ # files, as well as the code to regenerate the tables.
+ require_minimum_pandas_version()
+ require_minimum_pyarrow_version()
+
+ return vectorized_udf(f, returnType, functionType, "pandas")
+
+
+def vectorized_udf(
+ f=None,
+ returnType=None,
+ functionType=None,
+ kind: str = "pandas",
+):
+ assert kind in ["pandas", "arrow"], "kind should be either 'pandas' or
'arrow'"
+
+ # decorator @pandas_udf(returnType, functionType)
+ is_decorator = f is None or isinstance(f, (str, DataType))
+
+ if is_decorator:
+ # If DataType has been passed as a positional argument
+ # for decorator use it as a returnType
+ return_type = f or returnType
+
+ if functionType is not None:
+ # @pandas_udf(dataType, functionType=functionType)
+ # @pandas_udf(returnType=dataType, functionType=functionType)
+ eval_type = functionType
+ elif returnType is not None and isinstance(returnType, int):
+ # @pandas_udf(dataType, functionType)
+ eval_type = returnType
+ else:
+ # @pandas_udf(dataType) or @pandas_udf(returnType=dataType)
+ eval_type = None
+ else:
+ return_type = returnType
+
+ if functionType is not None:
+ eval_type = functionType
+ else:
+ eval_type = None
+
+ if return_type is None:
+ raise PySparkTypeError(
+ errorClass="CANNOT_BE_NONE",
+ messageParameters={"arg_name": "returnType"},
+ )
+
+ if kind == "pandas" and eval_type not in [
+ PythonEvalType.SQL_SCALAR_PANDAS_UDF,
+ PythonEvalType.SQL_SCALAR_PANDAS_ITER_UDF,
+ PythonEvalType.SQL_GROUPED_MAP_PANDAS_UDF,
+ PythonEvalType.SQL_GROUPED_MAP_PANDAS_ITER_UDF,
+ PythonEvalType.SQL_GROUPED_AGG_PANDAS_UDF,
+ PythonEvalType.SQL_GROUPED_AGG_PANDAS_ITER_UDF,
+ PythonEvalType.SQL_MAP_PANDAS_ITER_UDF,
+ PythonEvalType.SQL_MAP_ARROW_ITER_UDF,
+ PythonEvalType.SQL_COGROUPED_MAP_PANDAS_UDF,
+ PythonEvalType.SQL_GROUPED_MAP_PANDAS_UDF_WITH_STATE,
+ PythonEvalType.SQL_TRANSFORM_WITH_STATE_PANDAS_UDF,
+ PythonEvalType.SQL_TRANSFORM_WITH_STATE_PANDAS_INIT_STATE_UDF,
+ PythonEvalType.SQL_TRANSFORM_WITH_STATE_PYTHON_ROW_UDF,
+ PythonEvalType.SQL_TRANSFORM_WITH_STATE_PYTHON_ROW_INIT_STATE_UDF,
+ PythonEvalType.SQL_GROUPED_MAP_ARROW_UDF,
+ PythonEvalType.SQL_GROUPED_MAP_ARROW_ITER_UDF,
+ PythonEvalType.SQL_COGROUPED_MAP_ARROW_UDF,
+ None,
+ ]: # None means it should infer the type from type hints.
+ raise PySparkTypeError(
+ errorClass="INVALID_PANDAS_UDF_TYPE",
+ messageParameters={
+ "arg_name": "functionType",
+ "arg_type": str(eval_type),
+ },
+ )
+ if kind == "arrow" and eval_type not in [
+ PythonEvalType.SQL_SCALAR_ARROW_UDF,
+ PythonEvalType.SQL_SCALAR_ARROW_ITER_UDF,
+ PythonEvalType.SQL_GROUPED_AGG_ARROW_UDF,
+ PythonEvalType.SQL_GROUPED_AGG_ARROW_ITER_UDF,
+ None,
+ ]: # None means it should infer the type from type hints.
+ raise PySparkTypeError(
+ errorClass="INVALID_PANDAS_UDF_TYPE",
+ messageParameters={
+ "arg_name": "functionType",
+ "arg_type": str(eval_type),
+ },
+ )
+
+ if is_decorator:
+ return functools.partial(
+ _create_vectorized_udf,
+ returnType=return_type,
+ evalType=eval_type,
+ kind=kind,
+ )
+ else:
+ return _create_vectorized_udf(
+ f=f,
+ returnType=return_type,
+ evalType=eval_type,
+ kind=kind,
+ )
+
+
+# validate the pandas udf and return the adjusted eval type
def _validate_vectorized_udf(f, evalType, kind: str = "pandas") -> int:
assert kind in ["pandas", "arrow"], "kind should be either 'pandas' or
'arrow'"
@@ -190,8 +1018,49 @@ def _validate_vectorized_udf(f, evalType, kind: str =
"pandas") -> int:
return evalType
-# After the enums: importing this module first imports pyspark.sql.functions,
whose
-# re-export of the enums then finds them already defined here.
-from pyspark.sql.functions import arrow_udf, pandas_udf # noqa: E402,F401
-__all__ = ["arrow_udf", "pandas_udf", "PandasUDFType", "ArrowUDFType"]
+def _create_vectorized_udf(f, returnType, evalType, kind):
+ evalType = _validate_vectorized_udf(f, evalType, kind)
+
+ if is_remote():
+ from pyspark.sql.connect.udf import _create_udf as _create_connect_udf
+
+ return _create_connect_udf(f, returnType, evalType)
+ else:
+ return _create_udf(f, returnType, evalType)
+
+
+def _test() -> None:
+ import sys
+ import doctest
+ from pyspark.sql import SparkSession
+ import pyspark.sql.pandas.functions
+ from pyspark.testing.utils import have_pandas, have_pyarrow
+
+ globs = pyspark.sql.column.__dict__.copy()
+
+ if not have_pandas or not have_pyarrow:
+ del pyspark.sql.pandas.functions.pandas_udf.__doc__
+
+ if not have_pyarrow:
+ del pyspark.sql.pandas.functions.arrow_udf.__doc__
+
+ spark = (
+ SparkSession.builder.master("local[4]")
+ .appName("pyspark.sql.pandas.functions tests")
+ .getOrCreate()
+ )
+ globs["spark"] = spark
+
+ failure_count, test_count = doctest.testmod(
+ pyspark.sql.pandas.functions,
+ globs=globs,
+ optionflags=doctest.ELLIPSIS | doctest.NORMALIZE_WHITESPACE |
doctest.REPORT_NDIFF,
+ )
+ spark.stop()
+ if failure_count:
+ sys.exit(-1)
+
+
+if __name__ == "__main__":
+ _test()
diff --git a/python/pyspark/sql/udf.py b/python/pyspark/sql/udf.py
index 7986cc5..c1305df 100644
--- a/python/pyspark/sql/udf.py
+++ b/python/pyspark/sql/udf.py
@@ -23,8 +23,11 @@ Mirrors pyspark.sql.connect.udf and
pyspark.sql.functions.udf/pandas_udf.
import sys
from typing import Callable, Optional, Any
-from pyspark.sql.types import DataType, StringType
+from pyspark.errors import PySparkNotImplementedError, PySparkTypeError
+from pyspark.sql.pandas.types import to_arrow_type
+from pyspark.sql.types import DataType, StringType, StructType
from pyspark.serializers import CloudPickleSerializer
+from pyspark.util import PythonEvalType
def _as_concrete_datatype(returnType: Any) -> DataType:
@@ -56,10 +59,43 @@ class UserDefinedFunction:
name: Optional[str] = None,
deterministic: bool = True,
):
+ # The checks of PySpark Connect's UserDefinedFunction.
+ if not callable(func):
+ raise PySparkTypeError(
+ errorClass="NOT_EXPECTED_TYPE",
+ messageParameters={
+ "expected_type": "callable",
+ "arg_name": "func",
+ "arg_type": type(func).__name__,
+ },
+ )
+
+ if not isinstance(returnType, (DataType, str)):
+ raise PySparkTypeError(
+ errorClass="NOT_EXPECTED_TYPE",
+ messageParameters={
+ "expected_type": "DataType or str",
+ "arg_name": "returnType",
+ "arg_type": type(returnType).__name__,
+ },
+ )
+
+ if not isinstance(evalType, int):
+ raise PySparkTypeError(
+ errorClass="NOT_EXPECTED_TYPE",
+ messageParameters={
+ "expected_type": "int",
+ "arg_name": "evalType",
+ "arg_type": type(evalType).__name__,
+ },
+ )
+
self.func = func
# Normalize to a concrete, picklable DataType so the cloudpickled
command
# carries a real type (required by the pandas/Arrow worker, not just
str).
self.returnType = _as_concrete_datatype(returnType)
+ # As PySpark Connect's UDF checks its return type for its eval type
when it is made.
+ UserDefinedFunction._check_return_type(self.returnType, evalType)
self.evalType = evalType
self.name = name or (
func.__name__ if hasattr(func, "__name__") else "udf"
@@ -73,6 +109,188 @@ class UserDefinedFunction:
serializer = CloudPickleSerializer()
self.command = serializer.dumps((func, self.returnType))
+ # Verbatim from PySpark (pyspark.sql.udf.UserDefinedFunction).
+ @staticmethod
+ def _check_return_type(returnType: DataType, evalType: int) -> None:
+ if evalType == PythonEvalType.SQL_ARROW_BATCHED_UDF:
+ try:
+ to_arrow_type(returnType, timezone="UTC")
+ except TypeError:
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={
+ "feature": f"Invalid return type with Arrow-optimized
Python UDF: "
+ f"{returnType}"
+ },
+ )
+ elif (
+ evalType == PythonEvalType.SQL_SCALAR_PANDAS_UDF
+ or evalType == PythonEvalType.SQL_SCALAR_PANDAS_ITER_UDF
+ ):
+ try:
+ to_arrow_type(returnType, timezone="UTC")
+ except TypeError:
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={
+ "feature": f"Invalid return type with scalar Pandas
UDFs: {returnType}"
+ },
+ )
+ elif (
+ evalType == PythonEvalType.SQL_SCALAR_ARROW_UDF
+ or evalType == PythonEvalType.SQL_SCALAR_ARROW_ITER_UDF
+ ):
+ try:
+ to_arrow_type(returnType, timezone="UTC")
+ except TypeError:
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={
+ "feature": f"Invalid return type with scalar Arrow
UDFs: {returnType}"
+ },
+ )
+ elif (
+ evalType == PythonEvalType.SQL_GROUPED_MAP_PANDAS_UDF
+ or evalType == PythonEvalType.SQL_GROUPED_MAP_PANDAS_ITER_UDF
+ or evalType == PythonEvalType.SQL_GROUPED_MAP_PANDAS_UDF_WITH_STATE
+ ):
+ if isinstance(returnType, StructType):
+ try:
+ to_arrow_type(returnType, timezone="UTC")
+ except TypeError:
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={
+ "feature": f"Invalid return type with grouped map
Pandas UDFs or "
+ f"at groupby.applyInPandas(WithState):
{returnType}"
+ },
+ )
+ else:
+ raise PySparkTypeError(
+ errorClass="INVALID_RETURN_TYPE_FOR_PANDAS_UDF",
+ messageParameters={
+ "eval_type": "SQL_GROUPED_MAP_PANDAS_UDF or "
+ "SQL_GROUPED_MAP_PANDAS_ITER_UDF or "
+ "SQL_GROUPED_MAP_PANDAS_UDF_WITH_STATE",
+ "return_type": str(returnType),
+ },
+ )
+ elif (
+ evalType == PythonEvalType.SQL_MAP_PANDAS_ITER_UDF
+ or evalType == PythonEvalType.SQL_MAP_ARROW_ITER_UDF
+ ):
+ if isinstance(returnType, StructType):
+ try:
+ to_arrow_type(returnType, timezone="UTC")
+ except TypeError:
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={
+ "feature": f"Invalid return type in mapInPandas:
{returnType}"
+ },
+ )
+ else:
+ raise PySparkTypeError(
+ errorClass="INVALID_RETURN_TYPE_FOR_PANDAS_UDF",
+ messageParameters={
+ "eval_type": "SQL_MAP_PANDAS_ITER_UDF or
SQL_MAP_ARROW_ITER_UDF",
+ "return_type": str(returnType),
+ },
+ )
+ elif (
+ evalType == PythonEvalType.SQL_GROUPED_MAP_ARROW_UDF
+ or evalType == PythonEvalType.SQL_GROUPED_MAP_ARROW_ITER_UDF
+ ):
+ if isinstance(returnType, StructType):
+ try:
+ to_arrow_type(returnType, timezone="UTC")
+ except TypeError:
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={
+ "feature": "Invalid return type with grouped map
Arrow UDFs or "
+ f"at groupby.applyInArrow: {returnType}"
+ },
+ )
+ else:
+ raise PySparkTypeError(
+ errorClass="INVALID_RETURN_TYPE_FOR_ARROW_UDF",
+ messageParameters={
+ "eval_type": "SQL_GROUPED_MAP_ARROW_UDF or
SQL_GROUPED_MAP_ARROW_ITER_UDF",
+ "return_type": str(returnType),
+ },
+ )
+ elif evalType == PythonEvalType.SQL_COGROUPED_MAP_PANDAS_UDF:
+ if isinstance(returnType, StructType):
+ try:
+ to_arrow_type(returnType, timezone="UTC")
+ except TypeError:
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={
+ "feature": f"Invalid return type in
cogroup.applyInPandas: {returnType}"
+ },
+ )
+ else:
+ raise PySparkTypeError(
+ errorClass="INVALID_RETURN_TYPE_FOR_PANDAS_UDF",
+ messageParameters={
+ "eval_type": "SQL_COGROUPED_MAP_PANDAS_UDF",
+ "return_type": str(returnType),
+ },
+ )
+ elif evalType == PythonEvalType.SQL_COGROUPED_MAP_ARROW_UDF:
+ if isinstance(returnType, StructType):
+ try:
+ to_arrow_type(returnType, timezone="UTC")
+ except TypeError:
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={
+ "feature": f"Invalid return type in
cogroup.applyInArrow: {returnType}"
+ },
+ )
+ else:
+ raise PySparkTypeError(
+ errorClass="INVALID_RETURN_TYPE_FOR_ARROW_UDF",
+ messageParameters={
+ "eval_type": "SQL_COGROUPED_MAP_ARROW_UDF",
+ "return_type": str(returnType),
+ },
+ )
+ elif evalType == PythonEvalType.SQL_GROUPED_AGG_PANDAS_UDF:
+ try:
+ # StructType is not yet allowed as a return type, explicitly
check here to fail fast
+ if isinstance(returnType, StructType):
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={
+ "feature": f"Invalid return type with grouped
aggregate Pandas UDFs: "
+ f"{returnType}"
+ },
+ )
+ to_arrow_type(returnType, timezone="UTC")
+ except TypeError:
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={
+ "feature": f"Invalid return type with grouped
aggregate Pandas UDFs: "
+ f"{returnType}"
+ },
+ )
+ elif evalType == PythonEvalType.SQL_GROUPED_AGG_ARROW_UDF:
+ try:
+ # Different from SQL_GROUPED_AGG_PANDAS_UDF, StructType is
allowed here
+ to_arrow_type(returnType, timezone="UTC")
+ except TypeError:
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={
+ "feature": f"Invalid return type with grouped
aggregate Arrow UDFs: "
+ f"{returnType}"
+ },
+ )
+
def __call__(self, *args: Any, **kwargs: Any) -> Any:
"""
Call the UDF with Column arguments.
@@ -161,121 +379,18 @@ def udf(
return _udf_decorator
-def pandas_udf(
- f: Optional[Callable[..., Any]] = None,
- returnType: Optional[DataType] = None,
- functionType: Optional[str] = None,
-) -> Any:
- """
- Create a pandas UDF.
-
- Parameters
- ----------
- f : callable, optional
- The Python function to wrap as a pandas UDF.
- returnType : DataType, optional
- The return type of the UDF. Defaults to StringType().
- functionType : str, optional
- The type of pandas UDF. Defaults to "scalar".
- Options: "scalar", "grouped_map", "grouped_agg", "cogrouped_map", ...
-
- Returns
- -------
- UserDefinedFunction or callable
- If f is provided, returns a UDF. If f is None, returns a decorator.
-
- Examples
- --------
- >>> from pyspark.sql import functions as F
- >>> from pyspark.sql.types import IntegerType
- >>> @F.pandas_udf(IntegerType())
- ... def inc_id(s):
- ... return s + 1
- >>> result = spark.range(3).select(inc_id(F.col('id')).alias('inc_id'))
- """
- # Decorator form @pandas_udf(returnType) / @pandas_udf("ddl"): the first
- # positional is the return type, not the function.
- if f is not None and not callable(f):
- returnType, f = f, None
-
- if returnType is None:
- returnType = StringType()
-
- # Map function type names to eval types
- eval_type_map = {
- "scalar": 200, # SQL_SCALAR_PANDAS_UDF
- "grouped_map": 201, # SQL_GROUPED_MAP_PANDAS_UDF
- "grouped_agg": 202, # SQL_GROUPED_AGG_PANDAS_UDF
- "window_agg": 203, # SQL_WINDOW_AGG_PANDAS_UDF
- "scalar_iter": 204, # SQL_SCALAR_PANDAS_ITER_UDF
- "map_iter": 205, # SQL_MAP_PANDAS_ITER_UDF
- "cogrouped_map": 206, # SQL_COGROUPED_MAP_PANDAS_UDF
- }
-
- def _pandas_udf_decorator(func):
- et = eval_type_map.get(functionType) if functionType is not None else
None
- if et is None:
- # Modern form @pandas_udf("type"): infer the eval type from the
function's
- # pandas type hints (Series->Series = scalar,
Iterator[Series]->Iterator =
- # scalar_iter, Series->scalar = grouped_agg), mirroring
- # pyspark.sql.pandas.functions.pandas_udf.
- try:
- from inspect import signature
- from typing import get_type_hints
- from pyspark.sql.pandas.typehints import infer_eval_type
-
- et = infer_eval_type(signature(func), get_type_hints(func))
- except Exception:
- et = None
- if et is None:
- et = 200 # SQL_SCALAR_PANDAS_UDF
- return UserDefinedFunction(func, returnType, et)
-
- if f is not None:
- # Direct call
- return _pandas_udf_decorator(f)
- else:
- # Decorator call
- return _pandas_udf_decorator
-
-
-def arrow_udf(
- f: Optional[Callable[..., Any]] = None,
- returnType: Optional[DataType] = None,
- functionType: str = "scalar",
-) -> Any:
- """
- Create an Arrow user-defined function, mirroring
``pyspark.sql.functions.arrow_udf``.
-
- Arrow UDFs transfer data via Arrow and operate on ``pyarrow.Array``
values. This is
- the Arrow analogue of :func:`pandas_udf`.
-
- Parameters
- ----------
- f : callable, optional
- The Python function to wrap.
- returnType : DataType, optional
- Return type; defaults to StringType().
- functionType : str, optional
- "scalar" (default) or "scalar_iter".
- """
- # Decorator form @arrow_udf(returnType) / @arrow_udf("ddl").
- if f is not None and not callable(f):
- returnType, f = f, None
-
- if returnType is None:
- returnType = StringType()
-
- # SQL_SCALAR_ARROW_UDF = 250, SQL_SCALAR_ARROW_ITER_UDF = 251.
- eval_type_map = {"scalar": 250, "scalar_iter": 251}
- evalType = eval_type_map.get(functionType, 250)
-
- def _arrow_udf_decorator(func):
- return UserDefinedFunction(func, returnType, evalType)
-
- if f is not None:
- return _arrow_udf_decorator(f)
- return _arrow_udf_decorator
+def _create_udf(
+ f: Callable[..., Any],
+ returnType: Any,
+ evalType: int,
+ name: Optional[str] = None,
+ deterministic: bool = True,
+) -> UserDefinedFunction:
+ """The UDF of a function, its return type and eval type, as PySpark's
``_create_udf``
+ makes it; ``pyspark.sql.pandas.functions`` creates pandas and Arrow UDFs
with it."""
+ return UserDefinedFunction(
+ f, returnType=returnType, evalType=evalType, name=name,
deterministic=deterministic
+ )
class UDFRegistration:
diff --git a/python/pyspark/sql/udtf.py b/python/pyspark/sql/udtf.py
index 152fc71..47748f1 100644
--- a/python/pyspark/sql/udtf.py
+++ b/python/pyspark/sql/udtf.py
@@ -25,9 +25,11 @@ from pyspark._pyspark import (
AnalyzeResult, SkipRestOfInputTableException,
)
+import inspect
import sys
from typing import Any, Optional, Type
+from pyspark.errors import PySparkAttributeError, PySparkTypeError
from pyspark.serializers import CloudPickleSerializer
# Eval types for Python UDTFs (mirrors pyspark.util.PythonEvalType).
@@ -35,6 +37,52 @@ SQL_TABLE_UDF = 300
SQL_ARROW_TABLE_UDF = 301
+# Verbatim from PySpark (pyspark.sql.udtf): validates the handler class of a
UDTF.
+def _validate_udtf_handler(cls: Any, returnType: Optional[Any]) -> None:
+ """Validate the handler class of a UDTF."""
+
+ if not isinstance(cls, type):
+ raise PySparkTypeError(
+ errorClass="INVALID_UDTF_HANDLER_TYPE", messageParameters={"type":
type(cls).__name__}
+ )
+
+ if not hasattr(cls, "eval"):
+ raise PySparkAttributeError(
+ errorClass="INVALID_UDTF_NO_EVAL", messageParameters={"name":
cls.__name__}
+ )
+
+ has_analyze = hasattr(cls, "analyze")
+ has_analyze_staticmethod = has_analyze and isinstance(
+ inspect.getattr_static(cls, "analyze"), staticmethod
+ )
+ if returnType is None and not has_analyze_staticmethod:
+ raise PySparkAttributeError(
+ errorClass="INVALID_UDTF_RETURN_TYPE", messageParameters={"name":
cls.__name__}
+ )
+ if returnType is not None and has_analyze:
+ raise PySparkAttributeError(
+ errorClass="INVALID_UDTF_BOTH_RETURN_TYPE_AND_ANALYZE",
+ messageParameters={"name": cls.__name__},
+ )
+
+
+# Verbatim from PySpark Connect (pyspark.sql.connect.udtf): validates the
handler class of a
+# PyArrow UDTF.
+def _validate_arrow_udtf_handler(cls: Any, returnType: Optional[Any]) -> None:
+ """Validate the handler class of a PyArrow UDTF."""
+ # First run standard UDTF validation
+ _validate_udtf_handler(cls, returnType)
+
+ # Block analyze method usage in arrow UDTFs
+ # TODO(SPARK-53286): Support analyze method for Arrow UDTFs to enable
dynamic return types
+ has_analyze = hasattr(cls, "analyze")
+ if has_analyze:
+ raise PySparkAttributeError(
+ errorClass="INVALID_ARROW_UDTF_WITH_ANALYZE",
+ messageParameters={"name": cls.__name__},
+ )
+
+
class UserDefinedTableFunction:
"""A user-defined table function: a Python handler class that yields rows.
@@ -51,6 +99,8 @@ class UserDefinedTableFunction:
evalType: int = SQL_TABLE_UDF,
deterministic: bool = True,
):
+ # As PySpark's UserDefinedTableFunction, for udtf(), arrow_udtf() and
register().
+ _validate_udtf_handler(func, returnType)
self.func = func
self.returnType = returnType
self.evalType = evalType
@@ -130,6 +180,7 @@ def arrow_udtf(
operates on Arrow data (``SQL_ARROW_TABLE_UDF``).
"""
def _arrow_udtf_decorator(handler: Type) -> UserDefinedTableFunction:
+ _validate_arrow_udtf_handler(handler, returnType)
return UserDefinedTableFunction(
handler, returnType=returnType, evalType=SQL_ARROW_TABLE_UDF
)
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 755c2fb..d8c9453 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -424,7 +424,8 @@ def test_functions_udtf_and_arrow_decorators():
# functions.arrow_udf builds an Arrow scalar UDF.
au = F.arrow_udf(lambda x: x, IntegerType())
assert callable(au)
- aui = F.arrow_udf(lambda x: x, IntegerType(), functionType="scalar_iter")
+ # As in PySpark, the function type is a value of ArrowUDFType.
+ aui = F.arrow_udf(lambda x: x, IntegerType(),
functionType=F.ArrowUDFType.SCALAR_ITER)
assert callable(aui)
@@ -471,7 +472,9 @@ def test_functions_exports_fresh_process():
part =
importlib.import_module("pyspark.sql.functions.partitioning")
assert part is F.partitioning is sys.modules[part.__name__]
assert part.__all__ == ["years", "months", "days", "hours",
"bucket"]
- assert all(getattr(part, name) is getattr(F, name) for name in
part.__all__)
+ # As in PySpark, distinct from those of pyspark.sql.functions,
which are deprecated
+ # and call them.
+ assert all(getattr(part, name) is not getattr(F, name) for name in
part.__all__)
assert F.random is F.rand
assert F.column is not F.col
assert list(inspect.signature(F.first).parameters) == ["col",
"ignorenulls"]
@@ -1262,11 +1265,108 @@ def test_udf_nondeterministic_and_evaltypes():
# useArrow path selects the arrow eval type.
a = udf(lambda x: x, T.IntegerType(), useArrow=True)
assert callable(a)
- # pandas_udf with an explicit functionType.
- p = pandas_udf(lambda s: s, T.IntegerType(), functionType="scalar")
+ # pandas_udf with an explicit functionType, a value of PandasUDFType as in
PySpark.
+ from pyspark.sql.functions import PandasUDFType
+
+ p = pandas_udf(lambda s: s, T.IntegerType(),
functionType=PandasUDFType.SCALAR)
assert p is not None
+class _EvalHandler:
+ def eval(self, x):
+ yield (x,)
+
+
+class _AnalyzeHandler:
+ def eval(self, x):
+ yield (x,)
+
+ @staticmethod
+ def analyze(x):
+ pass
+
+
+_NOT_A_COLUMN = {"expected_type": "Column", "arg_name": "condition"}
+
+
[email protected]("call,error,condition,parameters", [
+ (lambda F: F.when("a", 1), "PySparkTypeError", "NOT_EXPECTED_TYPE",
+ {**_NOT_A_COLUMN, "arg_type": "str"}),
+ (lambda F: F.when(True, 1), "PySparkTypeError", "NOT_EXPECTED_TYPE",
+ {**_NOT_A_COLUMN, "arg_type": "bool"}),
+ (lambda F: F.when(F.col("a") > 1, 1).when("b", 2), "PySparkTypeError",
"NOT_EXPECTED_TYPE",
+ {**_NOT_A_COLUMN, "arg_type": "str"}),
+ (lambda F: F.col("a").when(F.col("b") > 1, 1), "PySparkTypeError",
"INVALID_WHEN_USAGE", {}),
+ (lambda F: F.when(F.col("a") > 1, 1).otherwise(0).when(F.col("b") > 1, 2),
+ "PySparkTypeError", "INVALID_WHEN_USAGE", {}),
+ (lambda F: F.partitioning.bucket("n", "c"), "PySparkTypeError",
"NOT_EXPECTED_TYPE",
+ {"expected_type": "Column or int", "arg_name": "numBuckets", "arg_type":
"str"}),
+ (lambda F: F.partitioning.months(3), "PySparkTypeError",
"NOT_EXPECTED_TYPE",
+ {"expected_type": "Column or str", "arg_name": "col", "arg_type": "int"}),
+ (lambda F: F.pandas_udf(), "PySparkTypeError", "CANNOT_BE_NONE",
{"arg_name": "returnType"}),
+ (lambda F: F.pandas_udf(lambda s: s), "PySparkTypeError", "CANNOT_BE_NONE",
+ {"arg_name": "returnType"}),
+ (lambda F: F.pandas_udf(lambda s: s, "long", 3), "PySparkTypeError",
"INVALID_PANDAS_UDF_TYPE",
+ {"arg_name": "functionType", "arg_type": "3"}),
+ (lambda F: F.arrow_udf(), "PySparkTypeError", "CANNOT_BE_NONE",
{"arg_name": "returnType"}),
+ (lambda F: F.arrow_udf("long", 3), "PySparkTypeError",
"INVALID_PANDAS_UDF_TYPE",
+ {"arg_name": "functionType", "arg_type": "3"}),
+ (lambda F: F.udtf(3, returnType="x int"), "PySparkTypeError",
"INVALID_UDTF_HANDLER_TYPE",
+ {"type": "int"}),
+ (lambda F: F.arrow_udtf(3), "PySparkTypeError",
"INVALID_UDTF_HANDLER_TYPE", {"type": "int"}),
+ (lambda F: F.udtf(type("T", (), {}), returnType="x int"),
"PySparkAttributeError",
+ "INVALID_UDTF_NO_EVAL", {"name": "T"}),
+ (lambda F: F.udtf(_EvalHandler), "PySparkAttributeError",
"INVALID_UDTF_RETURN_TYPE",
+ {"name": "_EvalHandler"}),
+ (lambda F: F.udtf(_AnalyzeHandler, returnType="x int"),
"PySparkAttributeError",
+ "INVALID_UDTF_BOTH_RETURN_TYPE_AND_ANALYZE", {"name": "_AnalyzeHandler"}),
+ (lambda F: F.arrow_udtf(_AnalyzeHandler), "PySparkAttributeError",
+ "INVALID_ARROW_UDTF_WITH_ANALYZE", {"name": "_AnalyzeHandler"}),
+])
+def test_functions_check_their_arguments_as_pyspark(call, error, condition,
parameters):
+ # Before any request, with PySpark's errors; the cases agree with official
PySpark 4.3.0.
+ from pyspark.errors import PySparkException
+ from pyspark.sql import functions as F
+
+ with pytest.raises(PySparkException) as raised:
+ call(F)
+ assert type(raised.value).__name__ == error
+ assert raised.value.getCondition() == condition
+ assert raised.value.getMessageParameters() == parameters
+
+
+def test_when_and_otherwise_build_one_case_when_as_pyspark():
+ from pyspark.errors import PySparkTypeError
+ from pyspark.sql import functions as F
+
+ branches = F.when(F.col("a") > 1, 1).when(F.col("b") > 1, F.col("b"))
+ assert repr(branches) == "Column<'CASE WHEN >(a, 1) THEN 1 WHEN >(b, 1)
THEN b END'>"
+ assert repr(branches.otherwise(0)) == (
+ "Column<'CASE WHEN >(a, 1) THEN 1 WHEN >(b, 1) THEN b ELSE 0 END'>")
+ # As PySpark's Column.otherwise, with a message rather than a condition.
+ with pytest.raises(PySparkTypeError, match=r"^otherwise\(\) can only be
applied on a Column"):
+ F.col("a").otherwise(0)
+ with pytest.raises(PySparkTypeError, match=r"^otherwise\(\) can only be
applied once"):
+ branches.otherwise(0).otherwise(2)
+
+
+def test_partition_transforms_are_those_of_pyspark_and_the_others_deprecated():
+ import warnings
+
+ import pyspark.sql.functions.partitioning as P
+ from pyspark.sql import functions as F
+
+ with pytest.warns(FutureWarning, match="use partitioning.years instead"):
+ assert repr(F.years("c")) == "Column<'years(c)'>"
+ with pytest.warns(FutureWarning, match="use partitioning.bucket instead"):
+ assert repr(F.bucket(2, "c")) == "Column<'bucket(2, c)'>"
+ with warnings.catch_warnings():
+ warnings.simplefilter("error")
+ assert repr(P.days("c")) == "Column<'days(c)'>"
+ assert repr(F.partitioning.bucket(F.lit(2), "c")) ==
"Column<'bucket(2, c)'>"
+ assert P.bucket is F.partitioning.bucket
+
+
def test_udtf_construction():
from pyspark.sql.udtf import udtf, UserDefinedTableFunction,
UDTFRegistration
@@ -1446,11 +1546,22 @@ def test_udf_defaults_and_registration():
# returnType defaults to StringType
u = udf(lambda x: x)
assert callable(u)
- p = pandas_udf(lambda s: s) # default returnType
- assert p is not None
- # pandas_udf grouped_map/other functionType branches
- for ft in ["scalar", "grouped_map", "grouped_agg"]:
- assert pandas_udf(lambda s: s, T.IntegerType(), functionType=ft) is
not None
+ # As in PySpark, a pandas UDF takes no default return type, and a grouped
map one a
+ # struct type.
+ from pyspark.errors import PySparkTypeError
+ from pyspark.sql.functions import PandasUDFType
+
+ with pytest.raises(PySparkTypeError, match="CANNOT_BE_NONE"):
+ pandas_udf(lambda s: s)
+ struct = T.StructType([T.StructField("a", T.IntegerType())])
+ for ft, rt in [
+ (PandasUDFType.SCALAR, T.IntegerType()),
+ (PandasUDFType.GROUPED_MAP, struct),
+ (PandasUDFType.GROUPED_AGG, T.IntegerType()),
+ ]:
+ assert pandas_udf(lambda s: s, rt, functionType=ft).evalType == ft
+ with pytest.raises(PySparkTypeError,
match="INVALID_RETURN_TYPE_FOR_PANDAS_UDF"):
+ pandas_udf(lambda s: s, T.IntegerType(),
functionType=PandasUDFType.GROUPED_MAP)
# UDFRegistration.register wires to the session's _registerPythonUdf (so
the name
# resolves in SQL) and returns the UDF.
from pyspark.sql.udf import UDFRegistration
@@ -2885,7 +2996,9 @@ def test_pandas_udf_grouped_agg():
from pyspark.sql.types import IntegerType
# Create pandas_udf without decorator to avoid execution issues
- p = pandas_udf(lambda s: s, IntegerType(), functionType="grouped_agg")
+ from pyspark.sql.functions import PandasUDFType
+
+ p = pandas_udf(lambda s: s, IntegerType(),
functionType=PandasUDFType.GROUPED_AGG)
assert callable(p)
@@ -2899,7 +3012,9 @@ def test_arrow_udf_types():
assert callable(u1)
# Scalar iter UDF
- u2 = arrow_udf(lambda x: x, IntegerType(), functionType="scalar_iter")
+ from pyspark.sql.functions import ArrowUDFType
+
+ u2 = arrow_udf(lambda x: x, IntegerType(),
functionType=ArrowUDFType.SCALAR_ITER)
assert callable(u2)
diff --git a/scripts/check_vendored_upstream.sh
b/scripts/check_vendored_upstream.sh
index d837858..108600c 100644
--- a/scripts/check_vendored_upstream.sh
+++ b/scripts/check_vendored_upstream.sh
@@ -139,6 +139,8 @@ VENDORED_FILES=(
"sql/pandas/typehints.py"
"sql/pandas/types.py"
"sql/pandas/utils.py"
+ # pandas_udf and arrow_udf, which create the UDF with
pyspark.sql.connect.udf._create_udf.
+ "sql/pandas/functions.py"
# SQL: core utilities
"sql/variant_utils.py"
diff --git a/scripts/gen_fn_dispatch.py b/scripts/gen_fn_dispatch.py
index d221745..6aebb90 100644
--- a/scripts/gen_fn_dispatch.py
+++ b/scripts/gen_fn_dispatch.py
@@ -149,6 +149,14 @@ skip_functions = {
# count_distinct(col, *cols) and countDistinct: a distinct count of all
the columns.
"count_distinct",
"countDistinct",
+ # when (a CASE WHEN), and the deprecated partition transforms, which call
those of
+ # pyspark.sql.functions.partitioning: written in functions.py as PySpark
Connect's.
+ "when",
+ "years",
+ "months",
+ "days",
+ "hours",
+ "bucket",
}
# Categorize functions by their parameter types
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]