This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new fb7ae92  [SPARK-60971][CONNECT][PYTHON] Infer the iterator forms of 
applyInPandas and applyInArrow from type hints, and validate map functions, as 
PySpark does
fb7ae92 is described below

commit fb7ae92e2593f4f9ad690583237f2455527e32ee
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 13:12:00 2026 +0900

    [SPARK-60971][CONNECT][PYTHON] Infer the iterator forms of applyInPandas 
and applyInArrow from type hints, and validate map functions, as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60971](https://issues.apache.org/jira/browse/SPARK-60971).
    
    **Eval types from type hints.** `GroupedData.applyInPandas` and 
`applyInArrow` now take the eval type of the function from its type hints, as 
PySpark 4.3.0 does. The inference uses `infer_group_pandas_eval_type_from_func` 
and `infer_group_arrow_eval_type_from_func` of the vendored 
`pyspark.sql.pandas.typehints`:
    
    | Function | Eval type |
    | --- | --- |
    | `(Iterator[pd.DataFrame]) -> Iterator[pd.DataFrame]`, optionally after a 
key | `SQL_GROUPED_MAP_PANDAS_ITER_UDF` (216) |
    | `(Iterator[pa.RecordBatch]) -> Iterator[pa.RecordBatch]`, optionally 
after a key | `SQL_GROUPED_MAP_ARROW_ITER_UDF` (215) |
    | other hints, or none | `SQL_GROUPED_MAP_PANDAS_UDF` (201) or 
`SQL_GROUPED_MAP_ARROW_UDF` (209), as before |
    
    When the hints cannot be read, these functions warn as PySpark does 
(`Cannot infer the eval type from type hints.`) and use the default form. 
PySpark 4.3.0's `applyInArrow` then raises `UnboundLocalError`, as its 
`eval_type` is unbound there; here it runs with the default form, as 
`applyInPandas` does.
    
    **Validation.** The map functions now validate their function as PySpark 
does, before they build the plan or analyze the input columns. They use 
`_validate_vectorized_udf`, which `pyspark.sql.pandas.functions` now has 
verbatim from PySpark. This covers:
    - the grouped and cogrouped `applyInPandas` and `applyInArrow`;
    - `applyInPandasWithState`;
    - `mapInPandas` and `mapInArrow`.
    
    For example, a grouped function of three arguments raises 
`INVALID_PANDAS_UDF`, and a value that is not callable raises `TypeError: 
unsupported callable`.
    
    ### Why are the changes needed?
    
    - A function of an iterator, which PySpark has supported since 4.1.0, got 
the eval type of the other form, so it failed on the Python worker with 
`PythonException`. For example: `def f(it: Iterator[pd.DataFrame]) -> 
Iterator[pd.DataFrame]` with `df.groupBy("k").applyInPandas(f, schema)`.
    - A function that PySpark rejects up front was sent to the server and 
failed there.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, these functions infer the eval type and validate the function as in 
PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - Against a local Spark 4.3.0 server, a probe of 18 cases gives the same 
results and warnings as official PySpark 4.3.0. The cases cover the iterator 
and plain forms, with and without keys and hints, partial hints, wrong arities, 
and values that are not callable, across the six functions. The one difference 
is the `applyInArrow` case above. Before this change, 15 of the 18 differed.
    - New offline tests check:
      - the eval type sent for each form;
      - the warning for partial hints;
      - the validation errors, raised before any request to the server.
    
      Without this change, 14 of their 18 cases fail.
    - A new live test runs the iterator forms, with and without keys. It also 
passes with official PySpark 4.3.0 and on a server started with 
`SPARK_SKIP_CONNECT_COMPAT_TESTS`, as CI's is.
    - With each module under `pyspark.sql` imported first, 
`_validate_vectorized_udf`, the UDF types and `pyspark.sql.conversion` import.
    - The lib tests and every integration test, e2e included, against the local 
server (643 passed), the build of all tests with `wasm-udf`, the offline Python 
suites (634 passed, 8 live cases skipped), and the live drop-in and Arrow 
suites (58 passed) and live cases (10 passed).
    - Clippy reports no new diagnostics. It flags the `input_columns` closures 
of two changed lines, which the diff shows as moved; it reports the same lint 
at every such closure in `group.rs`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #184 from HyukjinKwon/SPARK-60971-group-map-eval-types.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs     |  42 ++++++++++
 crates/pyspark-rs/src/group.rs         |  30 ++++---
 python/pyspark/sql/pandas/functions.py | 149 ++++++++++++++++++++++++++++++++-
 python/tests/test_dropin_e2e.py        |  34 ++++++++
 python/tests/test_dropin_offline.py    | 134 +++++++++++++++++++++++++++++
 5 files changed, 377 insertions(+), 12 deletions(-)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index faa8b58..6e5be21 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -1513,6 +1513,7 @@ impl PyDataFrame {
         barrier: bool,
         profile: Option<Bound<'_, PyAny>>,
     ) -> PyResult<PyDataFrame> {
+        validate_map_function(py, &func, eval_type::SQL_MAP_PANDAS_ITER_UDF)?;
         let cols = py.detach(|| self.dataframe.columns()).to_pyerr()?;
         let udf = build_map_udf(
             py,
@@ -1537,6 +1538,7 @@ impl PyDataFrame {
         barrier: bool,
         profile: Option<Bound<'_, PyAny>>,
     ) -> PyResult<PyDataFrame> {
+        validate_map_function(py, &func, eval_type::SQL_MAP_ARROW_ITER_UDF)?;
         let cols = py.detach(|| self.dataframe.columns()).to_pyerr()?;
         let udf = build_map_udf(
             py,
@@ -2453,6 +2455,46 @@ pub(crate) fn resolve_datatype(
     ))
 }
 
+/// Validate the function of a map UDF for its eval type with
+/// `pyspark.sql.pandas.functions._validate_vectorized_udf`, as the reference 
client does
+/// before it builds the UDF.
+pub(crate) fn validate_map_function(
+    py: Python<'_>,
+    func: &Bound<'_, PyAny>,
+    eval_type_id: i32,
+) -> PyResult<()> {
+    py.import("pyspark.sql.pandas.functions")?
+        .call_method1("_validate_vectorized_udf", (func, eval_type_id))?;
+    Ok(())
+}
+
+/// The eval type of a grouped map function from its type hints, by `infer` of
+/// `pyspark.sql.pandas.typehints` (the iterator forms, for example), as the 
reference
+/// client infers it. `default` is used when the hints give none, and when 
they cannot be
+/// read, after the reference client's `warning`.
+pub(crate) fn infer_group_map_eval_type(
+    py: Python<'_>,
+    func: &Bound<'_, PyAny>,
+    infer: &str,
+    warning: &str,
+    default: i32,
+) -> PyResult<i32> {
+    match py
+        .import("pyspark.sql.pandas.typehints")?
+        .call_method1(infer, (func,))
+    {
+        Ok(inferred) if inferred.is_none() => Ok(default),
+        Ok(inferred) => inferred.extract(),
+        Err(err) if err.is_instance_of::<pyo3::exceptions::PyException>(py) => 
{
+            let category = py.get_type::<pyo3::exceptions::PyUserWarning>();
+            py.import("warnings")?
+                .call_method1("warn", (warning, category))?;
+            Ok(default)
+        }
+        Err(err) => Err(err),
+    }
+}
+
 /// Build a map/group UDF expression: cloudpickle `(func, schema)` as the 
command and
 /// carry the resolved schema as the output type (mirrors the scalar UDF 
command layout).
 pub(crate) fn build_map_udf(
diff --git a/crates/pyspark-rs/src/group.rs b/crates/pyspark-rs/src/group.rs
index 21ce32b..4f23c3c 100644
--- a/crates/pyspark-rs/src/group.rs
+++ b/crates/pyspark-rs/src/group.rs
@@ -4,7 +4,9 @@ use pyo3::prelude::*;
 use spark_connect::group::{CoGroupedData as RustCoGroupedData, GroupedData as 
RustGroupedData};
 use spark_connect::udf::eval_type;
 
-use crate::dataframe::{build_map_udf, PyDataFrame};
+use crate::dataframe::{
+    build_map_udf, infer_group_map_eval_type, validate_map_function, 
PyDataFrame,
+};
 use crate::errors::ResultExt;
 use crate::functions::to_column;
 
@@ -151,15 +153,16 @@ impl PyGroupedData {
         func: Bound<'_, PyAny>,
         schema: Bound<'_, PyAny>,
     ) -> PyResult<PyDataFrame> {
-        let cols = py.detach(|| self.grouped_data.input_columns()).to_pyerr()?;
-        let udf = build_map_udf(
+        let eval_type_id = infer_group_map_eval_type(
             py,
-            "applyInPandas",
             &func,
-            &schema,
+            "infer_group_pandas_eval_type_from_func",
+            "Cannot infer the eval type from type hints.",
             eval_type::SQL_GROUPED_MAP_PANDAS_UDF,
-            &cols,
         )?;
+        validate_map_function(py, &func, eval_type_id)?;
+        let cols = py.detach(|| self.grouped_data.input_columns()).to_pyerr()?;
+        let udf = build_map_udf(py, "applyInPandas", &func, &schema, 
eval_type_id, &cols)?;
         Ok(PyDataFrame::new(self.grouped_data.apply_in_pandas(udf)))
     }
 
@@ -184,15 +187,17 @@ impl PyGroupedData {
         func: Bound<'_, PyAny>,
         schema: Bound<'_, PyAny>,
     ) -> PyResult<PyDataFrame> {
-        let cols = py.detach(|| self.grouped_data.input_columns()).to_pyerr()?;
-        let udf = build_map_udf(
+        // The reference client's warning, with its trailing space.
+        let eval_type_id = infer_group_map_eval_type(
             py,
-            "applyInArrow",
             &func,
-            &schema,
+            "infer_group_arrow_eval_type_from_func",
+            "Cannot infer the eval type from type hints. ",
             eval_type::SQL_GROUPED_MAP_ARROW_UDF,
-            &cols,
         )?;
+        validate_map_function(py, &func, eval_type_id)?;
+        let cols = py.detach(|| self.grouped_data.input_columns()).to_pyerr()?;
+        let udf = build_map_udf(py, "applyInArrow", &func, &schema, 
eval_type_id, &cols)?;
         Ok(PyDataFrame::new(self.grouped_data.apply_in_arrow(udf)))
     }
 
@@ -216,6 +221,7 @@ impl PyGroupedData {
         outputMode: &str,
         timeoutConf: &str,
     ) -> PyResult<PyDataFrame> {
+        validate_map_function(py, &func, 
eval_type::SQL_GROUPED_MAP_PANDAS_UDF_WITH_STATE)?;
         let cols = py.detach(|| self.grouped_data.input_columns()).to_pyerr()?;
         let udf = build_map_udf(
             py,
@@ -376,6 +382,7 @@ impl PyCoGroupedData {
         func: Bound<'_, PyAny>,
         schema: Bound<'_, PyAny>,
     ) -> PyResult<PyDataFrame> {
+        validate_map_function(py, &func, 
eval_type::SQL_COGROUPED_MAP_PANDAS_UDF)?;
         let cols = py.detach(|| self.inner.input_columns()).to_pyerr()?;
         let udf = build_map_udf(
             py,
@@ -396,6 +403,7 @@ impl PyCoGroupedData {
         func: Bound<'_, PyAny>,
         schema: Bound<'_, PyAny>,
     ) -> PyResult<PyDataFrame> {
+        validate_map_function(py, &func, 
eval_type::SQL_COGROUPED_MAP_ARROW_UDF)?;
         let cols = py.detach(|| self.inner.input_columns()).to_pyerr()?;
         let udf = build_map_udf(
             py,
diff --git a/python/pyspark/sql/pandas/functions.py 
b/python/pyspark/sql/pandas/functions.py
index d7d67de..fec7268 100644
--- a/python/pyspark/sql/pandas/functions.py
+++ b/python/pyspark/sql/pandas/functions.py
@@ -19,7 +19,13 @@
 Canonical home of the ``PandasUDFType`` / ``ArrowUDFType`` enums (mirroring 
reference
 pyspark); ``pyspark.sql.functions`` re-exports them from here.
 """
-from pyspark.util import PythonEvalType  # noqa: F401  
(pyspark.sql.pandas.typehints reads it here)
+import warnings
+from inspect import getfullargspec, signature
+from typing import get_type_hints
+
+from pyspark.errors import PySparkValueError
+from pyspark.sql.pandas.typehints import infer_eval_type
+from pyspark.util import PythonEvalType  # (pyspark.sql.pandas.typehints reads 
it here too)
 
 
 class PandasUDFType:
@@ -43,6 +49,147 @@ class ArrowUDFType:
     GROUPED_AGG_ITER = 254
 
 
+
+# Verbatim from PySpark: validates the function of a pandas or Arrow UDF for 
its eval type.
+def _validate_vectorized_udf(f, evalType, kind: str = "pandas") -> int:
+    assert kind in ["pandas", "arrow"], "kind should be either 'pandas' or 
'arrow'"
+
+    argspec = getfullargspec(f)
+
+    # pandas UDF by type hints.
+    if evalType in [
+        PythonEvalType.SQL_SCALAR_PANDAS_UDF,
+        PythonEvalType.SQL_SCALAR_PANDAS_ITER_UDF,
+        PythonEvalType.SQL_GROUPED_AGG_PANDAS_UDF,
+        PythonEvalType.SQL_GROUPED_AGG_PANDAS_ITER_UDF,
+    ]:
+        warnings.warn(
+            "In Python 3.6+ and Spark 3.0+, it is preferred to specify type 
hints for "
+            "pandas UDF instead of specifying pandas UDF type which will be 
deprecated "
+            "in the future releases. See SPARK-28264 for more details.",
+            UserWarning,
+        )
+    elif evalType in [
+        PythonEvalType.SQL_SCALAR_ARROW_UDF,
+        PythonEvalType.SQL_SCALAR_ARROW_ITER_UDF,
+        PythonEvalType.SQL_GROUPED_AGG_ARROW_UDF,
+        PythonEvalType.SQL_GROUPED_AGG_ARROW_ITER_UDF,
+    ]:
+        warnings.warn(
+            "It is preferred to specify type hints for "
+            "arrow UDF instead of specifying arrow UDF type.",
+            UserWarning,
+        )
+    elif evalType in [
+        PythonEvalType.SQL_GROUPED_MAP_PANDAS_UDF,
+        PythonEvalType.SQL_GROUPED_MAP_PANDAS_ITER_UDF,
+        PythonEvalType.SQL_MAP_PANDAS_ITER_UDF,
+        PythonEvalType.SQL_MAP_ARROW_ITER_UDF,
+        PythonEvalType.SQL_COGROUPED_MAP_PANDAS_UDF,
+        PythonEvalType.SQL_GROUPED_MAP_PANDAS_UDF_WITH_STATE,
+        PythonEvalType.SQL_TRANSFORM_WITH_STATE_PANDAS_UDF,
+        PythonEvalType.SQL_TRANSFORM_WITH_STATE_PANDAS_INIT_STATE_UDF,
+        PythonEvalType.SQL_TRANSFORM_WITH_STATE_PYTHON_ROW_UDF,
+        PythonEvalType.SQL_TRANSFORM_WITH_STATE_PYTHON_ROW_INIT_STATE_UDF,
+        PythonEvalType.SQL_GROUPED_MAP_ARROW_UDF,
+        PythonEvalType.SQL_GROUPED_MAP_ARROW_ITER_UDF,
+        PythonEvalType.SQL_COGROUPED_MAP_ARROW_UDF,
+        PythonEvalType.SQL_ARROW_BATCHED_UDF,
+    ]:
+        # In case of 'SQL_GROUPED_MAP_PANDAS_UDF', deprecation warning is 
being triggered
+        # at `apply` instead.
+        # In case of 'SQL_MAP_PANDAS_ITER_UDF', 'SQL_MAP_ARROW_ITER_UDF' and
+        # 'SQL_COGROUPED_MAP_PANDAS_UDF', the evaluation type will always be 
set.
+        # In case of 'SQL_ARROW_BATCHED_UDF', no deprecation warning is 
required since it is not
+        # exposed to users.
+        pass
+    elif len(argspec.annotations) > 0:
+        try:
+            type_hints = get_type_hints(f)
+        except NameError:
+            type_hints = {}
+        evalType = infer_eval_type(signature(f), type_hints, kind)
+        assert evalType is not None
+
+    if evalType is None:
+        # Set default is scalar UDF.
+        if kind == "pandas":
+            evalType = PythonEvalType.SQL_SCALAR_PANDAS_UDF
+        else:
+            evalType = PythonEvalType.SQL_SCALAR_ARROW_UDF
+
+    kind_str = "pandas_udfs" if kind == "pandas" else "arrow_udfs"
+
+    if (
+        (
+            evalType == PythonEvalType.SQL_SCALAR_PANDAS_UDF
+            or evalType == PythonEvalType.SQL_SCALAR_ARROW_UDF
+            or evalType == PythonEvalType.SQL_SCALAR_PANDAS_ITER_UDF
+            or evalType == PythonEvalType.SQL_SCALAR_ARROW_ITER_UDF
+        )
+        and len(argspec.args) == 0
+        and argspec.varargs is None
+    ):
+        raise PySparkValueError(
+            errorClass="INVALID_PANDAS_UDF",
+            messageParameters={
+                "detail": f"0-arg {kind_str} are not supported. "
+                f"Instead, create a 1-arg {kind_str} and ignore the arg in 
your function.",
+            },
+        )
+
+    if evalType == PythonEvalType.SQL_GROUPED_MAP_PANDAS_UDF and 
len(argspec.args) not in (1, 2):
+        raise PySparkValueError(
+            errorClass="INVALID_PANDAS_UDF",
+            messageParameters={
+                "detail": f"{kind_str} with function type GROUPED_MAP or the 
function in "
+                "groupby.applyInPandas must take either one argument (data) or 
"
+                "two arguments (key, data).",
+            },
+        )
+
+    if evalType == PythonEvalType.SQL_GROUPED_MAP_PANDAS_ITER_UDF and 
len(argspec.args) not in (
+        1,
+        2,
+    ):
+        raise PySparkValueError(
+            errorClass="INVALID_PANDAS_UDF",
+            messageParameters={
+                "detail": "the function in groupby.applyInPandas with iterator 
API must take "
+                "either one argument (batches: Iterator[pandas.DataFrame]) or 
two arguments "
+                "(key, batches: Iterator[pandas.DataFrame]).",
+            },
+        )
+
+    if evalType == PythonEvalType.SQL_GROUPED_MAP_ARROW_UDF and 
len(argspec.args) not in (1, 2):
+        raise PySparkValueError(
+            errorClass="INVALID_PANDAS_UDF",
+            messageParameters={
+                "detail": "the function in groupby.applyInArrow must take 
either one argument "
+                "(data) or two arguments (key, data).",
+            },
+        )
+
+    if evalType == PythonEvalType.SQL_COGROUPED_MAP_PANDAS_UDF and 
len(argspec.args) not in (2, 3):
+        raise PySparkValueError(
+            errorClass="INVALID_PANDAS_UDF",
+            messageParameters={
+                "detail": "the function in cogroup.applyInPandas must take 
either two arguments "
+                "(left, right) or three arguments (key, left, right).",
+            },
+        )
+
+    if evalType == PythonEvalType.SQL_COGROUPED_MAP_ARROW_UDF and 
len(argspec.args) not in (2, 3):
+        raise PySparkValueError(
+            errorClass="INVALID_PANDAS_UDF",
+            messageParameters={
+                "detail": "the function in cogroup.applyInArrow must take 
either two arguments "
+                "(left, right) or three arguments (key, left, right).",
+            },
+        )
+
+    return evalType
+
 # After the enums: importing this module first imports pyspark.sql.functions, 
whose
 # re-export of the enums then finds them already defined here.
 from pyspark.sql.functions import arrow_udf, pandas_udf  # noqa: E402,F401
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 4a4cd40..a245b21 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -356,6 +356,40 @@ def test_reader_schema_keeps_field_metadata(session, 
tmp_path, ddl):
     assert stream.schema.fields[0].metadata == {"comment": "the key"}
 
 
+def test_grouped_map_functions_take_iterators_by_their_type_hints(session):
+    from typing import Iterator, Tuple
+
+    import numpy as np
+    import pandas as pd
+
+    def scaled(batches: Iterator[pd.DataFrame]) -> Iterator[pd.DataFrame]:
+        for frame in batches:
+            yield frame.assign(v=frame.v * 10)
+
+    def shifted(key: Tuple[np.int64], batches: Iterator[pd.DataFrame]) -> 
Iterator[pd.DataFrame]:
+        for frame in batches:
+            yield frame.assign(v=frame.v + key[0])
+
+    def batches(it: Iterator[pa.RecordBatch]) -> Iterator[pa.RecordBatch]:
+        yield from it
+
+    def keyed_batches(
+        key: Tuple[pa.Scalar, ...], it: Iterator[pa.RecordBatch]
+    ) -> Iterator[pa.RecordBatch]:
+        yield from it
+
+    df = session.createDataFrame([(1, 1.0), (1, 2.0), (2, 3.0)], ["k", "v"])
+    schema = StructType([StructField("k", LongType()), StructField("v", 
DoubleType())])
+    grouped = df.groupBy("k")
+    for result, expected in (
+        (grouped.applyInPandas(scaled, schema), [(1, 10.0), (1, 20.0), (2, 
30.0)]),
+        (grouped.applyInPandas(shifted, schema), [(1, 2.0), (1, 3.0), (2, 
5.0)]),
+        (grouped.applyInArrow(batches, schema), [(1, 1.0), (1, 2.0), (2, 
3.0)]),
+        (grouped.applyInArrow(keyed_batches, schema), [(1, 1.0), (1, 2.0), (2, 
3.0)]),
+    ):
+        assert sorted(tuple(r) for r in result.collect()) == expected
+
+
 def test_collect_reads_a_result_across_stream_ends(session):
     # Takes several response streams when the server's senderMaxStreamSize is 
small.
     n = 300_000
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index bb0ffb8..23747e5 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -5232,6 +5232,140 @@ def 
test_exact_string_subclass_uses_generic_conversion():
             "data_type1": "StructType", "data_type2": data_type}
 
 
+# --------------------------------------------------------------------------- 
map functions
+
+
+def _group_map_functions():
+    from typing import Iterator, Tuple
+
+    import numpy as np
+    import pandas as pd
+    import pyarrow as pa
+
+    def pandas_iter(batches: Iterator[pd.DataFrame]) -> Iterator[pd.DataFrame]:
+        yield from batches
+
+    def pandas_iter_key(
+        key: Tuple[np.int64], batches: Iterator[pd.DataFrame]
+    ) -> Iterator[pd.DataFrame]:
+        yield from batches
+
+    def pandas_frame(frame: pd.DataFrame) -> pd.DataFrame:
+        return frame
+
+    def arrow_iter(batches: Iterator[pa.RecordBatch]) -> 
Iterator[pa.RecordBatch]:
+        yield from batches
+
+    def arrow_iter_key(
+        key: Tuple[pa.Scalar, ...], batches: Iterator[pa.RecordBatch]
+    ) -> Iterator[pa.RecordBatch]:
+        yield from batches
+
+    def arrow_table(table: pa.Table) -> pa.Table:
+        return table
+
+    def plain(data):
+        return data
+
+    return {f.__name__: f for f in (
+        pandas_iter, pandas_iter_key, pandas_frame, arrow_iter, 
arrow_iter_key, arrow_table, plain)}
+
+
+def _sent_group_map_eval_type(request):
+    """The eval type of the function in a captured ExecutePlanRequest: plan=3 
-> root=1 ->
+    group_map=31 -> func=3 -> python_udf=4 -> eval_type=2."""
+    root = parse_message(parse_message(parse_message(request)[3][0])[1][0])
+    udf = parse_message(parse_message(root[31][0])[3][0])
+    return parse_message(udf[4][0])[2][0]
+
+
+_ID_SCHEMA = T.StructType([T.StructField("id", T.LongType())])
+
+
[email protected]("method,name,eval_type", [
+    # SQL_GROUPED_MAP_PANDAS_ITER_UDF and SQL_GROUPED_MAP_PANDAS_UDF.
+    ("applyInPandas", "pandas_iter", 216),
+    ("applyInPandas", "pandas_iter_key", 216),
+    ("applyInPandas", "pandas_frame", 201),
+    ("applyInPandas", "plain", 201),
+    # SQL_GROUPED_MAP_ARROW_ITER_UDF and SQL_GROUPED_MAP_ARROW_UDF.
+    ("applyInArrow", "arrow_iter", 215),
+    ("applyInArrow", "arrow_iter_key", 215),
+    ("applyInArrow", "arrow_table", 209),
+    ("applyInArrow", "plain", 209),
+])
+def test_grouped_map_functions_infer_their_eval_type_as_pyspark(mock_connect, 
method, name, eval_type):
+    spark, service = mock_connect
+    grouped = spark.range(3).groupBy("id")
+    getattr(grouped, method)(_group_map_functions()[name], 
_ID_SCHEMA).collect()
+    assert _sent_group_map_eval_type(service.execute_requests[-1]) == eval_type
+
+
[email protected]("method,warning,eval_type", [
+    ("applyInPandas", "Cannot infer the eval type from type hints.", 201),
+    # PySpark 4.3.0 warns too, then raises UnboundLocalError, as its eval type 
is unbound;
+    # here the function keeps the default eval type, as for applyInPandas.
+    ("applyInArrow", "Cannot infer the eval type from type hints. ", 209),
+])
+def test_grouped_map_functions_warn_when_their_type_hints_cannot_be_inferred(
+    mock_connect, method, warning, eval_type
+):
+    import warnings
+
+    def partial(key, data: object) -> object:
+        return data
+
+    spark, service = mock_connect
+    with warnings.catch_warnings(record=True) as caught:
+        warnings.simplefilter("always")
+        getattr(spark.range(3).groupBy("id"), method)(partial, 
_ID_SCHEMA).collect()
+    assert [str(w.message) for w in caught if w.category is UserWarning] == 
[warning]
+    assert _sent_group_map_eval_type(service.execute_requests[-1]) == eval_type
+
+
[email protected]("build,detail", [
+    (lambda df: df.groupBy("id").applyInPandas(lambda a, b, c: a, _ID_SCHEMA),
+     "pandas_udfs with function type GROUPED_MAP or the function in 
groupby.applyInPandas "
+     "must take either one argument (data) or two arguments (key, data)."),
+    (lambda df: df.groupBy("id").applyInArrow(lambda a, b, c: a, _ID_SCHEMA),
+     "the function in groupby.applyInArrow must take either one argument 
(data) or two "
+     "arguments (key, data)."),
+    (lambda df: 
df.groupBy("id").cogroup(df.groupBy("id")).applyInPandas(lambda a: a, 
_ID_SCHEMA),
+     "the function in cogroup.applyInPandas must take either two arguments 
(left, right) or "
+     "three arguments (key, left, right)."),
+    (lambda df: df.groupBy("id").cogroup(df.groupBy("id")).applyInArrow(lambda 
a: a, _ID_SCHEMA),
+     "the function in cogroup.applyInArrow must take either two arguments 
(left, right) or "
+     "three arguments (key, left, right)."),
+])
+def test_map_functions_validate_their_arguments_as_pyspark(mock_connect, 
build, detail):
+    from pyspark.errors import PySparkValueError
+
+    spark, service = mock_connect
+    with pytest.raises(PySparkValueError) as error:
+        build(spark.range(3))
+    assert error.value.getCondition() == "INVALID_PANDAS_UDF"
+    assert error.value.getMessageParameters() == {"detail": detail}
+    # Validated before the input columns are analyzed, as in PySpark.
+    assert service.calls == 0
+
+
[email protected]("build", [
+    lambda df: df.mapInPandas(1, _ID_SCHEMA),
+    lambda df: df.mapInArrow(1, _ID_SCHEMA),
+    lambda df: df.groupBy("id").applyInPandas(1, _ID_SCHEMA),
+    lambda df: df.groupBy("id").applyInPandasWithState(1, _ID_SCHEMA, 
_ID_SCHEMA, "append", "NoTimeout"),
+])
+def test_map_functions_reject_what_is_not_callable_as_pyspark(mock_connect, 
build):
+    import warnings
+
+    spark, service = mock_connect
+    with warnings.catch_warnings():
+        warnings.simplefilter("ignore")
+        with pytest.raises(TypeError, match="^unsupported callable$"):
+            build(spark.range(3))
+    assert service.calls == 0
+
+
 # --------------------------------------------------------------------------- 
printSchema
 
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to