This is an automated email from the ASF dual-hosted git repository.
Yicong-Huang pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 4835344f3810 [SPARK-58601][4.2][PYTHON][TEST] Guard
mapInPandas/mapInArrow list-return tests with the legacy acceptAnyIterable flag
4835344f3810 is described below
commit 4835344f38103322dff501f582dbead4b6fa2472
Author: Yicong Huang <[email protected]>
AuthorDate: Wed Aug 26 09:56:18 2026 +0000
[SPARK-58601][4.2][PYTHON][TEST] Guard mapInPandas/mapInArrow list-return
tests with the legacy acceptAnyIterable flag
### What changes were proposed in this pull request?
This mirrors #58286 (branch-4.0) onto branch-4.2: it moves the
`mapInPandas` and `mapInArrow` list-return (non-`Iterator` iterable) test cases
under the
`spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled=true`
flag, and leaves the default `test_map_in_pandas` / `test_map_in_arrow` cases
returning a strict iterator. This matches the return-value contract introduced
on master by SPARK-58601 (strict `Iterator` by default; any iterable accepted
only under the legac [...]
### Why are the changes needed?
The connect old-client compatibility job runs a release branch's Python
tests against a `master` Spark Connect server. On master, SPARK-58601 tightened
`mapInPandas` / `mapInArrow` to require a strict `Iterator` return by default,
so a release branch's `test_map_in_pandas`, which returned a bare `list`, fails
against that server with `UDF_RETURN_TYPE`. #58286 fixed this for branch-4.0;
this PR keeps branch-4.2 consistent so the same tests stay green if the
compatibility job's target a [...]
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
Existing tests, restructured. `test_map_in_pandas` / `test_map_in_arrow`
assert the default iterator path; the new
`test_map_in_pandas_legacy_accept_any_iterable` /
`test_map_in_arrow_legacy_accept_any_iterable` assert the list path under the
legacy flag.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #58305 from Yicong-Huang/SPARK-58601-oldclient-test-42.
Authored-by: Yicong Huang <[email protected]>
Signed-off-by: Yicong-Huang <[email protected]>
---
python/pyspark/sql/tests/arrow/test_arrow_map.py | 13 +++++++++++++
python/pyspark/sql/tests/pandas/test_pandas_map.py | 14 +++++++++-----
2 files changed, 22 insertions(+), 5 deletions(-)
diff --git a/python/pyspark/sql/tests/arrow/test_arrow_map.py
b/python/pyspark/sql/tests/arrow/test_arrow_map.py
index 8dd095a176b9..c0ccb869b1bd 100644
--- a/python/pyspark/sql/tests/arrow/test_arrow_map.py
+++ b/python/pyspark/sql/tests/arrow/test_arrow_map.py
@@ -55,6 +55,19 @@ class MapInArrowTestsMixin:
expected = df.collect()
self.assertEqual(actual, expected)
+ def test_map_in_arrow_legacy_accept_any_iterable(self):
+ # With the legacy flag enabled, returning a non-Iterator iterable
(e.g. list) is accepted.
+ def list_not_iter(iterator):
+ return [batch for batch in iterator]
+
+ with self.sql_conf(
+
{"spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled":
True}
+ ):
+ df = self.spark.range(10)
+ actual = df.mapInArrow(list_not_iter, "id long").collect()
+ expected = df.collect()
+ self.assertEqual(actual, expected)
+
def test_map_in_arrow_with_limit(self):
def get_size(iterator):
for batch in iterator:
diff --git a/python/pyspark/sql/tests/pandas/test_pandas_map.py
b/python/pyspark/sql/tests/pandas/test_pandas_map.py
index bfcedc6c8899..19d6ea6ce851 100644
--- a/python/pyspark/sql/tests/pandas/test_pandas_map.py
+++ b/python/pyspark/sql/tests/pandas/test_pandas_map.py
@@ -86,11 +86,15 @@ class MapInPandasTestsMixin:
expected = df.collect()
self.assertEqual(actual, expected)
- # test returning list of DataFrames
- df = self.spark.range(10, numPartitions=3)
- actual = df.mapInPandas(lambda it: [pdf for pdf in it], "id
long").collect()
- expected = df.collect()
- self.assertEqual(actual, expected)
+ def test_map_in_pandas_legacy_accept_any_iterable(self):
+ # With the legacy flag enabled, returning a non-Iterator iterable
(e.g. list) is accepted.
+ with self.sql_conf(
+
{"spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled":
True}
+ ):
+ df = self.spark.range(10, numPartitions=3)
+ actual = df.mapInPandas(lambda it: [pdf for pdf in it], "id
long").collect()
+ expected = df.collect()
+ self.assertEqual(actual, expected)
def test_multiple_columns(self):
data = [(1, "foo"), (2, None), (3, "bar"), (4, "bar")]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
