This is an automated email from the ASF dual-hosted git repository.
Yicong-Huang pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new c0f17e821255 [SPARK-58601][4.0][PYTHON][TEST] Guard
mapInPandas/mapInArrow list-return tests with the legacy acceptAnyIterable flag
c0f17e821255 is described below
commit c0f17e8212556d28eebea59431bad7dff63de07b
Author: Yicong Huang <[email protected]>
AuthorDate: Wed Aug 26 06:58:59 2026 +0000
[SPARK-58601][4.0][PYTHON][TEST] Guard mapInPandas/mapInArrow list-return
tests with the legacy acceptAnyIterable flag
### What changes were proposed in this pull request?
This moves the `mapInPandas` and `mapInArrow` list-return (non-`Iterator`
iterable) test cases under the
`spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled=true`
flag, and leaves the default `test_map_in_pandas` / `test_map_in_arrow` cases
returning a strict iterator. This mirrors the return-value contract introduced
on master by SPARK-58601 (strict `Iterator` by default; any iterable accepted
only under the legacy flag).
### Why are the changes needed?
The `pyspark-connect-old-client` CI job runs `branch-4.0`'s Python tests
against a `master` Spark Connect server. On master, SPARK-58601 tightened
`mapInPandas` / `mapInArrow` to require a strict `Iterator` return by default,
so `branch-4.0`'s `test_map_in_pandas`, which returned a bare `list`, now fails
against that server with `UDF_RETURN_TYPE`. Setting the legacy flag makes the
server accept the list, keeping the old-client job green once this lands. The
change is also safe on `bra [...]
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
Existing tests, restructured. `test_map_in_pandas` / `test_map_in_arrow`
assert the default iterator path; the new
`test_map_in_pandas_legacy_accept_any_iterable` /
`test_map_in_arrow_legacy_accept_any_iterable` assert the list path under the
legacy flag.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #58286 from Yicong-Huang/SPARK-58601-oldclient-test.
Authored-by: Yicong Huang <[email protected]>
Signed-off-by: Yicong-Huang <[email protected]>
---
python/pyspark/sql/tests/arrow/test_arrow_map.py | 13 +++++++++++++
python/pyspark/sql/tests/pandas/test_pandas_map.py | 14 +++++++++-----
2 files changed, 22 insertions(+), 5 deletions(-)
diff --git a/python/pyspark/sql/tests/arrow/test_arrow_map.py
b/python/pyspark/sql/tests/arrow/test_arrow_map.py
index 86dca7ed92d0..62b129cde982 100644
--- a/python/pyspark/sql/tests/arrow/test_arrow_map.py
+++ b/python/pyspark/sql/tests/arrow/test_arrow_map.py
@@ -51,6 +51,19 @@ class MapInArrowTestsMixin(object):
expected = df.collect()
self.assertEqual(actual, expected)
+ def test_map_in_arrow_legacy_accept_any_iterable(self):
+ # With the legacy flag enabled, returning a non-Iterator iterable
(e.g. list) is accepted.
+ def list_not_iter(iterator):
+ return [batch for batch in iterator]
+
+ with self.sql_conf(
+
{"spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled":
True}
+ ):
+ df = self.spark.range(10)
+ actual = df.mapInArrow(list_not_iter, "id long").collect()
+ expected = df.collect()
+ self.assertEqual(actual, expected)
+
def test_multiple_columns(self):
data = [(1, "foo"), (2, None), (3, "bar"), (4, "bar")]
df = self.spark.createDataFrame(data, "a int, b string")
diff --git a/python/pyspark/sql/tests/pandas/test_pandas_map.py
b/python/pyspark/sql/tests/pandas/test_pandas_map.py
index 777d6d167048..c003fcd32433 100644
--- a/python/pyspark/sql/tests/pandas/test_pandas_map.py
+++ b/python/pyspark/sql/tests/pandas/test_pandas_map.py
@@ -79,11 +79,15 @@ class MapInPandasTestsMixin:
expected = df.collect()
self.assertEqual(actual, expected)
- # test returning list of DataFrames
- df = self.spark.range(10, numPartitions=3)
- actual = df.mapInPandas(lambda it: [pdf for pdf in it], "id
long").collect()
- expected = df.collect()
- self.assertEqual(actual, expected)
+ def test_map_in_pandas_legacy_accept_any_iterable(self):
+ # With the legacy flag enabled, returning a non-Iterator iterable
(e.g. list) is accepted.
+ with self.sql_conf(
+
{"spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled":
True}
+ ):
+ df = self.spark.range(10, numPartitions=3)
+ actual = df.mapInPandas(lambda it: [pdf for pdf in it], "id
long").collect()
+ expected = df.collect()
+ self.assertEqual(actual, expected)
def test_multiple_columns(self):
data = [(1, "foo"), (2, None), (3, "bar"), (4, "bar")]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
