This is an automated email from the ASF dual-hosted git repository.
Yicong-Huang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new a363a856de83 [SPARK-58601][PYTHON][TEST][FOLLOWUP] Cover the
mapInBatch acceptAnyIterable default in mapInArrow/mapInPandas
a363a856de83 is described below
commit a363a856de83c58074b8534e2ff743d4e6664c9e
Author: Yicong Huang <[email protected]>
AuthorDate: Thu Oct 1 17:05:45 2026 +0000
[SPARK-58601][PYTHON][TEST][FOLLOWUP] Cover the mapInBatch
acceptAnyIterable default in mapInArrow/mapInPandas
### What changes were proposed in this pull request?
Add positive tests for `mapInArrow` and `mapInPandas` that return a
non-`Iterator` iterable (a `list`) *without* setting
`spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled`,
asserting the value is accepted under the config's default. The tests are added
to the shared `MapInArrowTestsMixin` / `MapInPandasTestsMixin`, so they run in
both the classic and Spark Connect suites.
### Why are the changes needed?
The follow-up #58436 flipped
`spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled` to
default `true`, restoring the pre-4.3.0 lenient behavior where a
`mapInPandas`/`mapInArrow` UDF may return any iterable. The existing
`*_legacy_accept_any_iterable` tests set that flag to `true` explicitly, so
they would still pass if the default were flipped back to `false` and therefore
do not protect the default from regression. A test that relies on the default
(sets no conf [...]
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
New tests pass in both classic and Spark Connect modes for `mapInArrow` and
`mapInPandas`:
```
python/run-tests --testnames \
"pyspark.sql.tests.arrow.test_arrow_map
MapInArrowTests.test_map_in_arrow_accept_any_iterable_by_default,\
pyspark.sql.tests.pandas.test_pandas_map
MapInPandasTests.test_map_in_pandas_accept_any_iterable_by_default,\
pyspark.sql.tests.connect.arrow.test_parity_arrow_map
ArrowMapParityTests.test_map_in_arrow_accept_any_iterable_by_default,\
pyspark.sql.tests.connect.pandas.test_parity_pandas_map
MapInPandasParityTests.test_map_in_pandas_accept_any_iterable_by_default"
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
This pull request and its description were written by Isaac.
Closes #59168 from Yicong-Huang/mapinbatch-default-tests.
Authored-by: Yicong Huang <[email protected]>
Signed-off-by: Yicong-Huang <[email protected]>
---
python/pyspark/sql/tests/arrow/test_arrow_map.py | 13 +++++++++++++
python/pyspark/sql/tests/pandas/test_pandas_map.py | 10 ++++++++++
2 files changed, 23 insertions(+)
diff --git a/python/pyspark/sql/tests/arrow/test_arrow_map.py
b/python/pyspark/sql/tests/arrow/test_arrow_map.py
index 32525f4fcdd3..853ba731f1df 100644
--- a/python/pyspark/sql/tests/arrow/test_arrow_map.py
+++ b/python/pyspark/sql/tests/arrow/test_arrow_map.py
@@ -92,6 +92,19 @@ class MapInArrowTestsMixin:
expected = df.collect()
self.assertEqual(actual, expected)
+ def test_map_in_arrow_accept_any_iterable_by_default(self):
+ # The legacy flag defaults to true, so returning a non-Iterator
iterable (e.g. list) is
+ # accepted without setting any config. This guards the default against
regression: the
+ # *_legacy_accept_any_iterable tests above force the flag on and would
still pass if the
+ # default were flipped back to false.
+ def list_not_iter(iterator):
+ return [batch for batch in iterator]
+
+ df = self.spark.range(10)
+ actual = df.mapInArrow(list_not_iter, "id long").collect()
+ expected = df.collect()
+ self.assertEqual(actual, expected)
+
def test_map_in_arrow_with_limit(self):
def get_size(iterator):
for batch in iterator:
diff --git a/python/pyspark/sql/tests/pandas/test_pandas_map.py
b/python/pyspark/sql/tests/pandas/test_pandas_map.py
index 00ccc06aee94..fbe2c76874b4 100644
--- a/python/pyspark/sql/tests/pandas/test_pandas_map.py
+++ b/python/pyspark/sql/tests/pandas/test_pandas_map.py
@@ -120,6 +120,16 @@ class MapInPandasTestsMixin:
expected = df.collect()
self.assertEqual(actual, expected)
+ def test_map_in_pandas_accept_any_iterable_by_default(self):
+ # The legacy flag defaults to true, so returning a non-Iterator
iterable (e.g. list) is
+ # accepted without setting any config. This guards the default against
regression: the
+ # *_legacy_accept_any_iterable tests above force the flag on and would
still pass if the
+ # default were flipped back to false.
+ df = self.spark.range(10, numPartitions=3)
+ actual = df.mapInPandas(lambda it: [pdf for pdf in it], "id
long").collect()
+ expected = df.collect()
+ self.assertEqual(actual, expected)
+
def test_multiple_columns(self):
data = [(1, "foo"), (2, None), (3, "bar"), (4, "bar")]
df = self.spark.createDataFrame(data, "a int, b string")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
