Yicong-Huang opened a new pull request, #59168:
URL: https://github.com/apache/spark/pull/59168
### What changes were proposed in this pull request?
Add positive tests for `mapInArrow` and `mapInPandas` that return a
non-`Iterator` iterable (a `list`) *without* setting
`spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled`,
asserting the value is accepted under the config's default. The tests are added
to the shared `MapInArrowTestsMixin` / `MapInPandasTestsMixin`, so they run in
both the classic and Spark Connect suites.
### Why are the changes needed?
The follow-up #58436 flipped
`spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled` to
default `true`, restoring the pre-4.3.0 lenient behavior where a
`mapInPandas`/`mapInArrow` UDF may return any iterable. The existing
`*_legacy_accept_any_iterable` tests set that flag to `true` explicitly, so
they would still pass if the default were flipped back to `false` and therefore
do not protect the default from regression. A test that relies on the default
(sets no config) closes that gap; if the default regresses to `false`, the
returned `list` would raise `UDF_RETURN_TYPE` and these tests would fail.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
New tests pass in both classic and Spark Connect modes for `mapInArrow` and
`mapInPandas`:
```
python/run-tests --testnames \
"pyspark.sql.tests.arrow.test_arrow_map
MapInArrowTests.test_map_in_arrow_accept_any_iterable_by_default,\
pyspark.sql.tests.pandas.test_pandas_map
MapInPandasTests.test_map_in_pandas_accept_any_iterable_by_default,\
pyspark.sql.tests.connect.arrow.test_parity_arrow_map
ArrowMapParityTests.test_map_in_arrow_accept_any_iterable_by_default,\
pyspark.sql.tests.connect.pandas.test_parity_pandas_map
MapInPandasParityTests.test_map_in_pandas_accept_any_iterable_by_default"
```
### Was this patch authored or co-authored using generative AI tooling?
No
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
