cloud-fan commented on code in PR #57800:
URL: https://github.com/apache/spark/pull/57800#discussion_r3755586837
##########
python/pyspark/worker.py:
##########
@@ -1923,6 +1933,16 @@ def func(split_index: int, data:
Iterator[pa.RecordBatch]) -> Iterator[pa.Record
# invoke the UDF
output_batches = udf_func(input_batches)
+ # The declared signature is Iterator[...], so a strict iterator is
required. The
+ # legacy flag restores the pre-4.3.0 behavior of accepting any
iterable (e.g. list)
+ # by adapting it into an iterator before the shared element-type
verification.
+ if (
+ runner_conf.map_in_batch_legacy_accept_any_iterable
+ and not isinstance(output_batches, Iterator)
+ and hasattr(output_batches, "__iter__")
Review Comment:
**Blocking:**
The compatibility mode should accept every object Python can iterate, but
this guard excludes sequence-protocol objects that implement `__getitem__`
without `__iter__`; the pandas branch at `worker.py:2726` does the same. Please
attempt `iter(...)` and catch `TypeError` in both branches, with a
sequence-protocol regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
