Yicong-Huang commented on code in PR #59774:
URL: https://github.com/apache/spark/pull/59774#discussion_r4381234356
##########
python/pyspark/eval_handlers/_base.py:
##########
@@ -150,8 +150,8 @@ class EvalTypeHandler(Generic[InputBatch, OutputBatch],
metaclass=_EvalTypeHandl
variant_value: ClassVar[Any] = None
# Whether the handler's UDFs return an iterator.
_iter_based: ClassVar[bool] = False
- # Whether __init__ wraps the UDF in fail_on_stopiteration, so a
StopIteration raised in
- # user code fails the task instead of silently ending the loop that calls
the UDF.
+ # Whether __init__ wraps the UDF in fail_on_stopiteration, so a
StopIteration raised by
Review Comment:
Thanks, good catch. Rather than documenting the gap, fixed it: the iterator
profilers now call the UDF eagerly (as the unprofiled path does) and only defer
the profiled `next()` calls, so a StopIteration raised by the call fails with
STOP_ITERATION_OCCURRED with or without a profiler. Added
`test_iter_profiler_keeps_stop_iteration_error` covering no profiler / perf /
memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
