dtenedor commented on code in PR #43682:
URL: https://github.com/apache/spark/pull/43682#discussion_r1384198085
##########
python/pyspark/worker.py:
##########
@@ -995,6 +995,8 @@ def verify_result(result):
def func(*a: Any) -> Any:
try:
return f(*a)
+ except StopIteration:
+ raise
Review Comment:
I looked at this, but unlike scalar pyspark UDFs, we want to apply custom
treatment for UDTFs with respect to the `eval` vs. `terminate` vs. `cleanup`
methods. The idea is that if `eval` invokes `raise StopIteration()`, we then
call `terminate` as normal and return a successful result for the UDTF as a
whole.
To implement this behavior, if we used `fail_on_stopiteration` here, we'd
have to later catch a `PySparkRuntimeException` and check if it's
`error_class="STOP_ITERATION_OCCURRED"` manually, which seems pretty confusing.
It seems simpler to just `raise` here and catch the `StopIteration` on L1062
and L1154 instead, and then call `terminate`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
