dtenedor opened a new pull request, #43682:
URL: https://github.com/apache/spark/pull/43682
### What changes were proposed in this pull request?
This PR creates a Python UDTF API to stop consuming rows from the input
table.
If the UDTF raises a `StopIteration` exception in the `eval` method, then
the UDTF stops consuming rows from the input table for that input partition,
and finally calls the `terminate` method (if any).
For example:
```
@udtf
class TestUDTF:
def __init__(self):
self._total = 0
@staticmethod
def analyze(_):
return AnalyzeResult(
schema=StructType().add("total", IntegerType()),
withSinglePartition=True
)
def eval(self, _: Row):
self._total += 1
if self._total >= 3:
raise StopIteration("StopIteration at self._total >= 3")
def terminate(self):
yield self._total,
```
### Why are the changes needed?
This is useful when the UDTF logic knows that we don't have to scan the
input table anymore, and skip the rest of the I/O for that case.
### Does this PR introduce _any_ user-facing change?
Yes, see above.
### How was this patch tested?
This PR adds test coverage.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
