This is an automated email from the ASF dual-hosted git repository.
ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 11d3fec06d94 [SPARK-55315][PYTHON][TESTS] Allow eventually to take
custom exceptions
11d3fec06d94 is described below
commit 11d3fec06d9468719620df3448888249236ba71c
Author: Tian Gao <[email protected]>
AuthorDate: Tue Feb 3 13:48:42 2026 +0800
[SPARK-55315][PYTHON][TESTS] Allow eventually to take custom exceptions
### What changes were proposed in this pull request?
Allow `eventually` to take custom exceptions, while keeping the API
backward compatible.
### Why are the changes needed?
`eventually` is a good testing util. However while we "wait for the correct
result", multiple errors can happen. For example, we may read an unfinished
file and get a json parsing error - that should be fine if we expect it.
Having the capability to allow any user defined exceptions for `eventually`
is helpful to make our test run efficiently and more stable.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
CI should show no error.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #54096 from gaogaotiantian/allow-more-exceptions.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/testing/utils.py | 23 ++++++++++++-----------
1 file changed, 12 insertions(+), 11 deletions(-)
diff --git a/python/pyspark/testing/utils.py b/python/pyspark/testing/utils.py
index 6618c55772db..c0f6f5e5db2f 100644
--- a/python/pyspark/testing/utils.py
+++ b/python/pyspark/testing/utils.py
@@ -186,6 +186,7 @@ def eventually(
catch_timeout=False,
quiet=True,
interval=0.1,
+ expected_exceptions=tuple(),
):
"""
Wait a given amount of time for a condition to pass, else fail with an
error.
@@ -222,6 +223,14 @@ def eventually(
assert isinstance(catch_timeout, bool)
assert isinstance(quiet, bool)
assert isinstance(interval, float)
+ assert isinstance(expected_exceptions, (tuple, list))
+
+ expected_exceptions = list(expected_exceptions)
+ if catch_assertions:
+ expected_exceptions.append(AssertionError)
+ if catch_timeout:
+ expected_exceptions.append(TimeoutError)
+ expected_exceptions = tuple(expected_exceptions)
def decorator(condition: Callable) -> Callable:
assert isinstance(condition, Callable)
@@ -236,16 +245,8 @@ def eventually(
try:
lastValue = condition(*args, **kwargs)
- except AssertionError as e:
- if catch_assertions:
- lastValue = e
- else:
- raise e
- except TimeoutError as e:
- if catch_timeout:
- lastValue = e
- else:
- raise e
+ except expected_exceptions as e:
+ lastValue = e
if lastValue is True or lastValue is None:
return
@@ -254,7 +255,7 @@ def eventually(
print(f"\nAttempt #{numTries} failed!\n{lastValue}")
sleep(interval)
- if isinstance(lastValue, (AssertionError, TimeoutError)):
+ if isinstance(lastValue, expected_exceptions):
raise lastValue
else:
raise AssertionError(
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
