This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c579dab18f61 [SPARK-58298][PYTHON] Reduce the expected progress for
streaming data source tests
c579dab18f61 is described below
commit c579dab18f615b846f8c54ee1c92b6081b0e8253
Author: Tian Gao <[email protected]>
AuthorDate: Fri Jul 24 06:02:02 2026 +0000
[SPARK-58298][PYTHON] Reduce the expected progress for streaming data
source tests
### What changes were proposed in this pull request?
Reduce the expected progress update from 10 to 5
### Why are the changes needed?
10 is an arbitrary number just to confirm the progress is working properly.
Reducing the number keeps the coverage while reduce the time for the test to
run.
Also, we've experienced a few times when the first cold query took too long
so the 10 progress update did not finish in 30s. Reducing the number will also
stabilize the tests a bit.
### Does this PR introduce _any_ user-facing change?
No. Test only.
### How was this patch tested?
CI.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #57472 from gaogaotiantian/reduce-streaming-check.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Uros Bojanic <[email protected]>
---
python/pyspark/sql/tests/test_python_streaming_datasource.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/sql/tests/test_python_streaming_datasource.py
b/python/pyspark/sql/tests/test_python_streaming_datasource.py
index f6bfbdc655f5..704a2986ded8 100644
--- a/python/pyspark/sql/tests/test_python_streaming_datasource.py
+++ b/python/pyspark/sql/tests/test_python_streaming_datasource.py
@@ -305,7 +305,7 @@ class BasePythonStreamingDataSourceTestsMixin:
assertDataFrameEqual(df, [Row(batch_id * 2), Row(batch_id * 2 +
1)])
q = df.writeStream.foreachBatch(check_batch).start()
- wait_for_condition(q, lambda query: len(query.recentProgress) >= 10)
+ wait_for_condition(q, lambda query: len(query.recentProgress) >= 5)
q.stop()
q.awaitTermination()
self.assertIsNone(q.exception(), "No exception has to be propagated.")
@@ -400,7 +400,7 @@ class BasePythonStreamingDataSourceTestsMixin:
assertDataFrameEqual(df, [Row(batch_id * 2), Row(batch_id * 2 +
1)])
q = df.writeStream.foreachBatch(check_batch).start()
- wait_for_condition(q, lambda query: len(query.recentProgress) >= 10)
+ wait_for_condition(q, lambda query: len(query.recentProgress) >= 5)
q.stop()
q.awaitTermination()
self.assertIsNone(q.exception(), "No exception has to be propagated.")
@@ -456,7 +456,7 @@ class BasePythonStreamingDataSourceTestsMixin:
assertDataFrameEqual(df, [Row(batch_id * 2), Row(batch_id * 2 +
1)])
q = df.writeStream.foreachBatch(check_batch).start()
- wait_for_condition(q, lambda query: len(query.recentProgress) >= 10)
+ wait_for_condition(q, lambda query: len(query.recentProgress) >= 5)
q.stop()
q.awaitTermination()
self.assertIsNone(q.exception(), "No exception has to be propagated.")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
