This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new c7aa7de358d8 [SPARK-58298][PYTHON] Reduce the expected progress for
streaming data source tests
c7aa7de358d8 is described below
commit c7aa7de358d8fd931b0b8c4a80ea054144b353d0
Author: Tian Gao <[email protected]>
AuthorDate: Fri Jul 24 06:02:02 2026 +0000
[SPARK-58298][PYTHON] Reduce the expected progress for streaming data
source tests
### What changes were proposed in this pull request?
Reduce the expected progress update from 10 to 5
### Why are the changes needed?
10 is an arbitrary number just to confirm the progress is working properly.
Reducing the number keeps the coverage while reduce the time for the test to
run.
Also, we've experienced a few times when the first cold query took too long
so the 10 progress update did not finish in 30s. Reducing the number will also
stabilize the tests a bit.
### Does this PR introduce _any_ user-facing change?
No. Test only.
### How was this patch tested?
CI.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #57472 from gaogaotiantian/reduce-streaming-check.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Uros Bojanic <[email protected]>
(cherry picked from commit c579dab18f615b846f8c54ee1c92b6081b0e8253)
Signed-off-by: Uros Bojanic <[email protected]>
---
python/pyspark/sql/tests/test_python_streaming_datasource.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/sql/tests/test_python_streaming_datasource.py
b/python/pyspark/sql/tests/test_python_streaming_datasource.py
index f6bfbdc655f5..704a2986ded8 100644
--- a/python/pyspark/sql/tests/test_python_streaming_datasource.py
+++ b/python/pyspark/sql/tests/test_python_streaming_datasource.py
@@ -305,7 +305,7 @@ class BasePythonStreamingDataSourceTestsMixin:
assertDataFrameEqual(df, [Row(batch_id * 2), Row(batch_id * 2 +
1)])
q = df.writeStream.foreachBatch(check_batch).start()
- wait_for_condition(q, lambda query: len(query.recentProgress) >= 10)
+ wait_for_condition(q, lambda query: len(query.recentProgress) >= 5)
q.stop()
q.awaitTermination()
self.assertIsNone(q.exception(), "No exception has to be propagated.")
@@ -400,7 +400,7 @@ class BasePythonStreamingDataSourceTestsMixin:
assertDataFrameEqual(df, [Row(batch_id * 2), Row(batch_id * 2 +
1)])
q = df.writeStream.foreachBatch(check_batch).start()
- wait_for_condition(q, lambda query: len(query.recentProgress) >= 10)
+ wait_for_condition(q, lambda query: len(query.recentProgress) >= 5)
q.stop()
q.awaitTermination()
self.assertIsNone(q.exception(), "No exception has to be propagated.")
@@ -456,7 +456,7 @@ class BasePythonStreamingDataSourceTestsMixin:
assertDataFrameEqual(df, [Row(batch_id * 2), Row(batch_id * 2 +
1)])
q = df.writeStream.foreachBatch(check_batch).start()
- wait_for_condition(q, lambda query: len(query.recentProgress) >= 10)
+ wait_for_condition(q, lambda query: len(query.recentProgress) >= 5)
q.stop()
q.awaitTermination()
self.assertIsNone(q.exception(), "No exception has to be propagated.")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
