This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 56c09c2 [MINOR][CONNECT][PYTHON][TESTS] Wait for the first progress
of the query, not processAllAvailable, in the test of StreamingQuery.explain
56c09c2 is described below
commit 56c09c248bead67d06d0fd3c901cc98591ad043b
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 11:53:20 2026 +0900
[MINOR][CONNECT][PYTHON][TESTS] Wait for the first progress of the query,
not processAllAvailable, in the test of StreamingQuery.explain
### What changes were proposed in this pull request?
The live test `test_explain_prints_with_python_print_as_pyspark` now waits
for the first progress of its streaming query, rather than for
`processAllAvailable()`, before it checks `StreamingQuery.explain`.
### Why are the changes needed?
`processAllAvailable()` returns only when the source of the query has no
new rows. A `rate` source makes a row every second. On a slow machine each
batch took longer than that: on a local server under load, a batch took about
3.5 seconds for 4 rows, so there were always new rows. The call never returned,
and the test hung the live suite.
After its first progress, the query has the plan of a batch to explain.
### Does this PR introduce _any_ user-facing change?
No. It changes a test only.
### How was this patch tested?
The test passes with this client and with official PySpark 4.3.0, against a
local Spark 4.3.0 server under the load where it hung before, in about 10
seconds.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #260 from HyukjinKwon/minor-explain-test-progress.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/tests/test_dropin_e2e.py | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 1db0973..91445ae 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1465,6 +1465,7 @@ def
test_explain_prints_with_python_print_as_pyspark(session):
# sys.stdout, which redirect_stdout captures, and return None.
import contextlib
import io
+ import time
df = session.range(1)
printed = io.StringIO()
@@ -1480,7 +1481,12 @@ def
test_explain_prints_with_python_print_as_pyspark(session):
.writeStream.format("memory").queryName("explain_prints").start()
)
try:
- query.processAllAvailable()
+ # The plan of a batch of the query, after its first.
processAllAvailable() would wait
+ # for a rate source to have no new rows, which a slow machine never
reaches.
+ for _ in range(240):
+ if query.lastProgress is not None:
+ break
+ time.sleep(0.25)
printed = io.StringIO()
with contextlib.redirect_stdout(printed):
assert query.explain(True) is None
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]