This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 56c09c2  [MINOR][CONNECT][PYTHON][TESTS] Wait for the first progress 
of the query, not processAllAvailable, in the test of StreamingQuery.explain
56c09c2 is described below

commit 56c09c248bead67d06d0fd3c901cc98591ad043b
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 11:53:20 2026 +0900

    [MINOR][CONNECT][PYTHON][TESTS] Wait for the first progress of the query, 
not processAllAvailable, in the test of StreamingQuery.explain
    
    ### What changes were proposed in this pull request?
    
    The live test `test_explain_prints_with_python_print_as_pyspark` now waits 
for the first progress of its streaming query, rather than for 
`processAllAvailable()`, before it checks `StreamingQuery.explain`.
    
    ### Why are the changes needed?
    
    `processAllAvailable()` returns only when the source of the query has no 
new rows. A `rate` source makes a row every second. On a slow machine each 
batch took longer than that: on a local server under load, a batch took about 
3.5 seconds for 4 rows, so there were always new rows. The call never returned, 
and the test hung the live suite.
    
    After its first progress, the query has the plan of a batch to explain.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. It changes a test only.
    
    ### How was this patch tested?
    
    The test passes with this client and with official PySpark 4.3.0, against a 
local Spark 4.3.0 server under the load where it hung before, in about 10 
seconds.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #260 from HyukjinKwon/minor-explain-test-progress.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 python/tests/test_dropin_e2e.py | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 1db0973..91445ae 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1465,6 +1465,7 @@ def 
test_explain_prints_with_python_print_as_pyspark(session):
     # sys.stdout, which redirect_stdout captures, and return None.
     import contextlib
     import io
+    import time
 
     df = session.range(1)
     printed = io.StringIO()
@@ -1480,7 +1481,12 @@ def 
test_explain_prints_with_python_print_as_pyspark(session):
         .writeStream.format("memory").queryName("explain_prints").start()
     )
     try:
-        query.processAllAvailable()
+        # The plan of a batch of the query, after its first. 
processAllAvailable() would wait
+        # for a rate source to have no new rows, which a slow machine never 
reaches.
+        for _ in range(240):
+            if query.lastProgress is not None:
+                break
+            time.sleep(0.25)
         printed = io.StringIO()
         with contextlib.redirect_stdout(printed):
             assert query.explain(True) is None


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to