This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 925dc876 [MINOR][CONNECT][PYTHON][TESTS] Report the progress every 10 
ms in the live test of a progress handler that runs queries without end, so 
that its queries nest on a fast runner
925dc876 is described below

commit 925dc876f63f1f779fe176f62575a0aa96cc6886
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 12:42:13 2026 +0900

    [MINOR][CONNECT][PYTHON][TESTS] Report the progress every 10 ms in the live 
test of a progress handler that runs queries without end, so that its queries 
nest on a fast runner
    
    ### What changes were proposed in this pull request?
    
    The live test 
`test_progress_handler_that_runs_queries_without_end_raises_recursion_error` 
(SPARK-61188) needs the server to send an in-progress update for two queries: 
the outer query, and the query that its handler runs. Only then do the 
handler's queries nest up to the bound of nested callbacks. The script that the 
test runs in a subprocess now:
    
    - sets `spark.connect.progress.reportInterval` to `10ms` for its session, 
and
    - uses shuffles (`repartition(20)`) for both queries. The outer query sums 
`range(0, 2000, 1, 4)` (1999000), and the handler runs 
`range(100).repartition(20).count()`.
    
    ### Why are the changes needed?
    
    The test failed once in CI, in the Connect parity job of #394. The 
subprocess exited 0 with the right result, but its stderr had no 
`RecursionError`, so the queries did not nest.
    
    The server sends progress updates from its response sender. The sender 
waits for the next result for at most `spark.connect.progress.reportInterval` 
(2 s by default). After each wait, it sends the progress of the execution, if 
the execution is still tracked. So a query shorter than 2 s gets an update only 
if the sender wakes for its result before the execution ends and stops being 
tracked. A fast or idle runner can lose that race. With a 10 ms interval, the 
sender wakes on its timer a [...]
    
    In-progress updates per execution, measured over 40 executions of each 
query against a Spark 4.3.0 server:
    
    | Query | 2 s interval | 10 ms interval |
    |---|---|---|
    | `range(100).repartition(20).count()` (the handler's) | 2-3 | 10-16 |
    | `range(0, 2000, 1, 4).repartition(20)` summed (the outer) | 3 | 8-11 |
    
    On this machine the sender wins the race at 2 s too, so the CI failure does 
not reproduce locally. With 10 ms, the updates no longer depend on that race.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No, test only.
    
    ### How was this patch tested?
    
    - Against a local Spark 4.3.0 server started with `SPARK_TESTING=1`, as in 
CI, the test passed 10 of 10 times. The subprocess script on its own passed 20 
of 20 times, at 1.6-1.9 s each.
    - With the bound of SPARK-61188 lifted (`MAX_CALLBACK_DEPTH` set to 
1,000,000 in a scratch build), the test fails: the subprocess crashes with -11, 
as it did before the bound. So the test still catches the crash.
    - The live `python/tests/test_dropin_e2e.py` against the same server: 201 
passed.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #395 from HyukjinKwon/MINOR-progress-recursion-test-interval.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 python/tests/test_dropin_e2e.py | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index fc8cfa45..b26a142b 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1167,6 +1167,10 @@ def 
test_progress_handler_that_runs_queries_without_end_raises_recursion_error()
     # runs. Past the bound of nested callbacks, the query fails with a 
RecursionError, which the
     # handler's exception goes out with, instead of the native stack 
overflowing and the process
     # crashing (SIGSEGV). The outer query returns its result, as the handler's 
exceptions do.
+    # The queries nest only when the server sends an in-progress update of 
each one. A query that
+    # is shorter than the report interval (2 s) gets one only when the server 
wakes for its result
+    # before the execution ends, which a fast runner may not do. With an 
interval of 10 ms, the
+    # server sends them while the job runs, and each of these shuffles runs 
for several intervals.
     import subprocess
     import sys
 
@@ -1177,12 +1181,14 @@ def 
test_progress_handler_that_runs_queries_without_end_raises_recursion_error()
 import sys
 from pyspark.sql import SparkSession
 spark = SparkSession.builder.remote(%r).getOrCreate()
-spark.registerProgressHandler(lambda **kw: spark.range(4).count() if not 
kw["done"] else None)
-print(spark.range(2000000).selectExpr("sum(id) as t").collect())
+spark.conf.set("spark.connect.progress.reportInterval", "10ms")
+spark.registerProgressHandler(
+    lambda **kw: spark.range(100).repartition(20).count() if not kw["done"] 
else None)
+print(spark.range(0, 2000, 1, 4).repartition(20).selectExpr("sum(id) as 
t").collect())
 """ % remote], env=dict(os.environ, TZ="UTC"), capture_output=True, text=True, 
timeout=300)
     assert result.returncode == 0, result.stderr
-    assert "[Row(t=1999999000000)]" in result.stdout
-    assert "RecursionError" in result.stderr
+    assert "[Row(t=1999000)]" in result.stdout, result.stdout
+    assert "RecursionError" in result.stderr, (result.stdout, result.stderr)
 
 
 def test_progress_handler_runs_a_query_of_its_own_as_pyspark(session):


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to