This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 925dc876 [MINOR][CONNECT][PYTHON][TESTS] Report the progress every 10
ms in the live test of a progress handler that runs queries without end, so
that its queries nest on a fast runner
925dc876 is described below
commit 925dc876f63f1f779fe176f62575a0aa96cc6886
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 12:42:13 2026 +0900
[MINOR][CONNECT][PYTHON][TESTS] Report the progress every 10 ms in the live
test of a progress handler that runs queries without end, so that its queries
nest on a fast runner
### What changes were proposed in this pull request?
The live test
`test_progress_handler_that_runs_queries_without_end_raises_recursion_error`
(SPARK-61188) needs the server to send an in-progress update for two queries:
the outer query, and the query that its handler runs. Only then do the
handler's queries nest up to the bound of nested callbacks. The script that the
test runs in a subprocess now:
- sets `spark.connect.progress.reportInterval` to `10ms` for its session,
and
- uses shuffles (`repartition(20)`) for both queries. The outer query sums
`range(0, 2000, 1, 4)` (1999000), and the handler runs
`range(100).repartition(20).count()`.
### Why are the changes needed?
The test failed once in CI, in the Connect parity job of #394. The
subprocess exited 0 with the right result, but its stderr had no
`RecursionError`, so the queries did not nest.
The server sends progress updates from its response sender. The sender
waits for the next result for at most `spark.connect.progress.reportInterval`
(2 s by default). After each wait, it sends the progress of the execution, if
the execution is still tracked. So a query shorter than 2 s gets an update only
if the sender wakes for its result before the execution ends and stops being
tracked. A fast or idle runner can lose that race. With a 10 ms interval, the
sender wakes on its timer a [...]
In-progress updates per execution, measured over 40 executions of each
query against a Spark 4.3.0 server:
| Query | 2 s interval | 10 ms interval |
|---|---|---|
| `range(100).repartition(20).count()` (the handler's) | 2-3 | 10-16 |
| `range(0, 2000, 1, 4).repartition(20)` summed (the outer) | 3 | 8-11 |
On this machine the sender wins the race at 2 s too, so the CI failure does
not reproduce locally. With 10 ms, the updates no longer depend on that race.
### Does this PR introduce _any_ user-facing change?
No, test only.
### How was this patch tested?
- Against a local Spark 4.3.0 server started with `SPARK_TESTING=1`, as in
CI, the test passed 10 of 10 times. The subprocess script on its own passed 20
of 20 times, at 1.6-1.9 s each.
- With the bound of SPARK-61188 lifted (`MAX_CALLBACK_DEPTH` set to
1,000,000 in a scratch build), the test fails: the subprocess crashes with -11,
as it did before the bound. So the test still catches the crash.
- The live `python/tests/test_dropin_e2e.py` against the same server: 201
passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #395 from HyukjinKwon/MINOR-progress-recursion-test-interval.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/tests/test_dropin_e2e.py | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index fc8cfa45..b26a142b 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1167,6 +1167,10 @@ def
test_progress_handler_that_runs_queries_without_end_raises_recursion_error()
# runs. Past the bound of nested callbacks, the query fails with a
RecursionError, which the
# handler's exception goes out with, instead of the native stack
overflowing and the process
# crashing (SIGSEGV). The outer query returns its result, as the handler's
exceptions do.
+ # The queries nest only when the server sends an in-progress update of
each one. A query that
+ # is shorter than the report interval (2 s) gets one only when the server
wakes for its result
+ # before the execution ends, which a fast runner may not do. With an
interval of 10 ms, the
+ # server sends them while the job runs, and each of these shuffles runs
for several intervals.
import subprocess
import sys
@@ -1177,12 +1181,14 @@ def
test_progress_handler_that_runs_queries_without_end_raises_recursion_error()
import sys
from pyspark.sql import SparkSession
spark = SparkSession.builder.remote(%r).getOrCreate()
-spark.registerProgressHandler(lambda **kw: spark.range(4).count() if not
kw["done"] else None)
-print(spark.range(2000000).selectExpr("sum(id) as t").collect())
+spark.conf.set("spark.connect.progress.reportInterval", "10ms")
+spark.registerProgressHandler(
+ lambda **kw: spark.range(100).repartition(20).count() if not kw["done"]
else None)
+print(spark.range(0, 2000, 1, 4).repartition(20).selectExpr("sum(id) as
t").collect())
""" % remote], env=dict(os.environ, TZ="UTC"), capture_output=True, text=True,
timeout=300)
assert result.returncode == 0, result.stderr
- assert "[Row(t=1999999000000)]" in result.stdout
- assert "RecursionError" in result.stderr
+ assert "[Row(t=1999000)]" in result.stdout, result.stdout
+ assert "RecursionError" in result.stderr, (result.stdout, result.stderr)
def test_progress_handler_runs_a_query_of_its_own_as_pyspark(session):
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]