Hyukjin Kwon created SPARK-62079:
------------------------------------
Summary: The retry jitter of the Rust core is nearly the same for
the retries of a burst of calls, where PySpark draws random.uniform(0, jitter)
each time
Key: SPARK-62079
URL: https://issues.apache.org/jira/browse/SPARK-62079
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
PySpark's RetryPolicyState adds a jitter to a wait at or above the jitter
threshold (python/pyspark/sql/connect/client/retries.py:166-167): `wait_time +=
random.uniform(0, self.policy.jitter)`, a draw of Python's generator each time,
so the waits of the retries of calls that fail at once are spread over [0,
jitter], as a jitter is for: a burst of retries, as of the clients of a server
that restarts, does not hit it again at once.
The core's rand_jitter (crates/spark-connect-core/src/retries.rs) seeded one
round of xorshift64 with the nanoseconds of the clock at each call. The top
bits of the fraction, which make the jitter to the millisecond, depend only on
the bits of the clock from bit 25 up, which change every 33.5 ms. So the draws
of a window of 33.5 ms are the jitter of one millisecond.
Repro, of the core: 200 consecutive draws of rand_jitter(500) as integers.
- Expected, as random.uniform: about 165 distinct values of [0, 500), in both
halves.
- Actual: 1 distinct value ([118, 118, 118, ...]), in 20 of 20 runs.
The same makes
retries::tests::test_default_jitter_applies_only_at_or_above_the_threshold
flaky: its 20 draws of 500 ms all fall below 1 ms about one time in 2000, and
it then fails ("[3200, 3200, ...]").
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]