Hyukjin Kwon created SPARK-62079:
------------------------------------

             Summary: The retry jitter of the Rust core is nearly the same for 
the retries of a burst of calls, where PySpark draws random.uniform(0, jitter) 
each time
                 Key: SPARK-62079
                 URL: https://issues.apache.org/jira/browse/SPARK-62079
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


PySpark's RetryPolicyState adds a jitter to a wait at or above the jitter 
threshold (python/pyspark/sql/connect/client/retries.py:166-167): `wait_time += 
random.uniform(0, self.policy.jitter)`, a draw of Python's generator each time, 
so the waits of the retries of calls that fail at once are spread over [0, 
jitter], as a jitter is for: a burst of retries, as of the clients of a server 
that restarts, does not hit it again at once.

The core's rand_jitter (crates/spark-connect-core/src/retries.rs) seeded one 
round of xorshift64 with the nanoseconds of the clock at each call. The top 
bits of the fraction, which make the jitter to the millisecond, depend only on 
the bits of the clock from bit 25 up, which change every 33.5 ms. So the draws 
of a window of 33.5 ms are the jitter of one millisecond.

Repro, of the core: 200 consecutive draws of rand_jitter(500) as integers.
- Expected, as random.uniform: about 165 distinct values of [0, 500), in both 
halves.
- Actual: 1 distinct value ([118, 118, 118, ...]), in 20 of 20 runs.

The same makes 
retries::tests::test_default_jitter_applies_only_at_or_above_the_threshold 
flaky: its 20 draws of 500 ms all fall below 1 ms about one time in 2000, and 
it then fails ("[3200, 3200, ...]").




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to