[
https://issues.apache.org/jira/browse/SPARK-62079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-62079:
-----------------------------------
Labels: pull-request-available (was: )
> The retry jitter of the Rust core is nearly the same for the retries of a
> burst of calls, where PySpark draws random.uniform(0, jitter) each time
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62079
> URL: https://issues.apache.org/jira/browse/SPARK-62079
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> PySpark's RetryPolicyState adds a jitter to a wait at or above the jitter
> threshold (python/pyspark/sql/connect/client/retries.py:166-167): `wait_time
> += random.uniform(0, self.policy.jitter)`, a draw of Python's generator each
> time, so the waits of the retries of calls that fail at once are spread over
> [0, jitter], as a jitter is for: a burst of retries, as of the clients of a
> server that restarts, does not hit it again at once.
> The core's rand_jitter (crates/spark-connect-core/src/retries.rs) seeded one
> round of xorshift64 with the nanoseconds of the clock at each call. The top
> bits of the fraction, which make the jitter to the millisecond, depend only
> on the bits of the clock from bit 25 up, which change every 33.5 ms. So the
> draws of a window of 33.5 ms are the jitter of one millisecond.
> Repro, of the core: 200 consecutive draws of rand_jitter(500) as integers.
> - Expected, as random.uniform: about 165 distinct values of [0, 500), in both
> halves.
> - Actual: 1 distinct value ([118, 118, 118, ...]), in 20 of 20 runs.
> The same makes
> retries::tests::test_default_jitter_applies_only_at_or_above_the_threshold
> flaky: its 20 draws of 500 ms all fall below 1 ms about one time in 2000, and
> it then fails ("[3200, 3200, ...]").
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]