Kent Yao created SPARK-55401:
--------------------------------
Summary: Add retry logic and timeout handling to pyspark install
download
Key: SPARK-55401
URL: https://issues.apache.org/jira/browse/SPARK-55401
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 4.2.0
Reporter: Kent Yao
The test_install_spark test in pyspark.tests.test_install_spark is flaky in CI
due to network issues when downloading Spark distributions from Apache mirrors.
Current issues in pyspark/install.py:
1. No connection/read timeout on urllib.request.urlopen() - downloads can hang
indefinitely
2. No retry logic on download failures - a single transient network error
causes complete failure
3. Mirror failover only happens on exception, not on timeout/stall
Proposed fix:
- Add configurable timeout to urlopen() calls (both for mirror resolution and
download)
- Add retry logic with exponential backoff for download attempts
- Improve error reporting for debugging CI failures
This affects the pyspark-install CI job which downloads ~400MB from Apache
mirrors and frequently fails due to network instability.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
