Tian Gao created SPARK-54364:
--------------------------------
Summary: Separate test_install_spark out from our CI
Key: SPARK-54364
URL: https://issues.apache.org/jira/browse/SPARK-54364
Project: Spark
Issue Type: Test
Components: PySpark
Affects Versions: 4.1.0
Reporter: Tian Gao
test_install_spark can take 2000~3000s based on the network, but it's not
testing any local code. It basically download the spark from remote and try to
unpack it. Now it's part of pyspark-core which means any commit that changes
pyspark core will trigger this test in our CI. e.g.
[https://github.com/apache/spark/actions/runs/19347901449/job/55352935128]
This wastes a lot of our limited CI resources.
I think we should at least disable this on main commit triggered CI - so leave
it only for scheduled CIs.
If we are confident about how this works, we can have a single CI that run this
test once per day.
I think we are probably spending more than 10 hours every day to run this test
which does not give us any meaningful coverage.
[~dongjoon]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
