Ruifeng Zheng created SPARK-61446:
-------------------------------------

             Summary: Remove per-shard compilation fallback in Maven and hosted 
Python CI
                 Key: SPARK-61446
                 URL: https://issues.apache.org/jira/browse/SPARK-61446
             Project: Spark
          Issue Type: Improvement
          Components: Project Infra
    Affects Versions: 5.0.0
            Reporter: Ruifeng Zheng


The reusable Maven and hosted PySpark CI workflows allow shared precompilation 
failures to fall back to full compilation in each matrix shard. A deterministic 
compiler error is then repeated across the shards, consuming additional GitHub 
Actions runners without producing test coverage.

For example, the shared hosted PySpark precompile failed on AddCheckConstraint 
pattern compiler errors, after which all ten downstream shards attempted local 
compilation and failed:
https://github.com/apache/spark/actions/runs/41736782201/job/126267913955

Require successful shared precompilation and artifact download/extraction 
before running the test matrices, and remove the per-shard full-build fallback. 
Preserve the Maven Connect assembly build required by its tests. Transient 
artifact failures should fail the affected shard and require a rerun.

Draft PR: https://github.com/apache/spark/pull/60328




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to