zhengruifeng opened a new pull request, #60203: URL: https://github.com/apache/spark/pull/60203
### What changes were proposed in this pull request? Run the complete existing `build_and_test.yml` matrix on sampled upstream post-merge pushes. Pass an explicit all-enabled job selection from `build_main.yml` and clear the previous-commit diff baseline for upstream runs so test runners execute every module in each shard. This includes the JVM and PySpark shards, pandas API on Spark, periodic Python and old-client tests, SparkR, TPC-DS, Docker/Kubernetes/OIDC integration tests, lint, docs, UI, protobuf checks, and the Java 25 build. Fork pushes retain change-based selection, and scheduled workflows retain their explicit job selections and configurations. ### Why are the changes needed? Post-merge CI now runs on one in ten commits after #60177 ([SPARK-61016](https://issues.apache.org/jira/browse/SPARK-61016)), but a selected run still disables several jobs and filters tests to the triggering commit's changes. For example, a sampled documentation-only commit can leave changes from the preceding skipped commits without JVM or Python coverage in that run. Each selected run should test the full snapshot, including changes from skipped commits. Enabling the matrix alone is insufficient: `dev/run-tests.py` also uses the previous commit to filter modules and Python tests, so this PR disables that filtering for upstream runs. The sampling frequency stays at 10%; each selected run does more work. ### Does this PR introduce _any_ user-facing change? No. This changes CI coverage only. ### How was this patch tested? - `actionlint` v1.7.11 passed for both modified workflows; `git diff --check` passed. - Parsed the workflows in `spark-dev-313`, verified that all 18 selection flags are enabled for upstream pushes, and checked that no JVM or PySpark matrix exclusion applies to that selection. - Executed the actual precondition shell step with the full upstream selection and an explicit scheduled subset; both produced the expected job selections. - Executed the module-selection block from `dev/run-tests.py` with controlled diff helpers: sampled upstream runs retained all modules without reading the last-commit diff, while fork builds still selected affected modules and scheduled runs retained their requested modules. - Verified that existing job definitions, matrices, concurrency, permissions, and runner setup are preserved apart from the intended diff-baseline change. Full distributed test execution is left to CI. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
