nchammas opened a new pull request, #58555:
URL: https://github.com/apache/spark/pull/58555
### What changes were proposed in this pull request?
1. Expand the ignore patterns in `sparktestsupport` to cover almost all
cases where we do not want any test modules to fire.
2. Group the ignore patterns by rough domain or topic.
3. Sort the patterns within each group.
### Why are the changes needed?
1. I conducted an AI-assisted analysis of the most recent 2,000 PRs that
were merged to `master`. I found that about 5% of them triggered a full CI run
that they could have completely skipped.
Here are the top 10 examples by CI time consumed:
[#55685](https://github.com/apache/spark/pull/55685) (30h),
[#58072](https://github.com/apache/spark/pull/58072) (28h),
[#55800](https://github.com/apache/spark/pull/55800) (28h),
[#55878](https://github.com/apache/spark/pull/55878) (27h),
[#57972](https://github.com/apache/spark/pull/57972) (27h),
[#56996](https://github.com/apache/spark/pull/56996) (27h),
[#57741](https://github.com/apache/spark/pull/57741) (27h),
[#58206](https://github.com/apache/spark/pull/58206) (27h),
[#58098](https://github.com/apache/spark/pull/58098) (27h),
[#55972](https://github.com/apache/spark/pull/55972) (27h)
2. This is the first of multiple PRs I am planning to cut CI waste and
improve the precision of what tests we run for each PR. Some of these groups
will become new test modules down the line. Most will just stay as ignore
patterns. The groups will make it easier for maintainers to track why certain
files don't trigger tests.
3. Sorting the patterns makes future inserts easier to place and makes it
more obvious to the reader when there are unintentional gaps.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing doctests and CI.
### Was this patch authored or co-authored using generative AI tooling?
I wrote this patch and the associated analysis scripts (not posted; happy to
share if interested) with assistance from GitHub Copilot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
