Hi Robert, This looks like a useful CI refactoring to me. Thanks for doing this!
Cheers, Dmitri. On Sat, Jan 31, 2026 at 10:49 AM Robert Stupp <[email protected]> wrote: > Hi all, > > Currently we have a couple for different and independent workflows for CI > against the main/release branches and PRs: > - 1 workflow for Gradle based CI > - 1 workflow for the Helm charts > - 1 workflow for the Python Client > - 1 workflow for Regression Tests > - 1 workflow for Spark Plugin Regression Tests > - 1 workflow to check Markdown links > > The .asf.yaml file contains references to most but not all jobs in all > these workflows, which is a bit hard to maintain. > > During PR checks, it sometimes happens that a couple of these workflows > fail for intermediate remote reasons. Re-running those workflows requires > opening about 3-5 different workflow runs and to click on "rerun failed > jobs" in each of those. > > I started an experiment to unify all CI workflows and also simplify the > .asf.yaml file in this PR [3625]. > > A screenshot or how a PR workflow run looks like is in the first comment of > that PR. > > You may notice that there are actually three workflows: > - the ci.yml workflow with all the CI related jobs > - a new ci-pr.yml workflow and > - a new ci-main.yml workflow > > The latter two call the first one. The difference of both is the definition > of their concurrency-group: for PRs, the concurrency-group cancels previous > CI workflow runs. For main + release/* branches the workflow runs are not > cancelled. > For PRs, this means less useless GitHub runner usage, when changes are > pushed to a PR while CI jobs are still running. (We all use one limited > pool of runners in the whole Apache organization.) > For main + release/* branches the concurrency-group ensures that the latest > saved Gradle cache state is the most recent one, so eventually less work > due to caching for following CI runs. > > You may also notice a "fake" 'Required Checks' job. This is a "trick" so > that we only need to reference one check from the .asf.yaml file and would > never have to care about the correct references from .asf.yaml again. The > 'Required Checks' dummy job also references all jobs (the admin-tool tests > and python-client-tests are currently not referenced as required checks). > > The unification of all these workflows allowed to simplify & shorten a > couple of job definitions. The whole ci.yml file is < 400 lines including > the license header. > > The jobs themselves still do the same thing as before. Except for the Spark > Plugin Regtest one, which has been appended to the "normal" Regtest one, > because the setup for both is exactly the same. > > WDYT? I'm curious to hear your thoughts! > > Robert > > > [3625] https://github.com/apache/polaris/pull/3625 > [2] https://github.com/apache/polaris/actions >
