Dale Lane created FLINK-40248:
---------------------------------
Summary: Move publishing nightly snapshots from AZP to GHA
Key: FLINK-40248
URL: https://issues.apache.org/jira/browse/FLINK-40248
Project: Flink
Issue Type: Sub-task
Components: Build System / CI
Reporter: Dale Lane
Today, the nightly scheduled Azure Pipelines pipeline:
1) uploads release binaries to S3
2) pushes snapshot jars to Maven
Before we could migrate fully off of AZP, we need an equivalent nightly
scheduled job that does this from Github Actions.
We'll do this in a phased way, one release branch at a time.
h3. Approach
Add the equivalent snapshot publishing to the GHA nightly workflow, covering
both the binary release upload and the Maven snapshot deploy.
Which pipeline publishes a given branch will be configuration held in the GHA
nightly workflow itself.
Because that workflow is versioned per branch, each branch can be switched over
independently, and a branch that hasn't been switched over yet still *builds*
its snapshots on GHA (it just keeps them as build artifacts of the workflow run
rather than publishing them).
The aim of this is to make sure that the new GHA jobs get exercised on every
branch well before they become the canonical source of anything we publish.
The safety property we want throughout is that exactly one pipeline is
publishing a given branch at any time. Two pipelines publishing the same branch
would overwrite each other (neither publishing would arguably be worse, as a
branch's snapshots would silently go stale).
For each branch, enabling GHA and disabling AZP should be done together, as one
reviewable change.
h3. Rollout
The first branch that we migrate should have an extra safety net step that
let's us compare GHA's output against AZP's output on a few real nightly runs
before anything is depending on GHA output.
# *Both pipelines publish the branch.* GHA publishes to a separate location
that nothing consumes, so it can't overwrite what AZP is publishing. We compare
the two sets of output over a series of nightly runs to confirm parity.
# *Hand the branch over.* We make the switch described above, so GHA starts
publishing to the real location and AZP stops publishing the branch. We then
verify the published snapshots directly.
# *Migrate the remaining branches.* With the approach proven end to end, each
subsequent branch is switched over in a single step - no shadow phase needed.
h3. Not in scope
I'm not talking about migrating the nightly *test* jobs here, or the manual
release process. This is only about the nightly snapshot builds currently
produced from AZP
h3. Done when
This subtask will be complete once every active release branch publishes its
nightly snapshots from GHA, and AZP no longer does. At that point, the nightly
snapshot deployment configuration can be deleted from the AZP pipeline
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
