andrewmusselman commented on issue #751:
URL:
https://github.com/apache/tooling-trusted-releases/issues/751#issuecomment-4241486772
@dave2wave @sbp @alitheg draft here for review, I'm not sure how much of
this should go in `atr/docs` since it's an investigation for a decision
# Distribution Platform Investigation
## Overview
This document investigates remote promotion capabilities across the
distribution platforms ATR supports, and identifies what schema or model
changes would be needed. The goal is to determine whether ATR can promote
staged artifacts to production on each platform without re-uploading, and
whether this is worth the added complexity.
## Platform Capability Matrix
| Platform | Has staging | Has promotion API | Would we use promotion | ATR
staging URL configured | GH workflow slug |
|---|---|---|---|---|---|
| Artifact Hub | Yes | No | No | Yes (`staging.artifacthub.io`) |
`artifacthub` |
| Docker Hub | Yes (tags) | Yes (retag) | Maybe | No (TODO in code) |
`dockerhub` |
| GitHub Releases | Yes (prerelease) | Yes (prerelease→release) | Maybe |
Commented out in code | `github` |
| Maven Central | Yes (RAO) | Yes (close+release) | No | Yes
(`repository.apache.org`) | `mavencentral` |
| npm | Yes (dist-tags) | Yes (dist-tags) | Maybe | No | `npm` |
| npm (scoped) | Yes (dist-tags) | Yes (dist-tags) | Maybe | No | `npm` |
| PyPI | Yes (Test PyPI) | No | No | Yes (`test.pypi.org`) | `pypi` |
| Erlang (Hex) | No | No | No | Not configured | Not configured |
## Per-Platform Details
### Artifact Hub
**Staging**: Artifact Hub has a staging instance at
`staging.artifacthub.io`. ATR already has this URL configured in
`DistributionPlatformValue.template_staging_url`.
**Promotion**: Not applicable. Artifact Hub only records metadata about
packages hosted elsewhere (typically Helm chart repositories). There is no
concept of "promoting" a package — the metadata is indexed from the source
repository. When the source repository is updated, Artifact Hub re-indexes
automatically.
**Recommendation**: No promotion capability. Continue with current approach.
### Docker Hub
**Staging**: Docker Hub uses mutable tags. A staging version can be pushed
with a tag like `1.0.0-rc1` or `1.0.0-staging`. The ATR code has a TODO
comment: "Need to use staging tags?"
**Promotion**: Docker Hub supports retagging via the Docker Registry HTTP
API v2. A staged image can be "promoted" by adding a production tag to the same
manifest digest without re-pushing the image layers. This is done via `PUT
/v2/{name}/manifests/{reference}` with the manifest from the staging tag.
**Caveats**: Promotion only works if using mutable tags (i.e., the staging
and production tags point to the same repository). If staging uses a separate
repository, promotion requires a cross-repository mount operation which not all
registries support equivalently.
**Recommendation**: Promotion is feasible but adds complexity. ATR would
need to track the manifest digest from staging and verify it matches the
voted-on artifacts before retagging. Worth implementing if Docker Hub
distributions become common.
### GitHub Releases
**Staging**: GitHub Releases supports a `prerelease` boolean flag. A staging
release can be created with `"prerelease": true`. The ATR code has this
commented out with `template_staging_url` pointing to the releases API.
**Promotion**: A prerelease can be promoted to a full release by PATCH-ing
the release to set `"prerelease": false`. The release assets (uploaded files)
remain unchanged. This is a single API call.
**Caveats**: The GITHUB platform is currently commented out in
`DistributionPlatform`. Enabling it would require implementing the GitHub
Releases API client. GitHub Releases are tied to git tags, so the tag must
exist before creating the release.
**Recommendation**: Straightforward promotion API. Worth enabling if/when
the GITHUB platform is uncommented.
### Maven Central
**Staging**: Maven Central uses the Repository Apache Org (RAO) at
`repository.apache.org` for staging. PMCs stage artifacts to RAO, which creates
a staging repository. ATR has `template_staging_url` configured pointing to RAO.
**Promotion**: RAO supports promotion via the Nexus Staging REST API: close
the staging repository (triggers validation), then release it to Maven Central.
This is the standard Maven Central publishing workflow.
**Would we use it**: No, per dave2wave's assessment. RAO is used for
SNAPSHOTS, staging, and Maven Central promotion. ATR will continue to act as a
bridge because the existing RAO workflow is well-established and PMCs rely on
it.
**Recommendation**: Do not implement promotion. Continue using ATR as bridge
to RAO.
### npm
**Staging**: npm supports dist-tags, which are mutable aliases for specific
versions. A staging version can be published normally and tagged with `npm
dist-tag add package@version staging`.
**Promotion**: A staged version can be promoted to `latest` (production) by
running `npm dist-tag add package@version latest`. This does not re-upload the
package — it simply moves the `latest` pointer to the staged version. The ATR
code has a TODO: "Need to parse dist-tags."
**Caveats**: The version number remains the same. If the staging version was
`1.0.0`, promoting it to `latest` means `1.0.0` becomes the latest version.
There is no separate "staging" vs "production" namespace like Test PyPI vs PyPI.
**Recommendation**: Promotion is trivial (one API call to move the
dist-tag). Worth implementing when npm distribution support matures. ATR would
need to track which dist-tag was used for staging.
### npm (scoped)
Same capabilities as npm. The only difference is the package name format
(`@scope/package`). Promotion works identically via dist-tags.
### PyPI
**Staging**: Test PyPI (`test.pypi.org`) is a separate instance of PyPI used
for testing. ATR has `template_staging_url` configured for Test PyPI.
**Promotion**: PyPI does not support promotion from Test PyPI to production
PyPI. They are completely separate registries with separate user accounts and
package namespaces. Artifacts must be re-uploaded to production PyPI.
**Additional note**: Per the original issue, projects have been reported to
use production PyPI for staging rather than Test PyPI, which complicates the
staging model.
**Recommendation**: No promotion capability. Continue with re-upload
approach.
### Erlang (Hex)
**Staging**: Hex.pm has no separate staging instance. There is no equivalent
of Test PyPI — publishing goes directly to the production registry.
**Promotion**: Not applicable. Once a package version is published to
Hex.pm, it is immediately live. Publications can be reverted within one hour of
first publication, and versions can be flagged as retired, but there is no
staging-to-production promotion mechanism.
**Publishing**: Packages are published via `rebar3 hex publish` (Erlang) or
`mix hex.publish` (Elixir). Authentication uses API keys. The package tarball
is uploaded to CDN.
**Recommendation**: No promotion capability. Same model as PyPI — publish is
a one-shot operation. If ATR adds Hex support, it would need to upload directly
to Hex.pm at finish time.
## Schema Impact
The current `DistributionPlatformValue` dataclass already has
`template_staging_url` but lacks fields for promotion. To support promotion,
consider adding:
```python
@dataclasses.dataclass(frozen=True)
class DistributionPlatformValue:
name: str
gh_slug: str
template_url: str
template_staging_url: str | None = None
requires_owner_namespace: bool = False
default_owner_namespace: str | None = None
# New fields for promotion support:
supports_promotion: bool = False
promotion_mechanism: str | None = None # "retag", "dist-tag",
"api-patch", etc.
```
The `Distribution` model already has a `staging` boolean field. To track
promotion, consider adding:
```python
class Distribution(sqlmodel.SQLModel, table=True):
# ... existing fields ...
staging: bool = sqlmodel.Field(default=False)
promoted: bool = sqlmodel.Field(default=False)
promoted_at: datetime.datetime | None = sqlmodel.Field(default=None)
staging_revision_key: str | None = sqlmodel.Field(default=None) # Track
which revision was staged
```
The `staging_revision_key` field addresses the caveat from the original
issue: "We would have to track the exact revision of files that were uploaded
in the staging version... we also have to check that the files haven't been
updated in subsequent revisions."
## Recommendation
Promotion adds complexity with modest performance benefit. The priority
order for implementation:
1. **npm/npm-scoped** — Simplest promotion (dist-tag move), most value
2. **Docker Hub** — Feasible but requires manifest digest tracking
3. **GitHub Releases** — Straightforward but platform is currently disabled
4. **Maven Central** — Already decided against (dave2wave)
5. **Artifact Hub** — Not applicable
6. **PyPI** — Not possible
7. **Erlang (Hex)** — Not possible
Before implementing any promotion, the schema changes above and the
revision-tracking logic should be in place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
