asf-tooling commented on issue #751:
URL:
https://github.com/apache/tooling-trusted-releases/issues/751#issuecomment-4404154914
<!-- gofannon-issue-triage-bot v2 -->
**Automated triage** — analyzed at `main@837830e8`
**Type:** `discussion` • **Classification:** `no_action` •
**Confidence:** `high`
**Application domain(s):** `release_lifecycle`, `distribution_publishing`
### Summary
This is an active investigation/discussion about whether ATR should support
remote promotion of artifacts on third-party platforms instead of re-uploading.
@dave2wave set up a platform capability matrix and identified
`atr/models/sql.py` and `atr/models/distribution.py` as the key files to
extend. @andrewmusselman provided a comprehensive investigation draft and
follow-up RFC with corrections and schema proposals, but no final decision or
approved implementation exists yet. The team is still deliberating
platform-by-platform (Maven promotion via RAO is historical; PyPI has no
promotion; Docker/npm use mutable tags; Hex isn't modeled at all).
### Where this lives in the code today
#### `atr/storage/writers/distributions.py` —
`CommitteeMember.__upgrade_staging_to_final` (lines 272-301)
_currently does this_
Already implements the concept of upgrading a staging distribution record to
production in the database, which is the local side of what remote promotion
would need.
```python
async def __upgrade_staging_to_final(
self,
release_key: models.safe.ReleaseKey,
platform: models.sql.DistributionPlatform,
owner_namespace: str | None,
package: str,
version: str,
pending: bool,
upload_date: datetime.datetime | None,
api_url: str | None,
web_url: str | None,
) -> models.sql.Distribution | None:
tag = f"{release_key} {platform} {owner_namespace or ''} {package}
{version}"
existing = await self.__data.distribution(
release_key=str(release_key),
platform=platform,
owner_namespace=(owner_namespace or ""),
package=package,
version=version,
).demand(RuntimeError(f"Distribution {tag} not found"))
if existing.staging:
existing.staging = False
existing.pending = pending
existing.upload_date = upload_date
existing.api_url = api_url
existing.web_url = web_url
existing.created_by = self.__asf_uid
await self.__data.commit()
return existing
return None
```
#### `atr/shared/distribution.py` — `_template_url` (lines 439-458)
_needs modification_
Currently limits staging to ArtifactHub, PyPI, and Maven Central. Would need
extension if Docker Hub, npm, or other platforms gain staging/promotion support.
```python
def _template_url(
dd: distribution.Data,
staging: bool | None = None,
) -> str:
if staging is False:
return dd.platform.value.template_url
supported = {
sql.DistributionPlatform.ARTIFACT_HUB,
sql.DistributionPlatform.PYPI,
sql.DistributionPlatform.MAVEN,
}
if dd.platform not in supported:
raise RuntimeError("Staging is currently supported only for
ArtifactHub, PyPI and Maven Central.")
template_url = dd.platform.value.template_staging_url
if template_url is None:
raise RuntimeError("This platform does not provide a staging API
endpoint.")
return template_url
```
#### `atr/shared/distribution.py` — `DistributionPlatform` (lines 78-102)
_extension point_
Does not include HEX/Erlang as @andrewmusselman noted; would need extension
if new platforms are added as part of this investigation.
```python
class DistributionPlatform(enum.Enum):
"""Wrapper enum for distribution platforms."""
ARTIFACT_HUB = "Artifact Hub"
DOCKER_HUB = "Docker Hub"
MAVEN = "Maven Central"
NPM = "npm"
NPM_SCOPED = "npm (scoped)"
PYPI = "PyPI"
def to_sql(self) -> sql.DistributionPlatform:
"""Convert to SQL enum."""
match self:
case DistributionPlatform.ARTIFACT_HUB:
return sql.DistributionPlatform.ARTIFACT_HUB
case DistributionPlatform.DOCKER_HUB:
return sql.DistributionPlatform.DOCKER_HUB
case DistributionPlatform.MAVEN:
return sql.DistributionPlatform.MAVEN
case DistributionPlatform.NPM:
return sql.DistributionPlatform.NPM
case DistributionPlatform.NPM_SCOPED:
return sql.DistributionPlatform.NPM_SCOPED
case DistributionPlatform.PYPI:
return sql.DistributionPlatform.PYPI
```
#### `atr/storage/writers/distributions.py` — `CommitteeMember.record`
(lines 143-173)
_currently does this_
The record method already calls __upgrade_staging_to_final when a production
record is made for an existing staging record. This is the mechanism that would
be augmented for remote promotion.
```python
async def record(
self,
release_key: models.safe.ReleaseKey,
platform: models.sql.DistributionPlatform,
owner_namespace: models.safe.Alphanumeric | None,
package: models.safe.Alphanumeric,
version: models.safe.VersionKey,
staging: bool,
pending: bool,
upload_date: datetime.datetime | None,
api_url: str | None = None,
web_url: str | None = None,
) -> tuple[models.sql.Distribution, bool]:
namespace = str(owner_namespace) if owner_namespace else ""
existing = await self.__data.distribution(
str(release_key), platform, namespace, str(package), str(version)
).get()
dist = models.sql.Distribution(
platform=platform,
release_key=str(release_key),
owner_namespace=namespace,
package=str(package),
version=str(version),
staging=staging,
pending=pending,
retries=0,
upload_date=upload_date,
api_url=api_url,
web_url=web_url,
created_by=self.__asf_uid,
)
```
#### `atr/models/distribution.py` — `Data` (lines 94-108)
_extension point_
@dave2wave identified this model class as needing extension based on
investigation findings (e.g., promotion capabilities, staging revision
tracking).
```python
class Data(schema.Subset):
platform: sql.DistributionPlatform
owner_namespace: safe.Alphanumeric | None = None
package: safe.Alphanumeric
version: safe.VersionKey
details: bool
@pydantic.field_validator("owner_namespace", mode="before")
@classmethod
def empty_to_none(cls, v):
if v is None:
return None
if isinstance(v, str) and (v.strip() == ""):
return None
return v
```
#### `atr/tasks/distribution.py` — `status_check` (lines 30-66)
_currently does this_
Background task that retries pending distributions. If promotion were
implemented, a similar task or extension would handle checking promotion status
on remote platforms.
```python
@checks.with_model(args.DistributionStatusCheckArgs)
async def status_check(
task_args: args.DistributionStatusCheckArgs, *, task_id: int | None =
None
) -> results.Results | None:
log.info("Checking pending recorded distributions")
dists = []
async with db.session() as data:
dists = await data.distribution(
pending=True, _with_release=True, _with_release_project=True,
_limit=_BATCH_SIZE
).all()
for dist in dists:
name = f"{dist.platform} {dist.owner_namespace} {dist.package}
{dist.version}"
dd = dist.distribution_data()
if not dist.created_by:
log.warning(f"Distribution {name} has no creator, skipping")
continue
if not dist.release.project.committee_key:
log.warning(f"Distribution {name} has no committee, skipping")
continue
try:
async with
storage.write_as_committee_member(dist.release.project.committee_key,
dist.created_by) as w:
if dist.retries >= _RETRY_LIMIT:
await w.distributions.delete_distribution(
dist.safe_release_key, dist.platform,
dist.owner_namespace, dist.package, dist.version
)
log.error(f"Distribution {name} failed {_RETRY_LIMIT}
times, skipping")
continue
log.warning(f"Retrying distribution {name}")
await w.distributions.record_from_data(
dist.safe_release_key,
dist.staging,
dd,
allow_retries=True,
)
except (distribution.DistributionError, storage.AccessError) as e:
msg = f"Failed to record distribution: {e}"
log.error(msg)
```
### Proposed approach
This issue is an active investigation/discussion. The team has not yet
converged on a final design. @andrewmusselman's latest RFC (May 1) proposes
corrections to the investigation document and a narrower schema proposal, but
this is still awaiting review from @sbp, @dave2wave, and @alitheg. No code
changes should be proposed until the team agrees on: (1) which platforms will
support promotion (currently leaning toward Docker Hub retagging, GitHub
prerelease→release, and npm dist-tags as 'maybe'), (2) what schema changes are
needed in `DistributionPlatformValue` (e.g., adding `supports_promotion`,
`promotion_mechanism` fields), and (3) how revision tracking will be used to
verify that promoted files match those voted on.
The existing code already has the foundation for staging→production upgrades
via `__upgrade_staging_to_final` in the distributions writer, but remote
promotion would require new platform-specific API calls (e.g., Docker manifest
retag, GitHub PATCH prerelease:false, npm dist-tag add) that don't yet exist.
The investigation should conclude with a decision document before
implementation begins.
### Open questions
- Which platforms will the team ultimately decide to support promotion for?
The investigation leans toward 'maybe' for Docker Hub, GitHub Releases, and npm.
- What schema changes to DistributionPlatformValue in atr/models/sql.py are
needed? @andrewmusselman's RFC proposes adding supports_promotion and
promotion_mechanism fields but this hasn't been approved.
- How will revision tracking be extended to verify that promoted artifacts
match those voted on? The issue body identifies this as a necessary
prerequisite.
- Should Hex/Erlang be added as a new DistributionPlatform member as
@dave2wave received an email about it?
- Does the investigation output belong in atr/docs/ or elsewhere?
@andrewmusselman raised this question.
_The agent reviewed this issue and is not proposing patches in this run.
Review the existing-code citations and open questions above before deciding
next steps._
### Files examined
- `atr/get/finish.py`
- `atr/get/distribution.py`
- `atr/models/distribution.py`
- `atr/post/distribution.py`
- `atr/post/finish.py`
- `atr/shared/distribution.py`
- `atr/storage/writers/distributions.py`
- `atr/tasks/distribution.py`
---
*Draft from a triage agent. A human reviewer should validate before merging
any change. The agent did not run tests or verify diffs apply.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
