asf-tooling commented on issue #751:
URL:
https://github.com/apache/tooling-trusted-releases/issues/751#issuecomment-4403683322
<!-- gofannon-issue-triage-bot v2 -->
**Automated triage** — analyzed at `main@837830e8`
**Type:** `discussion` • **Classification:** `no_action` •
**Confidence:** `high`
**Application domain(s):** `release_lifecycle`, `distribution_publishing`
### Summary
This is an active investigation/RFC about whether ATR should support remote
promotion of artifacts on third-party platforms (DockerHub, Maven Central, npm,
PyPI) instead of re-uploading. @dave2wave framed the investigation as filling
in platform capabilities in the SQL model, @andrewmusselman posted a
comprehensive draft and follow-up RFC with corrections and schema proposals.
The team has not yet reached consensus on an implementation approach — the most
recent comment (7 days ago) is an RFC requesting review.
### Where this lives in the code today
#### `atr/storage/writers/distributions.py` —
`CommitteeMember.__upgrade_staging_to_final` (lines 272-301)
_currently does this_
Existing upgrade-from-staging logic that flips the staging flag in the
database but does NOT perform any remote promotion API call on the platform
itself.
```python
async def __upgrade_staging_to_final(
self,
release_key: models.safe.ReleaseKey,
platform: models.sql.DistributionPlatform,
owner_namespace: str | None,
package: str,
version: str,
pending: bool,
upload_date: datetime.datetime | None,
api_url: str | None,
web_url: str | None,
) -> models.sql.Distribution | None:
tag = f"{release_key} {platform} {owner_namespace or ''} {package}
{version}"
existing = await self.__data.distribution(
release_key=str(release_key),
platform=platform,
owner_namespace=(owner_namespace or ""),
package=package,
version=version,
).demand(RuntimeError(f"Distribution {tag} not found"))
if existing.staging:
existing.staging = False
existing.pending = pending
existing.upload_date = upload_date
existing.api_url = api_url
existing.web_url = web_url
existing.created_by = self.__asf_uid
await self.__data.commit()
return existing
return None
```
#### `atr/storage/writers/distributions.py` — `CommitteeMember.record`
(lines 143-181)
_extension point_
The record method already detects the staging→production transition and
calls __upgrade_staging_to_final. This is where remote promotion API calls
would be injected, per the investigation.
```python
async def record(
self,
release_key: models.safe.ReleaseKey,
platform: models.sql.DistributionPlatform,
owner_namespace: models.safe.Alphanumeric | None,
package: models.safe.Alphanumeric,
version: models.safe.VersionKey,
staging: bool,
pending: bool,
upload_date: datetime.datetime | None,
api_url: str | None = None,
web_url: str | None = None,
) -> tuple[models.sql.Distribution, bool]:
namespace = str(owner_namespace) if owner_namespace else ""
existing = await self.__data.distribution(
str(release_key), platform, namespace, str(package), str(version)
).get()
...
# If we're doing production and existing was for staging, upgrade it
if (not staging) and existing.staging:
upgraded = await self.__upgrade_staging_to_final(
release_key,
platform,
namespace,
str(package),
str(version),
pending,
upload_date,
api_url,
web_url,
)
if upgraded is not None:
self.__write_as.append_to_audit_log(
asf_uid=self.__asf_uid,
operation="distribution_upgrade",
release_key=str(release_key),
platform=platform.name,
)
return upgraded, False
```
#### `atr/shared/distribution.py` — `DistributionPlatform` (lines 78-102)
_extension point_
The wrapper enum for distribution platforms; @dave2wave noted Erlang/Hex is
not modeled here at all. New platforms and promotion capabilities would extend
this.
```python
class DistributionPlatform(enum.Enum):
"""Wrapper enum for distribution platforms."""
ARTIFACT_HUB = "Artifact Hub"
DOCKER_HUB = "Docker Hub"
MAVEN = "Maven Central"
NPM = "npm"
NPM_SCOPED = "npm (scoped)"
PYPI = "PyPI"
def to_sql(self) -> sql.DistributionPlatform:
"""Convert to SQL enum."""
match self:
case DistributionPlatform.ARTIFACT_HUB:
return sql.DistributionPlatform.ARTIFACT_HUB
case DistributionPlatform.DOCKER_HUB:
return sql.DistributionPlatform.DOCKER_HUB
case DistributionPlatform.MAVEN:
return sql.DistributionPlatform.MAVEN
case DistributionPlatform.NPM:
return sql.DistributionPlatform.NPM
case DistributionPlatform.NPM_SCOPED:
return sql.DistributionPlatform.NPM_SCOPED
case DistributionPlatform.PYPI:
return sql.DistributionPlatform.PYPI
```
#### `atr/models/distribution.py` — `Data` (lines 94-108)
_extension point_
The distribution data model that @dave2wave referenced as needing updates
for new platform information and promotion capabilities.
```python
class Data(schema.Subset):
platform: sql.DistributionPlatform
owner_namespace: safe.Alphanumeric | None = None
package: safe.Alphanumeric
version: safe.VersionKey
details: bool
@pydantic.field_validator("owner_namespace", mode="before")
@classmethod
def empty_to_none(cls, v):
if v is None:
return None
if isinstance(v, str) and (v.strip() == ""):
return None
return v
```
#### `atr/tasks/distribution.py` — `status_check` (lines 30-61)
_currently does this_
Background retry task for pending distributions. Would need extension if
remote promotion becomes a new async operation type.
```python
@checks.with_model(args.DistributionStatusCheckArgs)
async def status_check(
task_args: args.DistributionStatusCheckArgs, *, task_id: int | None =
None
) -> results.Results | None:
log.info("Checking pending recorded distributions")
dists = []
async with db.session() as data:
dists = await data.distribution(
pending=True, _with_release=True, _with_release_project=True,
_limit=_BATCH_SIZE
).all()
for dist in dists:
name = f"{dist.platform} {dist.owner_namespace} {dist.package}
{dist.version}"
dd = dist.distribution_data()
...
try:
async with
storage.write_as_committee_member(dist.release.project.committee_key,
dist.created_by) as w:
if dist.retries >= _RETRY_LIMIT:
await w.distributions.delete_distribution(
dist.safe_release_key, dist.platform,
dist.owner_namespace, dist.package, dist.version
)
log.error(f"Distribution {name} failed {_RETRY_LIMIT}
times, skipping")
continue
log.warning(f"Retrying distribution {name}")
await w.distributions.record_from_data(
dist.safe_release_key,
dist.staging,
dd,
allow_retries=True,
)
except (distribution.DistributionError, storage.AccessError) as e:
msg = f"Failed to record distribution: {e}"
log.error(msg)
```
### Proposed approach
This issue is an active RFC/investigation. The team is gathering
platform-specific information to decide which platforms support remote
promotion and whether the complexity is justified. @andrewmusselman's last
comment (May 1) proposes narrower schema changes and patches but these are
explicitly RFC'd for team review. No implementation work should proceed until
the team converges on: (1) which platforms to support promotion for, (2) what
schema changes to `DistributionPlatformValue` are needed (e.g.,
`has_promotion`, `supports_staging`), (3) whether to add revision-tracking to
staged distributions, and (4) whether Hex/Erlang should be added as a platform
first. The existing `__upgrade_staging_to_final` method in `CommitteeMember` is
the natural extension point for injecting actual remote promotion API calls
once a decision is made.
### Open questions
- Which platforms will actually get remote promotion support? The draft
suggests Docker Hub, npm, and GitHub Releases are 'maybe' candidates.
- How should revision-tracking for staged distributions work? The issue body
notes that promoted files must match those voted on, but no schema for tracking
this exists yet.
- Should Hex/Erlang be added as a DistributionPlatform member before or
after the promotion investigation concludes?
- What fields should be added to DistributionPlatformValue (has_promotion,
supports_staging, etc.) and what's the minimal schema change the team agrees on?
- The last comment's patches are RFC'd but not accepted — needs explicit
team sign-off before implementation.
_The agent reviewed this issue and is not proposing patches in this run.
Review the existing-code citations and open questions above before deciding
next steps._
### Files examined
- `atr/get/finish.py`
- `atr/get/distribution.py`
- `atr/models/distribution.py`
- `atr/post/distribution.py`
- `atr/post/finish.py`
- `atr/shared/distribution.py`
- `atr/storage/writers/distributions.py`
- `atr/tasks/distribution.py`
---
*Draft from a triage agent. A human reviewer should validate before merging
any change. The agent did not run tests or verify diffs apply.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
