zgxme opened a new pull request, #70817:
URL: https://github.com/apache/doris/pull/70817
### Background
The thirdparty launcher starts external-regression fixtures from Compose
templates, builds a Kerberos fixture image, and runs helper containers for
filesystem operations and Hive baselines. These fixtures use images from
several public registries.
### What problem does this PR solve?
Issue Number: close #70816
Related PR: None
PR Type: Feature
Problem Summary:
CI operators using a local registry currently need to rewrite image
references across templates, build inputs and helper scripts. Add `native` and
`mapped` image modes configured through `custom_settings.env` or environment
variables, with one resolver and an explicit image catalog covering those entry
points. The change also preserves platform-specific defaults and user
overrides, provides image inventory and prefetch commands, and adds contract
tests. The default `native` mode preserves existing image references.
### Details
- **Change map:** +859/-78 across 34 files: fixture scripts, catalog and
templates +408/-77; behavior tests +305/-1; documentation +84; CI +62. Start
with the 197-line resolver and 72-line catalog, then the launcher integration.
Most template edits parameterize 69 existing image references. The diff
profiler groups `docker/thirdparties` under test infrastructure; these figures
separate that infrastructure by purpose.
- **Scope:** Compose images, Kerberos build bases, filesystem helpers and
Hive baseline tools under `docker/thirdparties`. Artifact downloads, cloud-data
credentials, host networking and external CI-adapter migration remain separate.
IPv6 registry literals are unsupported.
- **Design:** Resolve explicitly declared images after user and platform
defaults, retain tags/digests, and reject missing mappings for selected
components. Local build outputs stay local; explicit references under the
target registry remain unchanged. Generic mapping supports Harbor or another
registry. Runtime source rewriting was rejected because build bases and helper
containers need the same explicit contract. Proposed in #70816, opened with
this PR; no public design discussion has taken place yet.
- **Usage:**
```bash
export THIRDPARTY_IMAGE_MODE=mapped
export THIRDPARTY_IMAGE_REGISTRY=harbor.example.com:443
export THIRDPARTY_IMAGE_REGISTRY_MAP=docker.io=dockerhub
bash docker/thirdparties/run-thirdparties-docker.sh -c mysql --list-images
```
This configuration maps `mysql:5.7.36` to
`harbor.example.com:443/dockerhub/library/mysql:5.7.36`. `--pull-images`
fetches missing images without starting fixtures; `--image-config-version`
reports interface version `1` for CI adapters.
- **Default behavior:** `native` preserves references and existing Docker
Hub daemon mirrors. Registry and mapping default to empty. `mapped` prefetches
missing remote images before fixture mutation; cached tags are not
force-refreshed. Settings are read on each invocation.
- **Compatibility:** No FE/BE code, protocols, metadata or storage formats
change, so mixed-version clusters are unaffected. Native image defaults and
Iceberg's platform-specific PostgreSQL selection are retained. Fluss image
overrides survive environment rendering. Operators should use the launcher to
initialize image variables before operating generated Compose files. Existing
CI adapters must detect interface version `1` before replacing their legacy
rewriting path; older packages still need compatibility handling.
- **Persistence and failure handling:** Missing mappings and pull failures
stop deployment before fixture teardown or data initialization.
Inventory/version queries do not contact Docker. Stops do not probe registries
or prefetch images. Authentication and certificate trust use existing Docker
configuration; no database metadata is persisted by this feature.
- **Behavior change:** Opting into `mapped` routes image references through
the configured registry and prefetches missing images before deployment.
Default `native` image references are unchanged.
### Testing
- `python3 docker/thirdparties/test/test_image_registry.py`: 14 local tests
passed, covering mapping, digests, invalid input, private overrides, component
selection, platform defaults, helper/snapshot calls, failure-before-mutation
behavior and real Compose v2 interpolation. Daemon operations are stubbed.
- Five existing shell test scripts passed: Docker command, launcher stop,
Hive bootstrap groups, Fluss networking and JuiceFS mirror. The stop test
additionally covers mapped mode without registry access.
- All 69 Compose image defaults match the baseline in native mode. Bash
syntax checks, workflow YAML parsing and `git diff --check` passed.
- Live Harbor checks: all 49 remote manifests/config blobs passed from a
separate client and confirmed `linux/amd64`. The committed resolver/pull
functions completed 13 representative full pulls on a rootful Docker daemon: 11
new pulls and two verified cached images, covering Docker Hub, GHCR, Quay, ICR
and the digest-pinned Kerberos base.
- Rootless fixture startup, Kerberos build, live snapshot integration and
full external regression were not run. The remaining 36 images were checked
through manifests/config blobs only. ShellCheck was unavailable; the new GitHub
workflow has not run yet.
### Release note
Thirdparty regression fixtures support configurable image-registry mappings
and image inventory/prefetch commands. Native mode remains the default and
preserves existing image references.
### Check List (For Author)
- Behavior changed: Yes (see Details)
- Does this need documentation: Yes; included in
`docker/thirdparties/README.md`. No website update is needed for this fixture
tooling.
- Pre-submission status: Partially verified: rootless fixture startup,
builds and complete external regression remain unverified.
### AI Disclosure
- Tool: Codex (GPT-6)
- Generated: Code, tests, documentation and this description
- Human review: not stated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]