zgxme opened a new pull request, #70817:
URL: https://github.com/apache/doris/pull/70817

   ### Background
   
   The thirdparty launcher starts external-regression fixtures from Compose 
templates, builds a Kerberos fixture image, and runs helper containers for 
filesystem operations and Hive baselines. These fixtures use images from 
several public registries.
   
   ### What problem does this PR solve?
   
   Issue Number: close #70816
   Related PR: None
   PR Type: Feature
   
   Problem Summary:
   
   CI operators using a local registry currently need to rewrite image 
references across templates, build inputs and helper scripts. Add `native` and 
`mapped` image modes configured through `custom_settings.env` or environment 
variables, with one resolver and an explicit image catalog covering those entry 
points. The change also preserves platform-specific defaults and user 
overrides, provides image inventory and prefetch commands, and adds contract 
tests. The default `native` mode preserves existing image references.
   
   ### Details
   
   - **Change map:** +859/-78 across 34 files: fixture scripts, catalog and 
templates +408/-77; behavior tests +305/-1; documentation +84; CI +62. Start 
with the 197-line resolver and 72-line catalog, then the launcher integration. 
Most template edits parameterize 69 existing image references. The diff 
profiler groups `docker/thirdparties` under test infrastructure; these figures 
separate that infrastructure by purpose.
   - **Scope:** Compose images, Kerberos build bases, filesystem helpers and 
Hive baseline tools under `docker/thirdparties`. Artifact downloads, cloud-data 
credentials, host networking and external CI-adapter migration remain separate. 
IPv6 registry literals are unsupported.
   - **Design:** Resolve explicitly declared images after user and platform 
defaults, retain tags/digests, and reject missing mappings for selected 
components. Local build outputs stay local; explicit references under the 
target registry remain unchanged. Generic mapping supports Harbor or another 
registry. Runtime source rewriting was rejected because build bases and helper 
containers need the same explicit contract. Proposed in #70816, opened with 
this PR; no public design discussion has taken place yet.
   - **Usage:**
   
     ```bash
     export THIRDPARTY_IMAGE_MODE=mapped
     export THIRDPARTY_IMAGE_REGISTRY=harbor.example.com:443
     export THIRDPARTY_IMAGE_REGISTRY_MAP=docker.io=dockerhub
     bash docker/thirdparties/run-thirdparties-docker.sh -c mysql --list-images
     ```
   
     This configuration maps `mysql:5.7.36` to 
`harbor.example.com:443/dockerhub/library/mysql:5.7.36`. `--pull-images` 
fetches missing images without starting fixtures; `--image-config-version` 
reports interface version `1` for CI adapters.
   - **Default behavior:** `native` preserves references and existing Docker 
Hub daemon mirrors. Registry and mapping default to empty. `mapped` prefetches 
missing remote images before fixture mutation; cached tags are not 
force-refreshed. Settings are read on each invocation.
   - **Compatibility:** No FE/BE code, protocols, metadata or storage formats 
change, so mixed-version clusters are unaffected. Native image defaults and 
Iceberg's platform-specific PostgreSQL selection are retained. Fluss image 
overrides survive environment rendering. Operators should use the launcher to 
initialize image variables before operating generated Compose files. Existing 
CI adapters must detect interface version `1` before replacing their legacy 
rewriting path; older packages still need compatibility handling.
   - **Persistence and failure handling:** Missing mappings and pull failures 
stop deployment before fixture teardown or data initialization. 
Inventory/version queries do not contact Docker. Stops do not probe registries 
or prefetch images. Authentication and certificate trust use existing Docker 
configuration; no database metadata is persisted by this feature.
   - **Behavior change:** Opting into `mapped` routes image references through 
the configured registry and prefetches missing images before deployment. 
Default `native` image references are unchanged.
   
   ### Testing
   
   - `python3 docker/thirdparties/test/test_image_registry.py`: 14 local tests 
passed, covering mapping, digests, invalid input, private overrides, component 
selection, platform defaults, helper/snapshot calls, failure-before-mutation 
behavior and real Compose v2 interpolation. Daemon operations are stubbed.
   - Five existing shell test scripts passed: Docker command, launcher stop, 
Hive bootstrap groups, Fluss networking and JuiceFS mirror. The stop test 
additionally covers mapped mode without registry access.
   - All 69 Compose image defaults match the baseline in native mode. Bash 
syntax checks, workflow YAML parsing and `git diff --check` passed.
   - Live Harbor checks: all 49 remote manifests/config blobs passed from a 
separate client and confirmed `linux/amd64`. The committed resolver/pull 
functions completed 13 representative full pulls on a rootful Docker daemon: 11 
new pulls and two verified cached images, covering Docker Hub, GHCR, Quay, ICR 
and the digest-pinned Kerberos base.
   - Rootless fixture startup, Kerberos build, live snapshot integration and 
full external regression were not run. The remaining 36 images were checked 
through manifests/config blobs only. ShellCheck was unavailable; the new GitHub 
workflow has not run yet.
   
   ### Release note
   
   Thirdparty regression fixtures support configurable image-registry mappings 
and image inventory/prefetch commands. Native mode remains the default and 
preserves existing image references.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: Yes; included in 
`docker/thirdparties/README.md`. No website update is needed for this fixture 
tooling.
   - Pre-submission status: Partially verified: rootless fixture startup, 
builds and complete external regression remain unverified.
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6)
   - Generated: Code, tests, documentation and this description
   - Human review: not stated
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to