morningman opened a new pull request, #70699:
URL: https://github.com/apache/doris/pull/70699

   Backport of #70662 to `branch-4.1`.
   
   ### Backport scope and verification
   
   - Resolve the older base-image and component Dockerfile conflicts using the 
runtime dependency and packaging changes from the merged source PR.
   - Omit `docker/runtime/all-in-one/4.1/` and its two script tests because 
that builder is absent from this release branch. The legacy all-in-one 
Dockerfile is unchanged. This scope was confirmed by the requester.
   - Preserve this branch's Broker image build: stage 
`extensions/apache_hdfs_broker`, consume the newly built Java 8/17 base, and 
skip Broker only when its payload is absent. Add two build-script tests for 
Broker, with and without MS.
   - Backport validation: all eight build-script tests pass; Bash syntax, 
ShellCheck using repository configuration, Python syntax and `git diff --cached 
--check` pass.
   - Docker image builds, image-layer inspection, vulnerability scans and 
cluster smoke tests were not rerun for this backport. Pre-submission status: 
Partially verified on the release branch.
   
   The original description below is retained for context. Its Testing section 
reports validation of the original master PR, including the all-in-one builder 
omitted here; those results are not release-branch reruns.
   
   ---
   
   ### Background
   
   Doris runtime images combine release binaries with OS packages, a JVM, and 
operational tools. FE/BE/MS share one component base; all-in-one and 
doris-compose assemble their own environments. Updating the component base 
alone therefore leaves the other builders' packages and copied artifacts 
unaffected.
   
   ### What problem does this PR solve?
   
   Issue Number: #69975
   Related PR: None
   PR Type: Enhancement
   
   Problem Summary:
   
   Image scans report outdated runtime dependencies, unnecessary development 
packages, and shared example private keys. Component, all-in-one, and compose 
builds now apply OS patches, use maintained JVMs, and sanitize payloads before 
final copies. More than half the diff is tests and migration documentation for 
these independent build paths. New FE payloads omit Arthas and example keys; 
MySQL SSL deployments must supply their own certificate stores.
   
   For example, updating a component image does not sanitize a separately 
supplied release directory:
   
   | Build case | Before | After |
   | --- | --- | --- |
   | Component script's base | Builds `base-latest`, consumes `base-6.0` | 
Every component consumes the base just built |
   | All-in-one `--source tarball` | Copies default MySQL keys from the release 
| Removes keys before the final copy, as in image/local modes |
   | Compose JaCoCo download | Extracts an unchecked ZIP | Rejects a mismatched 
SHA512 before extraction |
   
   ### Details
   
   - **Change map:** Runtime/build: +294/-278 lines; tests: +283/-4; 
documentation: +198/-91. No Doris engine or generated-output changes.
   
   | Part | Changed lines | Purpose |
   | --- | ---: | --- |
   | Base dependencies and shared patch policy | 161 | Maintain JVM/Go inputs 
and upgrade retained packages over HTTPS. |
   | Component and test-image packaging | 216 | Sanitize each input path and 
isolate build/coverage tools. |
   | Build flows | 195 | Verify release archives, consume the correct base, and 
forward image references. |
   
   - **Before and after:** Component bases contain one Temurin JDK and rebuilt 
Operator tools, excluding kernel headers and general profiling tools. 
All-in-one uses a pinned Temurin 17 JRE. Compose uses a pinned Temurin 17 JDK 
and checksum-verified JaCoCo 0.8.16; unzip and patchelf are absent from its 
runtime.
   - **Configs and defaults:** Component builds retain Java 8 for 2.x and Java 
17 for 3.x+; manual builds default to `JAVA_VERSION=17`. All-in-one uses 
`BASE_IMAGE` instead of `JDK_PKG`. Compose retains `JDK_IMAGE`, `OUTPUT_PATH`, 
and the coverage-agent path; external payloads use `ARTIFACT_SOURCE=dir` with a 
`doris-output` named context. Image overrides are build-time settings.
   - **Impact on existing users:** Rebuilt images immediately omit the removed 
artifacts. Container diagnostics need separate tools. Compose changes 
Debian/Liberica to Ubuntu Jammy/Temurin while preserving non-root execution and 
its timezone. No SQL or metadata migration is required.
   - **Migration:** Build compose images from the repository root; pass 
external release directories through the documented named context. Mount 
private MySQL stores and set `mysql_ssl_default_server_certificate`, 
`mysql_ssl_default_ca_certificate`, and their password properties. Java 8 
builds need the appropriate base override. Rollback redeploys previous image 
digests, including their older dependencies.
   
   ### Testing
   
   All checks ran locally.
   
   - Eight script tests passed: checksum rejection/formats, architecture/Java 
selection, optional MS, base handoff, all-in-one overrides, flavors and 
directory contexts.
   - Built all-in-one from component images, local output and release 
directories. ARM64 base/full smoke tests passed startup, payload checks, 
writes/reads, Stream Load and schema change.
   - Java UDF/JNI queries passed in all-in-one and compose; compose FE/BE ran 
as a non-root user through `gosu`.
   - JaCoCo generated readable coverage on Java 8/17; an incorrect SHA512 
failed before extraction. Its tools had no Trivy findings.
   - AMD64 all-in-one/compose runtime environments passed JVM/client/tool 
checks. Layer checks found no removed example keys or Arthas. A no-MS fixture 
verified filtering of unrelated output and executable permissions.
   - Earlier component checks covered both architectures, MS/FoundationDB 
startup and MySQL `VERIFY_IDENTITY`.
   
   | OS scan, Trivy 0.76.2 | Critical/High | Medium/Low |
   | --- | --- | --- |
   | Component bases, original four variants | 0 | 29 / 19 |
   | All-in-one | 0 | 17 / 16 |
   | Compose | 0 | 45 / 46 |
   
   Remaining OS findings had no fixed versions listed in the scan database. 
Application JAR vulnerabilities require separate updates.
   
   ### Notes for Reviewers
   
   Keep both Go utilities: entry points invoke `doris-debug`, and disaggregated 
BE uses `dorisctl`. Prune keys before final copies because later deletion 
leaves their bytes in shipped layers. Compose retains its writable payload 
modes for non-root operation and the JaCoCo license alongside its tools.
   
   ### Release note
   
   Runtime builders use maintained JVMs and patched packages and omit shared 
example MySQL keys and Arthas. SSL deployments must mount their own stores. 
Compose builds require the repository root as context; external releases use a 
named context. Previous image digests remain the rollback path.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: Yes; runtime and image READMEs updated. 
Update website Docker deployment migration instructions before release.
   - Pre-submission status: Partially verified: full AMD64 clusters, older 
Doris releases, and external-catalog/cloud compatibility matrices have not run.
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6).
   - Generated: Dockerfiles, scripts, tests, documentation, and this 
description.
   - Human review: not stated.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to