morningman opened a new pull request, #70699: URL: https://github.com/apache/doris/pull/70699
Backport of #70662 to `branch-4.1`. ### Backport scope and verification - Resolve the older base-image and component Dockerfile conflicts using the runtime dependency and packaging changes from the merged source PR. - Omit `docker/runtime/all-in-one/4.1/` and its two script tests because that builder is absent from this release branch. The legacy all-in-one Dockerfile is unchanged. This scope was confirmed by the requester. - Preserve this branch's Broker image build: stage `extensions/apache_hdfs_broker`, consume the newly built Java 8/17 base, and skip Broker only when its payload is absent. Add two build-script tests for Broker, with and without MS. - Backport validation: all eight build-script tests pass; Bash syntax, ShellCheck using repository configuration, Python syntax and `git diff --cached --check` pass. - Docker image builds, image-layer inspection, vulnerability scans and cluster smoke tests were not rerun for this backport. Pre-submission status: Partially verified on the release branch. The original description below is retained for context. Its Testing section reports validation of the original master PR, including the all-in-one builder omitted here; those results are not release-branch reruns. --- ### Background Doris runtime images combine release binaries with OS packages, a JVM, and operational tools. FE/BE/MS share one component base; all-in-one and doris-compose assemble their own environments. Updating the component base alone therefore leaves the other builders' packages and copied artifacts unaffected. ### What problem does this PR solve? Issue Number: #69975 Related PR: None PR Type: Enhancement Problem Summary: Image scans report outdated runtime dependencies, unnecessary development packages, and shared example private keys. Component, all-in-one, and compose builds now apply OS patches, use maintained JVMs, and sanitize payloads before final copies. More than half the diff is tests and migration documentation for these independent build paths. New FE payloads omit Arthas and example keys; MySQL SSL deployments must supply their own certificate stores. For example, updating a component image does not sanitize a separately supplied release directory: | Build case | Before | After | | --- | --- | --- | | Component script's base | Builds `base-latest`, consumes `base-6.0` | Every component consumes the base just built | | All-in-one `--source tarball` | Copies default MySQL keys from the release | Removes keys before the final copy, as in image/local modes | | Compose JaCoCo download | Extracts an unchecked ZIP | Rejects a mismatched SHA512 before extraction | ### Details - **Change map:** Runtime/build: +294/-278 lines; tests: +283/-4; documentation: +198/-91. No Doris engine or generated-output changes. | Part | Changed lines | Purpose | | --- | ---: | --- | | Base dependencies and shared patch policy | 161 | Maintain JVM/Go inputs and upgrade retained packages over HTTPS. | | Component and test-image packaging | 216 | Sanitize each input path and isolate build/coverage tools. | | Build flows | 195 | Verify release archives, consume the correct base, and forward image references. | - **Before and after:** Component bases contain one Temurin JDK and rebuilt Operator tools, excluding kernel headers and general profiling tools. All-in-one uses a pinned Temurin 17 JRE. Compose uses a pinned Temurin 17 JDK and checksum-verified JaCoCo 0.8.16; unzip and patchelf are absent from its runtime. - **Configs and defaults:** Component builds retain Java 8 for 2.x and Java 17 for 3.x+; manual builds default to `JAVA_VERSION=17`. All-in-one uses `BASE_IMAGE` instead of `JDK_PKG`. Compose retains `JDK_IMAGE`, `OUTPUT_PATH`, and the coverage-agent path; external payloads use `ARTIFACT_SOURCE=dir` with a `doris-output` named context. Image overrides are build-time settings. - **Impact on existing users:** Rebuilt images immediately omit the removed artifacts. Container diagnostics need separate tools. Compose changes Debian/Liberica to Ubuntu Jammy/Temurin while preserving non-root execution and its timezone. No SQL or metadata migration is required. - **Migration:** Build compose images from the repository root; pass external release directories through the documented named context. Mount private MySQL stores and set `mysql_ssl_default_server_certificate`, `mysql_ssl_default_ca_certificate`, and their password properties. Java 8 builds need the appropriate base override. Rollback redeploys previous image digests, including their older dependencies. ### Testing All checks ran locally. - Eight script tests passed: checksum rejection/formats, architecture/Java selection, optional MS, base handoff, all-in-one overrides, flavors and directory contexts. - Built all-in-one from component images, local output and release directories. ARM64 base/full smoke tests passed startup, payload checks, writes/reads, Stream Load and schema change. - Java UDF/JNI queries passed in all-in-one and compose; compose FE/BE ran as a non-root user through `gosu`. - JaCoCo generated readable coverage on Java 8/17; an incorrect SHA512 failed before extraction. Its tools had no Trivy findings. - AMD64 all-in-one/compose runtime environments passed JVM/client/tool checks. Layer checks found no removed example keys or Arthas. A no-MS fixture verified filtering of unrelated output and executable permissions. - Earlier component checks covered both architectures, MS/FoundationDB startup and MySQL `VERIFY_IDENTITY`. | OS scan, Trivy 0.76.2 | Critical/High | Medium/Low | | --- | --- | --- | | Component bases, original four variants | 0 | 29 / 19 | | All-in-one | 0 | 17 / 16 | | Compose | 0 | 45 / 46 | Remaining OS findings had no fixed versions listed in the scan database. Application JAR vulnerabilities require separate updates. ### Notes for Reviewers Keep both Go utilities: entry points invoke `doris-debug`, and disaggregated BE uses `dorisctl`. Prune keys before final copies because later deletion leaves their bytes in shipped layers. Compose retains its writable payload modes for non-root operation and the JaCoCo license alongside its tools. ### Release note Runtime builders use maintained JVMs and patched packages and omit shared example MySQL keys and Arthas. SSL deployments must mount their own stores. Compose builds require the repository root as context; external releases use a named context. Previous image digests remain the rollback path. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes; runtime and image READMEs updated. Update website Docker deployment migration instructions before release. - Pre-submission status: Partially verified: full AMD64 clusters, older Doris releases, and external-catalog/cloud compatibility matrices have not run. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Dockerfiles, scripts, tests, documentation, and this description. - Human review: not stated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
