lokidundun opened a new pull request, #3194:
URL: https://github.com/apache/hugegraph/pull/3194
<!--
Thank you very much for contributing to Apache HugeGraph, we are happy
that you want to help us improve it!
Here are some tips for you:
1. If this is your first time, please read the [contributing
guidelines](https://github.com/apache/hugegraph/blob/master/docs/CONTRIBUTING.md)
2. If a PR fix/close an issue, type the message "close xxx" (xxx is the
link of related
issue) in the content, GitHub will auto link it (Required)
3. Name the PR title in "Google Commit Format", start with "feat | fix |
perf | refactor | doc | chore",
such like: "feat(core): support the PageRank algorithm" or "fix: wrong
break in the compute loop" (module is optional)
skip it if you are unsure about which is the best component.
4. One PR address one issue, better not to mix up multiple issues.
5. Put an `x` in the `[ ]` to mark the item as CHECKED. `[x]` (or click
it directly after
published)
-->
## Purpose of the PR
- close #3179 <!-- or use "fix #xxx", "xxx" is the ID-link of related issue,
e.g: close #1024 -->
<!--
Please explain more context in this section, clarify why the changes are
needed.
e.g:
- If you propose a new API, clarify the use case for a new API.
- If you fix a bug, you can clarify why it is a bug, and should be
associated with an issue.
-->
## Main Changes
- Move runtime package installation before `COPY --from=build` in all four
Dockerfiles, allowing dependency layers to survive application source changes.
- Keep configuration edits that depend on copied files after the
corresponding `COPY`.
- Build the Server, PD, and Store distribution modules with `-pl` and `-am`,
reducing the reactor from 38 to 27 modules while retaining required
dependencies.
- Preserve existing runtime packages, configurable Maven arguments, and
identical shared Maven build stages across all four Dockerfiles.
The existing single-job Bake flow and QEMU-based ARM64 build remain in place.
## Verifying these changes
- [ ] Trivial rework / code cleanup without any test coverage. (No Need)
- [x] Already covered by existing checks: multi-platform image inspection,
Compose integration, Gremlin CRUD, and Standalone smoke tests.
- [x] Additional verification: before/after benchmarks and
distribution/image content comparisons.
### Runtime dependency layer benchmark
Both variants used the same GitHub-hosted Ubuntu runner class and separate
GHCR registry caches. After seeding each cache, identical configuration
comments were added to trigger distribution changes and a full Maven rebuild.
| Scenario / metric | Before | After | Reduction |
| --- | ---: | ---: | ---: |
| Initial cache seed: image build | 552 s | 510 s | 42 s (7.6%) |
| Initial cache seed: complete job | 704 s | 669 s | 35 s (5.0%) |
| Source change: image build | 601 s | 461 s | 140 s (23.3%) |
| Source change: complete job | 755 s | 602 s | 153 s (20.3%) |
Before the change, the four ARM64 runtime package installation steps ran
under QEMU and took 150.6–160.5 seconds each. After the change, all eight
runtime dependency layers across amd64 and arm64 were restored from registry
cache in 2.0–9.3 seconds per layer, without rerunning package installation.
Cache export was measured separately:
| Cache export vertex | Before | After |
| --- | ---: | ---: |
| Shared build cache | 205.2 s | 165.7 s |
| PD | 87.8 s | 5.5 s |
| Store | 56.1 s | 4.9 s |
| HStore Server | 83.3 s | 6.6 s |
| Standalone Server | 40.3 s | 8.1 s |
These BuildKit vertices overlap; their durations must not be added together
or treated as independent end-to-end savings.
### Maven reactor benchmark
This separate experiment compared the full reactor with the scoped reactor
using the same source and existing runtime layer optimization.
| Metric | Full reactor | Scoped reactor | Reduction |
| --- | ---: | ---: | ---: |
| Reactor modules | 38 | 27 | 11 modules |
| Maven-reported build time | 270 s | 176 s | 94 s (34.8%) |
| Distribution build, cache and local export | 424 s | 269 s | 155 s (36.6%)
|
| Complete distribution job, including artifact upload | 492 s | 351 s | 141
s (28.7%) |
The distribution-job timings above are not complete image publication
timings.
### Correctness checks
- Compared 391 distribution files, including permissions and normalized
nested JAR contents.
- Compared all four images on both amd64 and arm64: 780 records per
architecture covering application files, installed packages, and image
configuration, with no missing, added, or changed records.
- JAR comparisons normalized ZIP timestamps, compression, entry order, and
specified build-time metadata; this verifies effective content equivalence
rather than byte-identical archives.
- The runtime-layer CI built and published all four images with both
architectures, passing local platform inspection, Compose, Gremlin CRUD,
Standalone smoke tests, and published manifest checks. Functional checks in
this flow ran on amd64.
- A separate native ARM experiment passed functional checks on both
architectures and final multi-platform publication. That experimental workflow
is not included in this PR.
### Successful CI runs
| Validation | Result | CI run |
| --- | --- | --- |
| Runtime layer benchmark, multi-platform build and publication | All 4 jobs
passed |
[33950351952](https://github.com/lokidundun/incubator-hugegraph/actions/runs/33950351952)
|
| Full/scoped distribution benchmark, content comparison, and native
architecture validation | All 6 jobs passed |
[33953324289](https://github.com/lokidundun/incubator-hugegraph/actions/runs/33953324289)
|
| Full/scoped image content comparison on amd64 and arm64 | Both jobs passed
|
[33953049148](https://github.com/lokidundun/incubator-hugegraph/actions/runs/33953049148)
|
These are fork validation runs for the implementation approach. The Maven
experiments enabled the same module selection through `MAVEN_ARGS`; this PR
places that selection directly in the Dockerfiles. These runs do not replace CI
on the final PR commit.
Measurements are single-run observations using GHCR, not the official Docker
Hub publication environment. Savings from separate experiments are not additive.
### Commands for local verification
Run from the repository root using Bash.
Build the selected distributions:
```bash
mvn install \
-pl
hugegraph-server/hugegraph-dist,hugegraph-pd/hg-pd-dist,hugegraph-store/hg-store-dist
\
-am -e -B -ntp \
-Dmaven.test.skip=true \
-Dmaven.javadoc.skip=true
```
Inspect the shared Bake configuration:
```bash
docker buildx bake -f docker/bake.hcl --print
```
Build all four images through the shared Bake flow:
```bash
docker buildx bake -f docker/bake.hcl --progress=plain
```
The default Bake targets include amd64 and arm64. Local multi-platform
loading requires a compatible builder and the containerd image store; building
ARM64 on an amd64 host also requires emulation.
Check a direct Dockerfile build:
```bash
docker buildx build \
--platform linux/amd64 \
-f hugegraph-server/Dockerfile \
-t hugegraph-standalone:pr-check \
--load .
```
The commands above are build checks; the linked CI runs provide the
integration, content-comparison, and publication validation.
## Does this PR potentially affect the following parts?
- [ ] Dependencies
- [ ] Modify configurations
- [ ] The public API
- [x] Other affects: Docker build layer caching and Maven reactor selection.
- [ ] Nope
## Documentation Status
- [ ] `Doc - TODO`
- [ ] `Doc - Done`
- [x] `Doc - No Need`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
