morningman opened a new pull request, #410: URL: https://github.com/apache/doris-thirdparty/pull/410
## What this changes Run [31988123966](https://github.com/apache/doris-thirdparty/actions/runs/31988123966) took **5h01m** end to end. Per-job: | job | wall clock | |---|---| | Build (Linux-arm64) | 1h27m54s | | Build (macOS-arm64) | 1h51m06s | | Build (Linux) | 2h45m49s | | Build (macOS-x86_64) | **4h22m41s** | | Update Docker Image | 25m24s, started 06:58 | Four things fall out of that, and this PR addresses each. ### 1. The Docker image waited 1h39m for nothing `update-docker` only consumes `doris-thirdparty-prebuilt-linux-x86_64.tar.xz`, which the Linux job published at **05:19**. But as `needs: build` it depended on the whole matrix, so it did not start until **06:58** - held up by macOS x86_64, which finished at 06:56. A matrix job cannot be depended on per-leg, so the build job becomes a reusable workflow (`.github/workflows/build-target.yml`) called once per platform. `update-docker` now depends on `build-linux-x86_64` alone. The provenance check it already runs against the archive is unchanged. Splitting this way is also what makes items 3 and 4 below expressible per platform. The package lists and the ldb-toolchain choice, which the matrix entries used to carry, are derived from `RUNNER_OS` / `RUNNER_ARCH` inside the reusable workflow - the two Linux entries and the two macOS entries were identical except for the toolchain script. ### 2. Every macOS build sat idle for exactly 10 minutes thrift's configure finds the dotnet the runner images ship, so `make` runs `dotnet build -c Release` for `lib/netstd`. The build itself finishes in seconds (`Build succeeded`, 12s on macos-14, 46s on macos-15-intel), and then the log goes completely silent before `Making install in compiler/cpp`: | runner | gap | |---|---| | macos-14 | 02:56:58.63 -> 03:06:58.66 = **600.03s** | | macos-15-intel | 04:00:38.09 -> 04:10:36.88 = **598.79s** | | ubuntu-22.04 | **0.01s** | An exact 10 minutes on both Macs is not compute, it is the compiler server that `dotnet build` leaves behind holding the stdout it inherited for its full keep-alive. `UseSharedCompilation=false` and `MSBUILDDISABLENODEREUSE=1` stop it from being started at all. 10 minutes back on each macOS job. ### 3. macOS x86_64 is 87% of the pipeline's wall clock `macos-15-intel` is roughly **2x slower** than the M1 runner at compiling (lance-c 41m43s vs 22m19s, grpc 2.1x, arrow 2.3x) and about **8x slower** at spawning processes - which is what a tree of 70+ autotools packages does all day. Identical configure scripts, same probe count: | package | macos-14 | ubuntu-22.04-arm | macos-15-intel | |---|---|---|---| | curl configure (576 probes) | 21s | 19s | **178s** | | openssl (~6500 log lines) | 43s | 34s | **373s** | It already runs with `continue-on-error`, so it is now built only once its published archive is older than `vars.MACOS_X86_MAX_AGE_DAYS` (default **7**) rather than on every thirdparty change. The prerelease job reads the asset's `updatedAt` off the release, so this self-regulates with no extra cron. `workflow_dispatch` with `force_build` still builds it unconditionally, and the threshold is a repository variable so it can be retuned without a code change. ### 4. Nothing was cached between runs `ccache` is installed on every runner, but nothing routed compilation through it and nothing persisted it, so a thirdparty change touching one package still rebuilt all ~80 from scratch. This sets `ENABLE_THIRDPARTY_CCACHE=ON` and carries `$CCACHE_DIR` across runs with `actions/cache` (`CCACHE_MAXSIZE=2G` compressed, four platforms inside the repository's 10 GB Actions cache budget). The cache is saved **even when the build fails**, which is the point: the prerelease job retries a failed build up to `max_attempts=3`, and today a flaky download mirror can cost three consecutive four-hour builds. This half only does something once apache/doris honours `ENABLE_THIRDPARTY_CCACHE` (apache/doris#66839). Until then it is an unused environment variable and the cache stays empty - the workflow is safe to merge in either order. ## Also `success` and `failure` now list every build job. `success` had to move to `!cancelled() && !failure()`: `build-macos-x86_64` is skipped on most runs, and a skipped dependency would otherwise skip `success` too, leaving the release note stuck at `BUILDING` - which is precisely the state that stops all future scheduled runs from doing anything. ## Testing `actionlint` is clean on both files. The behaviour itself can only be exercised by running the pipeline; the reusable workflow is a faithful move of the existing steps, with `matrix.config.name` / `matrix.config.os` replaced by `inputs.target` / `RUNNER_OS` / `RUNNER_ARCH`, plus the ccache and dotnet environment above. Worth a `workflow_dispatch` run with `force_build=true` before relying on it, so that all four platforms and the Docker job are exercised in one go. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
