morningman opened a new pull request, #409:
URL: https://github.com/apache/doris-thirdparty/pull/409
## Problem
The Actions tab is solid red. Over the last 15 hours of scheduled runs:
| workflow | result |
|---|---|
| Build (master) | 28 / 28 green |
| Build (4.0) | 29 / 29 green |
| Build (1.2-lts) | **0 / 28** |
| Build (2.1) | **0 / 28** |
| Build (3.0) | **0 / 28** |
| Build (3.1) | **0 / 28** |
| Build (2.0) | 8 green (build skipped) / 20 red |
That is a handful of real failures amplified by one state machine bug.
## The amplifier
The pipeline keeps its entire state in the body of the `automation*` release.
`prerelease` reads `Doris Version:` out of it to decide whether anything
needs
building — but the `failure` job overwrote the note with only `Status:
*FAILURE*`
plus a checksum block, dropping that line:
```bash
# success job -- preserves the version
content="$(gh release view "${tag_name}" | sed -n '/Update Time:/,/Doris
Version:/p')"
echo -ne "${content}\nStatus: *SUCCESS*\n..." >release_note.md
# failure job -- overwrites the whole note, version is gone
echo -ne "Status: *FAILURE*\n..." >release_note.md
```
Thirty minutes later `prerelease` read an empty version, logged `The first
release was detected` and rebuilt everything, which failed again and wiped
the
version again. Straight from a run this morning:
```
Last Version:
Current Version: a3052de2e5cbc397d67b9cd9d5cd9701aaa6426c
```
`branch-1.2-lts` shows how bad this got: no commits in years, still running a
full macOS build every 30 minutes.
`manual-build.yml` had the same `failure` job and writes to the same
`automation` release as the scheduled master build, so one red manual build
could drop master into the loop too.
## The real failures underneath
- **2.1 / 3.0 / 3.1** — `LZO2_DOWNLOAD` in those branches still points at
`https://fossies.org/linux/misc/lzo-2.10.tar.gz`, which now answers **410
Gone**. master and 4.0 already moved to `oberhumer.com`. 3.1 additionally
shares 4.0's dead libuuid mirror, `nchc.dl.sourceforge.net`.
- **1.2-lts** — curl 7.79.0 no longer builds against the `libngtcp2 1.24.0`
on
the macOS runner image: `configure` finds ngtcp2, does not find
`libngtcp2_crypto_openssl` (renamed upstream), enables it anyway, and
`vquic/ngtcp2.c` dies on the missing header.
- **2.0** — the Linux job ran out of disk: `fatal error: error writing to
/tmp/ccAeBk6L.s: No space left on device` while building aws-sdk-cpp.
## Changes
**State machine**
- `failure` records the version it tried to build, exactly like `success`
does,
so the state can no longer be lost.
- That removes the accidental retry the loop was providing, so `prerelease`
gains a bounded one: a run whose predecessor ended in `FAILURE` retries up
to
3 times, then stops. A note with no parseable counter counts as out of
attempts, so a malformed note can never re-open the loop.
- `failure` is gated on `needs.prerelease.result == 'success'` so it never
writes an empty version.
**Scheduling**
- 1.2-lts, 2.0, 2.1 and 3.0 become `workflow_dispatch` only — no more
polling.
Trigger them from the Actions tab when a build is needed. Their `build` and
`success` jobs now honour `force_build` so a manual run actually builds.
- `concurrency` group per workflow. A full build takes hours while the
schedule
fires every 30 minutes, so runs were overlapping and racing on the same
tag.
**3.1, so it can go green**
- lzo and libuuid are pre-seeded in `prerelease` and repaired in `build`, the
same way `build-4.0.yml` already does it. No-ops once the upstream fix
lands,
since the checksums are identical.
- `fail-fast: false`. A single red platform used to cancel the other two, so
nobody could see whether Linux or Intel macOS were healthy.
- Drops `easimon/maximize-build-space`. It reserves only `root-reserve-mb` on
`/` and gcc writes its temporaries to `/tmp` on `/` — that is how 2.0 died.
master and 4.0 already build fine on the stock runner disk.
**General hardening**
- `prerelease` moves off `macos-14` for master and 3.1. Without GNU `md5sum`,
`download-thirdparty.sh` prints `Warn: md5sum is not installed`, skips
verification entirely and packs the 0-byte files failed downloads leave
behind
into the source tarball — so the build jobs fail on a checksum `prerelease`
never looked at.
- `gh release delete-asset ... --yes` before upload. `--yes` matters:
without it
gh cannot confirm the prompt on a runner, so the existing `delete-asset` in
`build-4.0.yml` silently did nothing and left the plain upload to fail.
- `actions/checkout` v4 to v5, clearing the Node 20 deprecation warning on
every
run.
## Upstream
Companion PRs fix the mirrors at the source in `apache/doris`:
- `branch-3.1`: lzo + libuuid
- `branch-4.0`: libuuid
## Testing
- Every workflow parses as YAML and every `run:` block passes `bash -n`.
- The rewritten `check_diff` state machine was extracted from `build-3.1.yml`
and run against stubbed `gh`/`git` over 11 cases: unchanged HEAD, new HEAD
with and without `thirdparty/` changes, the three retry steps, exhaustion
at
attempt 3, an unparseable counter, and a replay of the old broken note
showing
the loop now terminates.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
