This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new bf5dfaface8 [fix](ci) harden the OSS upload: retry once, serialize
cron deploys, use the acceleration endpoint (#4165)
bf5dfaface8 is described below
commit bf5dfaface89d8d7b48bc8c054fb5a3974515bce
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Tue Sep 22 00:17:57 2026 +0800
[fix](ci) harden the OSS upload: retry once, serialize cron deploys, use
the acceleration endpoint (#4165)
## What
The 2026-09-21 12:05 cron deploy ([run
35597560821](https://github.com/apache/doris-website/actions/runs/35597560821/job/106325798481))
spent **2h04m** in "Upload files to OSS" and then failed, so the
asf-site deploy was skipped and the site stayed on the 08:43 build.
From the step log:
- The runner → OSS link was ~50x slower than usual for the whole two
hours: ~2 successful PUTs/s (the 08:06 run did ~110/s), and the final ~9
MB `.actioninfo` PUT alone took 16 minutes (6 s normally). #4163 had
changed the global CSS, so every HTML page (12,320 objects) had to be
re-sent. The slow segment was the GitHub runner (US) reaching the
bucket's regional endpoint over the public internet at Beijing evening
peak.
- **6 of the 12,320 PUTs** hit the SDK's 60 s timeouts (`net/http:
timeout awaiting response headers` ×4, `write tcp … i/o timeout` ×2).
The action exits 1 on any failed upload and has no retry (the OSS SDK it
embeds never reads its `RetryTimes`), so a 0.05% failure rate failed the
step - after 99.95% of the upload had gone through and been recorded in
the bucket's `.actioninfo` manifest.
## How
1. **Retry the upload step once** (`cron-deploy-website.yml`,
`manual-deploy-website.yml`). The first attempt gets `continue-on-error:
true`; a second identical step runs only when `steps.oss-upload.outcome
== 'failure'`, and a failure there still fails the job. The action
writes `.actioninfo` before exiting non-zero, so the retry only re-sends
the objects that failed plus the manifest. In this incident that would
have been 6 objects, and the job would have gone on to deploy asf-site
at ~15:25 instead of stopping.
2. **Concurrency group for the cron workflow** (`cron-deploy-website`,
`cancel-in-progress: false`). A slow run and the next 2-hourly trigger
otherwise push the same objects through the same pipe and race for the
manifest. The newer run queues instead; only the latest pending run is
kept, which is what a deploy-master cron wants. The manual workflow
deliberately gets no group: sharing one with the cron would let a cron
trigger evict a human's pending manual run, and separate manual runs are
rare and supervised.
3. **Upload through the transfer-acceleration endpoint** (all four
workflows that upload to the bucket). Transfer acceleration was enabled
on the website bucket on 2026-09-21, so the upload steps now use
`oss-accelerate.aliyuncs.com`: the runner enters Alibaba's backbone at
the nearest PoP instead of crossing the public internet to the bucket's
region. The endpoint is a public hostname, not a credential, so it is
written into the workflows rather than the `ALIYUN_OSS_ENDPOINT` secret
(changing that needs an INFRA ticket); the secret is now unreferenced
and can be deleted whenever convenient. The action prefixes the bucket
itself (`http://<bucket>.<endpoint>/…`), so the value stays a bare
hostname. Downloads are untouched: `cdnd.selectdb.com` keeps its
regional OSS origin, which already reaches the bucket over Alibaba's
network - pointing the CDN at the acceleration endpoint would only add a
hop and a second traffic bill.
Not done on purpose:
- **No `timeout-minutes` on the upload step.** The action writes its
manifest only at the very end, so killing a slow upload discards the
record of everything sent so far and the next run starts the whole
upload again. A 30-minute cap would have left the site undeployable for
as long as the slow period lasted, whereas the uncapped run plus one
retry would have deployed. A genuine hang is not possible either: every
request is bounded by the SDK's 30 s connect / 60 s read-write / 60 s
header timeouts. The job keeps GitHub's default 6 h ceiling.
## Follow-up outside this PR
- `cron-generate-pdf.yml` / `manual-generate-pdf.yml` upload `build-pdf`
with the action's default `incremental: true`, which would overwrite the
shared root `.actioninfo` with a manifest containing only the PDFs and
force the next website deploy to re-send all ~63k objects. It is latent
today because "Generate PDF" has failed on every recent run, but worth
`incremental: false` before anyone fixes the generator.
## Verified
All four files parse (js-yaml); the retry steps' `with:` blocks are
byte-identical to the first attempts'; `if:` uses `outcome` (not
`conclusion`, which `continue-on-error` reports as success). The SDK's
`urlMaker.Init` turns a bare hostname into
`http://<bucket>.<hostname>/`, so the endpoint value needs no scheme. An
anonymous request to `<bucket>.oss-accelerate.aliyuncs.com` is answered
by OSS (403 AccessDenied, as on the regional endpoint), i.e. the name
resolves and routes; whether acceleration is enabled on the bucket is
only visible in the console, where it was confirmed. Not exercisable
before merge: the workflows run on `schedule` / `workflow_dispatch` from
master - the first cron run after merge is the real test, and reverting
the endpoint commit alone restores the regional path.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01MvJw7xdszLCGBQdrvi7N9Z
---------
Co-authored-by: morningman <[email protected]>
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
---
.github/workflows/cron-deploy-website.yml | 40 +++++++++++++++++++++++++++--
.github/workflows/cron-generate-pdf.yml | 4 +--
.github/workflows/manual-deploy-website.yml | 27 +++++++++++++++++--
.github/workflows/manual-generate-pdf.yml | 4 +--
4 files changed, 67 insertions(+), 8 deletions(-)
diff --git a/.github/workflows/cron-deploy-website.yml
b/.github/workflows/cron-deploy-website.yml
index 3ba9185ee0b..6007b7495df 100644
--- a/.github/workflows/cron-deploy-website.yml
+++ b/.github/workflows/cron-deploy-website.yml
@@ -4,6 +4,15 @@ on:
schedule:
- cron: '0 */2 * * *' # Runs every 2 hours.
permissions: write-all
+# One deploy at a time. A run whose OSS upload crawls (the 2026-09-21 12:05
+# run spent two hours on it) must not overlap the next cron trigger: both
+# would push the same objects through the same slow pipe and race for the
+# bucket's .actioninfo manifest. Queue the newer run instead of cancelling
+# the running one - killing it mid-upload discards the manifest it writes at
+# the end, and the next run starts the whole upload over.
+concurrency:
+ group: cron-deploy-website
+ cancel-in-progress: false
jobs:
build-and-deploy:
runs-on: ubuntu-latest
@@ -85,13 +94,40 @@ jobs:
# ls ./build/assets/files/
- name: Upload files to OSS
+ id: oss-upload
+ # The action has no per-request retry (the OSS SDK it embeds
never
+ # reads its RetryTimes), so a single PUT hitting the SDK's 60s
+ # timeout among tens of thousands fails the whole step - on
+ # 2026-09-21 that was 6 of 12,320 uploads after two hours on a
slow
+ # link. It does write its .actioninfo manifest before exiting
+ # non-zero, so the retry below only re-sends the objects that
+ # failed, plus the manifest. continue-on-error keeps the job
alive
+ # for that retry; the retry step fails the job if it fails too.
+ continue-on-error: true
uses: ./.github/actions/aliyun-oss-website-action
with:
accessKeyId: ${{ secrets.ALIYUN_ACCESS_KEY_ID }}
accessKeySecret: ${{ secrets.ALIYUN_ACCESS_KEY_SECRET }}
bucket: ${{ secrets.ALIYUN_OSS_BUCKET }}
- # use your own endpoint
- endpoint: ${{ secrets.ALIYUN_OSS_ENDPOINT }}
+ # Transfer-acceleration endpoint of the website bucket
(enabled
+ # on the bucket on 2026-09-21): the runner enters Alibaba's
+ # backbone at the nearest PoP instead of crossing the public
+ # internet to the bucket's region. Uploads only - the CDN
keeps
+ # its regional origin for downloads.
+ endpoint: oss-accelerate.aliyuncs.com
+ folder: build
+ onlyUpload: true
+ otherCacheControl: 0
+ skipSetting: true
+
+ - name: Upload files to OSS (retry)
+ if: steps.oss-upload.outcome == 'failure'
+ uses: ./.github/actions/aliyun-oss-website-action
+ with:
+ accessKeyId: ${{ secrets.ALIYUN_ACCESS_KEY_ID }}
+ accessKeySecret: ${{ secrets.ALIYUN_ACCESS_KEY_SECRET }}
+ bucket: ${{ secrets.ALIYUN_OSS_BUCKET }}
+ endpoint: oss-accelerate.aliyuncs.com
folder: build
onlyUpload: true
otherCacheControl: 0
diff --git a/.github/workflows/cron-generate-pdf.yml
b/.github/workflows/cron-generate-pdf.yml
index 7d834303f07..87f85273c49 100644
--- a/.github/workflows/cron-generate-pdf.yml
+++ b/.github/workflows/cron-generate-pdf.yml
@@ -34,7 +34,7 @@ jobs:
accessKeyId: ${{ secrets.ALIYUN_ACCESS_KEY_ID }}
accessKeySecret: ${{ secrets.ALIYUN_ACCESS_KEY_SECRET }}
bucket: ${{ secrets.ALIYUN_OSS_BUCKET }}
- # use your own endpoint
- endpoint: ${{ secrets.ALIYUN_OSS_ENDPOINT }}
+ # Transfer-acceleration endpoint, same as the website deploys.
+ endpoint: oss-accelerate.aliyuncs.com
folder: build-pdf
onlyUpload: true
diff --git a/.github/workflows/manual-deploy-website.yml
b/.github/workflows/manual-deploy-website.yml
index ff2351cf33f..0bc5d14e748 100644
--- a/.github/workflows/manual-deploy-website.yml
+++ b/.github/workflows/manual-deploy-website.yml
@@ -63,13 +63,36 @@ jobs:
export DORIS_COMMIT=`git rev-parse HEAD`
- name: Upload files to OSS
+ id: oss-upload
+ # Same retry as cron-deploy-website.yml: the action has no
+ # per-request retry, one timed-out PUT fails the step, and the
+ # .actioninfo manifest it writes before exiting lets a second run
+ # re-send only what failed.
+ continue-on-error: true
uses: ./.github/actions/aliyun-oss-website-action
with:
accessKeyId: ${{ secrets.ALIYUN_ACCESS_KEY_ID }}
accessKeySecret: ${{ secrets.ALIYUN_ACCESS_KEY_SECRET }}
bucket: ${{ secrets.ALIYUN_OSS_BUCKET }}
- # use your own endpoint
- endpoint: ${{ secrets.ALIYUN_OSS_ENDPOINT }}
+ # Transfer-acceleration endpoint of the website bucket
(enabled
+ # on the bucket on 2026-09-21): the runner enters Alibaba's
+ # backbone at the nearest PoP instead of crossing the public
+ # internet to the bucket's region. Uploads only - the CDN
keeps
+ # its regional origin for downloads.
+ endpoint: oss-accelerate.aliyuncs.com
+ folder: build
+ onlyUpload: true
+ otherCacheControl: 0
+ skipSetting: true
+
+ - name: Upload files to OSS (retry)
+ if: steps.oss-upload.outcome == 'failure'
+ uses: ./.github/actions/aliyun-oss-website-action
+ with:
+ accessKeyId: ${{ secrets.ALIYUN_ACCESS_KEY_ID }}
+ accessKeySecret: ${{ secrets.ALIYUN_ACCESS_KEY_SECRET }}
+ bucket: ${{ secrets.ALIYUN_OSS_BUCKET }}
+ endpoint: oss-accelerate.aliyuncs.com
folder: build
onlyUpload: true
otherCacheControl: 0
diff --git a/.github/workflows/manual-generate-pdf.yml
b/.github/workflows/manual-generate-pdf.yml
index 5bf1858d364..a0f12caa338 100644
--- a/.github/workflows/manual-generate-pdf.yml
+++ b/.github/workflows/manual-generate-pdf.yml
@@ -42,7 +42,7 @@ jobs:
accessKeyId: ${{ secrets.ALIYUN_ACCESS_KEY_ID }}
accessKeySecret: ${{ secrets.ALIYUN_ACCESS_KEY_SECRET }}
bucket: ${{ secrets.ALIYUN_OSS_BUCKET }}
- # use your own endpoint
- endpoint: ${{ secrets.ALIYUN_OSS_ENDPOINT }}
+ # Transfer-acceleration endpoint, same as the website deploys.
+ endpoint: oss-accelerate.aliyuncs.com
folder: build-pdf
onlyUpload: true
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
