This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 82ccae85825 [ci](i18n) Share one translation workflow and keep batches
across unrelated merges (#4413)
82ccae85825 is described below
commit 82ccae858254f24b1b3a85f181ec289e386d33ae
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Tue Oct 20 14:09:59 2026 +0100
[ci](i18n) Share one translation workflow and keep batches across unrelated
merges (#4413)
## Why
`japanese-translation.yml` and `korean-translation.yml` are about 210
identical lines with five logical differences, and three of those (the
label check before model calls, the exact `paths` input, the fixed
locale) had been added to the Korean side only. Comparing them also
turned up two behaviours worth fixing:
- **Master movement discarded whole batches after the model calls.**
Generation checked master only after the last page, and any move
discarded everything. Two of the last 12 Japanese runs failed this way:
on 10-18 after 50 minutes of generation; on 10-19 after 25 minutes,
although master had already moved before generation started (the
generate job waited 40 minutes for a runner after discovery). Most of
those moves were unrelated merges.
- **The Japanese workflow's `locale` input was a second way to run
Korean generation**, with the Japanese defaults: manual `generate`, no
label check, `japanese-translation-*` artifacts.
## What changed
### 1. One reusable workflow
`.github/workflows/translation-run.yml` (`workflow_call`) holds the
discovery and generation jobs. The Japanese and Korean workflows keep
only their triggers, manual defaults, the lock and one call with their
locale and artifact prefix. Secrets are passed explicitly, not
inherited. Mode and filters are set once as workflow-level env instead
of two to four times per file. Artifact names are unchanged
(`japanese-translation-plan-<run>`,
`korean-translation-generation-<run>`, …).
### 2. Same inputs and checks for both languages
- Japanese gains the exact `paths` input and the label check before auth
loading, which is now `translate-$I18N_LOCALE`.
- The Japanese `locale` input is removed. Each workflow runs only its
own language.
### 3. Unrelated master merges no longer discard a batch
New `scripts/i18n/master-drift.cjs` decides what a master move means for
a batch:
- **Unrelated merge:** the batch stays valid. The branch stays on the
source commit, the PR body records the newer master, and the PR checks
run against current master.
- **Change to a batch page** (English source, target or state shard):
that page is dropped and stays pending for the next run. A Dev/4.x copy
is dropped with its primary. The publisher stops instead.
- **Change to translation configuration, routing identity or the
pipeline:** the batch is discarded before any further model call. This
covers the locale's scope, glossary, protection exceptions, inventory
and top-level state files, `translations/config.json`, `config/i18n.js`,
`docusaurus.config.js`, `versions.json` and `scripts/i18n/`. A master
that no longer contains the source commit is discarded too.
When the checks run:
- Generation checks master before the first model call, before each page
and after the last one. The publisher checks again before pushing.
- The generate job checks out current master instead of the discovery
commit and replans, so a long runner queue no longer dooms the batch
before it starts.
### 4. The replan reuses an unchanged discovery plan
`workflow-scan.cjs` records what a plan depends on: the checkout, the
open-PR reservations and the filters. The generate job reuses the
downloaded discovery plan when all three are unchanged, which skips the
2.5–4.7 minute recount of every native document. The planner is
deterministic.
### 5. Lock name
The shared concurrency group is renamed from `japanese-translation-ja`
to `docs-translation`.
### 6. Korean labels for the new Hive and Lance Catalog categories
#4410 turned the Hive and Lance catalog pages into sidebar categories
with type-mapping subpages, which created four new Korean dictionary
keys (Dev and 4.x). The translation pipeline translates documents but
never writes sidebar labels, so the Korean coverage audit reported them
as missing. Like the existing Iceberg, Paimon, Hudi and MaxCompute
Catalog categories, they name Doris catalog connectors and stay in
English. They are added in sidebar order, and each one has its reason
recorded in `translations/scope/ko.json`.
### Considered and not adopted
A `blob:none` checkout. Reconciliation reads the managed files of every
open translation PR with `git show`. In a partial clone each of those
blobs is a separate network fetch, which costs more than the ~30 s the
checkout would save once several translation PRs are open.
## Verification
- All 21 translation test suites listed in Build Check pass on Node
20.20.2 (193 tests). `workflow.test.cjs` has 9 new tests, and the
workflow contract tests grew from 5 to 7.
- New tests cover:
- drift detection against a real local bare remote: unrelated merge,
page change, moved source, configuration change, rewritten master;
- generation stopping before any model call on a batch-wide change, both
before the batch and before a page;
- a page changed before its turn being skipped, and a page changed after
its model call being dropped with its Dev/4.x copy;
- the manifest being kept when a batch is discarded after the last page;
- the plan fingerprint, and the replan reusing a discovery plan without
recounting;
- the label check failing for a missing label;
- the caller/shared workflow contract.
- Mutation checks: each of these breaks a test — removing the per-page
check, moving it inside the per-page `catch`, keeping copies of dropped
pages, turning rename detection back on, removing the ancestor check.
- `scripts/i18n/simulate-workflow.cjs` (local bare remote, GitHub CLI
and Codex mocks) passes for the default, incremental and ten-page
fallback scenarios, and for the new
`I18N_SIM_MASTER_DRIFT=unrelated|page|config`.
- `unrelated` publishes and notes the newer master in the PR body.
- `page` skips the page with no model request.
- `config` stops before any model request.
- The default scenario failed on master before this PR (three requests
instead of two). Its page, `engines.md`, declares `"language": "en"`,
and the Codex mock copied it. The mock now localizes a declared front
matter language, as a real model does.
- Korean coverage audit on this branch: sidebar labels missing 0 (was
4), retained technical 31. `check-configuration --all` and the Korean
translation gate (`check-translations --base c99440a`) report no errors.
- actionlint 1.7.12 reports nothing new. The two remaining findings also
exist on master: the `queue` concurrency key (newer than actionlint) and
the age of `actions/setup-python@v4`.
## Merge note
Merge while no Japanese or Korean translation run is queued or running.
Because the lock is renamed, a run started under the old name and a run
started under the new one could otherwise overlap. Today's scheduled
runs start at 16:23 (Japanese) and 16:43 UTC (Korean).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: morningman <[email protected]>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
---
.github/workflows/build-check.yml | 2 +-
.github/workflows/japanese-translation.yml | 217 ++-------------
.github/workflows/korean-translation.yml | 224 ++--------------
...{korean-translation.yml => translation-run.yml} | 115 ++++----
.../ko/docusaurus-plugin-content-docs/current.json | 8 +
.../version-4.x.json | 8 +
scripts/i18n/__tests__/korean-workflow.test.cjs | 102 -------
scripts/i18n/__tests__/multi-locale.test.cjs | 5 +-
.../i18n/__tests__/translation-workflows.test.cjs | 145 ++++++++++
scripts/i18n/__tests__/workflow.test.cjs | 297 +++++++++++++++++----
scripts/i18n/master-drift.cjs | 57 ++++
scripts/i18n/simulate-workflow.cjs | 54 +++-
scripts/i18n/workflow-generate.cjs | 61 +++--
scripts/i18n/workflow-publish.cjs | 12 +-
scripts/i18n/workflow-scan.cjs | 63 +++--
translations/README.md | 73 +++--
translations/scope/ko.json | 24 ++
17 files changed, 797 insertions(+), 670 deletions(-)
diff --git a/.github/workflows/build-check.yml
b/.github/workflows/build-check.yml
index deccbbae576..0afc09c1992 100644
--- a/.github/workflows/build-check.yml
+++ b/.github/workflows/build-check.yml
@@ -119,7 +119,7 @@ jobs:
node --test scripts/i18n/__tests__/planner.test.cjs
node --test
scripts/i18n/__tests__/translation-engine.test.cjs
node --test scripts/i18n/__tests__/korean-toolchain.test.cjs
- node --test scripts/i18n/__tests__/korean-workflow.test.cjs
+ node --test
scripts/i18n/__tests__/translation-workflows.test.cjs
node --test scripts/i18n/__tests__/workflow.test.cjs
node --test scripts/i18n/__tests__/codex-adapter.test.cjs
node --test
scripts/i18n/__tests__/translate-document.test.cjs
diff --git a/.github/workflows/japanese-translation.yml
b/.github/workflows/japanese-translation.yml
index 1e5b856030e..6ff02492752 100644
--- a/.github/workflows/japanese-translation.yml
+++ b/.github/workflows/japanese-translation.yml
@@ -18,9 +18,8 @@ on:
path:
description: Optional repository-relative source path prefix
type: string
- locale:
- default: ja
- description: Configured translation locale (defaults to ja)
+ paths:
+ description: Optional JSON array of exact English .md/.mdx files
(exclusive with path)
type: string
schedule:
- cron: '23 4,16 * * *'
@@ -30,198 +29,28 @@ permissions:
pull-requests: read
concurrency:
- # One entire run across scheduled/manual triggers, locales and modes.
- # Keep this stable so an in-flight run also holds the same lock.
- group: japanese-translation-ja
+ # One entire run at a time across both translation workflows, every trigger
+ # and every mode. Rename it only while no translation run is queued or
+ # running; otherwise the run in flight and the next one hold different locks.
+ group: docs-translation
cancel-in-progress: false
queue: max
jobs:
- discover:
- if: github.ref == 'refs/heads/master'
- runs-on: ubuntu-latest
- timeout-minutes: 20
- env:
- I18N_LOCALE: ${{ inputs.locale || 'ja' }}
- outputs:
- can_generate: ${{ steps.scan.outputs.can_generate }}
- source_commit: ${{ steps.scan.outputs.source_commit }}
- steps:
- - uses: actions/checkout@v4
- with:
- ref: master
- fetch-depth: 0
- persist-credentials: false
- - uses: actions/setup-node@v4
- with:
- node-version: 20
- cache: yarn
- - run: yarn --frozen-lockfile
- - name: Reconcile every open master PR with managed Japanese files
- env:
- GH_TOKEN: ${{ github.token }}
- GITHUB_REPOSITORY: ${{ github.repository }}
- run: node scripts/i18n/workflow-prs.cjs
"$RUNNER_TEMP/japanese-translation"
- - name: Plan all current and historical pending documents
- id: scan
- env:
- I18N_MODE: ${{ inputs.mode || 'generate' }}
- I18N_PLUGIN: ${{ inputs.plugin }}
- I18N_VERSION: ${{ inputs.version }}
- I18N_PATH: ${{ inputs.path }}
- I18N_LIMIT: '20'
- run: node scripts/i18n/workflow-scan.cjs
"$RUNNER_TEMP/japanese-translation"
- - uses: actions/upload-artifact@v4
- if: always()
- with:
- name: japanese-translation-plan-${{ github.run_id }}
- path: ${{ runner.temp }}/japanese-translation/
- if-no-files-found: warn
- retention-days: 30
-
- generate:
- needs: discover
- env:
- I18N_LOCALE: ${{ inputs.locale || 'ja' }}
- if: needs.discover.outputs.can_generate == 'true'
- runs-on: ubuntu-latest
- timeout-minutes: 120
- steps:
- - uses: actions/checkout@v4
- with:
- ref: ${{ needs.discover.outputs.source_commit }}
- fetch-depth: 0
- persist-credentials: false
- - uses: actions/setup-node@v4
- with:
- node-version: 20
- cache: yarn
- - uses: actions/setup-python@v4
- with:
- python-version: '3.9'
- - run: yarn --frozen-lockfile
- - uses: actions/download-artifact@v4
- with:
- name: japanese-translation-plan-${{ github.run_id }}
- path: ${{ runner.temp }}/japanese-translation
- - name: Check external configuration
- id: config
- env:
- OSS_AK: ${{ secrets.OSS_AK }}
- OSS_SK: ${{ secrets.OSS_SK }}
- APP_ID: ${{ secrets.I18N_GITHUB_APP_ID }}
- APP_PRIVATE_KEY: ${{ secrets.I18N_GITHUB_APP_PRIVATE_KEY }}
- I18N_GITHUB_PAT: ${{ secrets.I18N_GITHUB_PAT }}
- I18N_MODE: ${{ inputs.mode || 'generate' }}
- run: |
- if [[ -n "$APP_ID" && -n "$APP_PRIVATE_KEY" ]]; then
- echo 'auth_kind=app' >> "$GITHUB_OUTPUT"
- elif [[ -n "$I18N_GITHUB_PAT" ]]; then
- echo 'auth_kind=pat' >> "$GITHUB_OUTPUT"
- else
- echo 'auth_kind=none' >> "$GITHUB_OUTPUT"
- fi
- if [[ -n "$OSS_AK" && -n "$OSS_SK" && ( -n "$APP_ID" && -n
"$APP_PRIVATE_KEY" || -n "$I18N_GITHUB_PAT" || "$I18N_MODE" == 'validate' ) ]];
then
- echo 'ready=true' >> "$GITHUB_OUTPUT"
- else
- echo 'ready=false' >> "$GITHUB_OUTPUT"
- echo 'OSS Codex auth or a GitHub write credential is missing. No
model call or PR write was attempted.' >> "$GITHUB_STEP_SUMMARY"
- fi
- - name: Verify personal token identity before model calls
- if: steps.config.outputs.auth_kind == 'pat'
- env:
- GH_TOKEN: ${{ secrets.I18N_GITHUB_PAT }}
- run: gh api user --jq '.login' > /dev/null
- - name: Reconcile again immediately before model calls
- if: steps.config.outputs.ready == 'true'
- env:
- GH_TOKEN: ${{ github.token }}
- GITHUB_REPOSITORY: ${{ github.repository }}
- run: node scripts/i18n/workflow-prs.cjs
"$RUNNER_TEMP/japanese-translation"
- - name: Replan exact source and open PRs
- if: steps.config.outputs.ready == 'true'
- id: fresh
- env:
- I18N_MODE: ${{ inputs.mode || 'generate' }}
- I18N_PLUGIN: ${{ inputs.plugin }}
- I18N_VERSION: ${{ inputs.version }}
- I18N_PATH: ${{ inputs.path }}
- I18N_LIMIT: '20'
- run: node scripts/i18n/workflow-scan.cjs
"$RUNNER_TEMP/japanese-translation"
- - name: Install ossutil for the pinned Codex package and auth pool
- if: steps.fresh.outputs.can_generate == 'true'
- run: |
- temp_dir="$(mktemp -d)"
- trap 'rm -rf "$temp_dir"' EXIT
- curl -fsSL -o "$temp_dir/ossutil.zip"
https://gosspublic.alicdn.com/ossutil/1.7.19/ossutil-v1.7.19-linux-amd64.zip
- unzip -q "$temp_dir/ossutil.zip" -d "$temp_dir"
- sudo install -m 0755 "$temp_dir/ossutil-v1.7.19-linux-amd64/ossutil"
/usr/local/bin/ossutil
- - name: Install the verified Codex runtime used by Doris review
- if: steps.fresh.outputs.can_generate == 'true'
- env:
- OSS_AK: ${{ secrets.OSS_AK }}
- OSS_SK: ${{ secrets.OSS_SK }}
- run: |
- archive="$RUNNER_TEMP/codex-exec-goal.tar.gz"
- package="$RUNNER_TEMP/codex-exec-goal"
- ossutil -i "$OSS_AK" -k "$OSS_SK" -e oss-cn-hongkong.aliyuncs.com \
- cp -f
oss://doris-community-ci/codex/releases/codex-exec-goal-0.156.1-a310d33f7ebb-x86_64-unknown-linux-gnu.tar.gz
"$archive"
- echo
"2ee3073479bc8c94436684b55a13d7a3f299e768d04436d90e0427814a3954bc $archive" |
sha256sum --check
- mkdir -p "$package"
- tar -xzf "$archive" -C "$package"
- (cd "$package" && sha256sum --check SHA256SUMS)
- test "$(jq -r .version "$package/codex-package.json")" = 0.156.1
- "$package/bin/codex" exec --help | grep -q -- '--output-schema'
- "$package/bin/codex" exec --help | grep -q -- '--disable'
- "$package/bin/codex" exec --help | grep -q -- '--ignore-user-config'
- "$package/bin/codex" exec --help | grep -q -- '--ephemeral'
- echo "CODEX_BIN=$package/bin/codex" >> "$GITHUB_ENV"
- - name: Load an eligible Codex auth snapshot
- if: steps.fresh.outputs.can_generate == 'true'
- id: auth
- env:
- OSS_AK: ${{ secrets.OSS_AK }}
- OSS_SK: ${{ secrets.OSS_SK }}
- run: node scripts/i18n/codex-auth.cjs load
- - name: Generate a bounded, validated batch
- if: steps.auth.outcome == 'success'
- id: translation
- env:
- I18N_CODEX_MODEL: ${{ vars.I18N_CODEX_MODEL || 'gpt-6-sol' }}
- I18N_MODE: ${{ inputs.mode || 'generate' }}
- I18N_PATH: ${{ inputs.path }}
- I18N_MAX_PAGES: ${{ inputs.mode == 'validate' && '1' || '10' }}
- GITHUB_REPOSITORY: ${{ github.repository }}
- run: node scripts/i18n/workflow-generate.cjs
"$RUNNER_TEMP/japanese-translation"
- - uses: actions/create-github-app-token@v2
- if: steps.translation.outputs.created == 'true' &&
steps.config.outputs.auth_kind == 'app'
- id: app
- with:
- app-id: ${{ secrets.I18N_GITHUB_APP_ID }}
- private-key: ${{ secrets.I18N_GITHUB_APP_PRIVATE_KEY }}
- owner: apache
- repositories: doris-website
- permission-contents: write
- permission-pull-requests: write
- - name: Run shared PR admission checks and open reviewable PR
- if: steps.translation.outputs.created == 'true' &&
steps.config.outputs.auth_kind != 'none'
- env:
- GH_TOKEN: ${{ steps.app.outputs.token || secrets.I18N_GITHUB_PAT }}
- I18N_PR_AUTH_KIND: ${{ steps.config.outputs.auth_kind }}
- run: node scripts/i18n/workflow-publish.cjs
"$RUNNER_TEMP/japanese-translation"
- - name: Report staged validation without GitHub write credentials
- if: steps.translation.outputs.created == 'true' &&
steps.config.outputs.auth_kind == 'none'
- run: echo 'One-page Codex candidate is in the generation artifact. A
GitHub App or personal token is required for a translation PR.' >>
"$GITHUB_STEP_SUMMARY"
- - name: Sync refreshed Codex auth without overwriting a newer snapshot
- if: always() && steps.auth.outcome == 'success'
- env:
- OSS_AK: ${{ secrets.OSS_AK }}
- OSS_SK: ${{ secrets.OSS_SK }}
- run: node scripts/i18n/codex-auth.cjs sync
- - uses: actions/upload-artifact@v4
- if: always()
- with:
- name: japanese-translation-generation-${{ github.run_id }}
- path: ${{ runner.temp }}/japanese-translation/
- if-no-files-found: warn
- retention-days: 30
+ translation:
+ uses: ./.github/workflows/translation-run.yml
+ with:
+ locale: ja
+ workspace: japanese-translation
+ # Scheduled runs have no inputs and generate.
+ mode: ${{ inputs.mode || 'generate' }}
+ plugin: ${{ inputs.plugin }}
+ version: ${{ inputs.version }}
+ path: ${{ inputs.path }}
+ paths: ${{ inputs.paths }}
+ secrets:
+ OSS_AK: ${{ secrets.OSS_AK }}
+ OSS_SK: ${{ secrets.OSS_SK }}
+ I18N_GITHUB_APP_ID: ${{ secrets.I18N_GITHUB_APP_ID }}
+ I18N_GITHUB_APP_PRIVATE_KEY: ${{ secrets.I18N_GITHUB_APP_PRIVATE_KEY }}
+ I18N_GITHUB_PAT: ${{ secrets.I18N_GITHUB_PAT }}
diff --git a/.github/workflows/korean-translation.yml
b/.github/workflows/korean-translation.yml
index 11487252b54..ac0e425fc9d 100644
--- a/.github/workflows/korean-translation.yml
+++ b/.github/workflows/korean-translation.yml
@@ -22,9 +22,7 @@ on:
description: Optional JSON array of exact English .md/.mdx files
(exclusive with path)
type: string
# Twenty minutes after the Japanese schedule; the shared lock below holds
- # this run while a Japanese run is still in progress. Scheduled runs have no
- # inputs, so the mode falls back to generate; manual runs keep the dry-run
- # default.
+ # this run while a Japanese run is still in progress.
schedule:
- cron: '43 4,16 * * *'
@@ -33,205 +31,29 @@ permissions:
pull-requests: read
concurrency:
- # One entire run across scheduled/manual triggers, locales and modes.
- # Keep this stable so an in-flight run also holds the same lock.
- group: japanese-translation-ja
+ # One entire run at a time across both translation workflows, every trigger
+ # and every mode. Rename it only while no translation run is queued or
+ # running; otherwise the run in flight and the next one hold different locks.
+ group: docs-translation
cancel-in-progress: false
queue: max
jobs:
- discover:
- if: github.ref == 'refs/heads/master'
- runs-on: ubuntu-latest
- timeout-minutes: 20
- env:
- I18N_LOCALE: ko
- outputs:
- can_generate: ${{ steps.scan.outputs.can_generate }}
- source_commit: ${{ steps.scan.outputs.source_commit }}
- steps:
- - uses: actions/checkout@v4
- with:
- ref: master
- fetch-depth: 0
- persist-credentials: false
- - uses: actions/setup-node@v4
- with:
- node-version: 20
- cache: yarn
- - run: yarn --frozen-lockfile
- - name: Reconcile every open master PR with managed Korean files
- env:
- GH_TOKEN: ${{ github.token }}
- GITHUB_REPOSITORY: ${{ github.repository }}
- run: node scripts/i18n/workflow-prs.cjs
"$RUNNER_TEMP/korean-translation"
- - name: Plan all current and historical pending documents
- id: scan
- env:
- I18N_MODE: ${{ inputs.mode || 'generate' }}
- I18N_PLUGIN: ${{ inputs.plugin }}
- I18N_VERSION: ${{ inputs.version }}
- I18N_PATH: ${{ inputs.path }}
- I18N_PATHS: ${{ inputs.paths }}
- I18N_LIMIT: '20'
- run: node scripts/i18n/workflow-scan.cjs
"$RUNNER_TEMP/korean-translation"
- - uses: actions/upload-artifact@v4
- if: always()
- with:
- name: korean-translation-plan-${{ github.run_id }}
- path: ${{ runner.temp }}/korean-translation/
- if-no-files-found: warn
- retention-days: 30
-
- generate:
- needs: discover
- env:
- I18N_LOCALE: ko
- if: needs.discover.outputs.can_generate == 'true'
- runs-on: ubuntu-latest
- timeout-minutes: 120
- steps:
- - uses: actions/checkout@v4
- with:
- ref: ${{ needs.discover.outputs.source_commit }}
- fetch-depth: 0
- persist-credentials: false
- - uses: actions/setup-node@v4
- with:
- node-version: 20
- cache: yarn
- - uses: actions/setup-python@v4
- with:
- python-version: '3.9'
- - run: yarn --frozen-lockfile
- - uses: actions/download-artifact@v4
- with:
- name: korean-translation-plan-${{ github.run_id }}
- path: ${{ runner.temp }}/korean-translation
- - name: Check external configuration
- id: config
- env:
- OSS_AK: ${{ secrets.OSS_AK }}
- OSS_SK: ${{ secrets.OSS_SK }}
- APP_ID: ${{ secrets.I18N_GITHUB_APP_ID }}
- APP_PRIVATE_KEY: ${{ secrets.I18N_GITHUB_APP_PRIVATE_KEY }}
- I18N_GITHUB_PAT: ${{ secrets.I18N_GITHUB_PAT }}
- I18N_MODE: ${{ inputs.mode || 'generate' }}
- run: |
- if [[ -n "$APP_ID" && -n "$APP_PRIVATE_KEY" ]]; then
- echo 'auth_kind=app' >> "$GITHUB_OUTPUT"
- elif [[ -n "$I18N_GITHUB_PAT" ]]; then
- echo 'auth_kind=pat' >> "$GITHUB_OUTPUT"
- else
- echo 'auth_kind=none' >> "$GITHUB_OUTPUT"
- fi
- if [[ -n "$OSS_AK" && -n "$OSS_SK" && ( -n "$APP_ID" && -n
"$APP_PRIVATE_KEY" || -n "$I18N_GITHUB_PAT" || "$I18N_MODE" == 'validate' ) ]];
then
- echo 'ready=true' >> "$GITHUB_OUTPUT"
- else
- echo 'ready=false' >> "$GITHUB_OUTPUT"
- echo 'OSS Codex auth or a GitHub write credential is missing. No
model call or PR write was attempted.' >> "$GITHUB_STEP_SUMMARY"
- fi
- - name: Verify personal token identity before model calls
- if: steps.config.outputs.auth_kind == 'pat'
- env:
- GH_TOKEN: ${{ secrets.I18N_GITHUB_PAT }}
- run: gh api user --jq '.login' > /dev/null
- - name: Reconcile again immediately before model calls
- if: steps.config.outputs.ready == 'true'
- env:
- GH_TOKEN: ${{ github.token }}
- GITHUB_REPOSITORY: ${{ github.repository }}
- run: node scripts/i18n/workflow-prs.cjs
"$RUNNER_TEMP/korean-translation"
- - name: Replan exact source and open PRs
- if: steps.config.outputs.ready == 'true'
- id: fresh
- env:
- I18N_MODE: ${{ inputs.mode || 'generate' }}
- I18N_PLUGIN: ${{ inputs.plugin }}
- I18N_VERSION: ${{ inputs.version }}
- I18N_PATH: ${{ inputs.path }}
- I18N_PATHS: ${{ inputs.paths }}
- I18N_LIMIT: '20'
- run: node scripts/i18n/workflow-scan.cjs
"$RUNNER_TEMP/korean-translation"
- - name: Verify the Korean PR label before model calls
- if: steps.fresh.outputs.can_generate == 'true'
- env:
- GH_TOKEN: ${{ github.token }}
- run: test "$(gh api repos/apache/doris-website/labels/translate-ko
--jq .name)" = translate-ko
- - name: Install ossutil for the pinned Codex package and auth pool
- if: steps.fresh.outputs.can_generate == 'true'
- run: |
- temp_dir="$(mktemp -d)"
- trap 'rm -rf "$temp_dir"' EXIT
- curl -fsSL -o "$temp_dir/ossutil.zip"
https://gosspublic.alicdn.com/ossutil/1.7.19/ossutil-v1.7.19-linux-amd64.zip
- unzip -q "$temp_dir/ossutil.zip" -d "$temp_dir"
- sudo install -m 0755 "$temp_dir/ossutil-v1.7.19-linux-amd64/ossutil"
/usr/local/bin/ossutil
- - name: Install the verified Codex runtime used by Doris review
- if: steps.fresh.outputs.can_generate == 'true'
- env:
- OSS_AK: ${{ secrets.OSS_AK }}
- OSS_SK: ${{ secrets.OSS_SK }}
- run: |
- archive="$RUNNER_TEMP/codex-exec-goal.tar.gz"
- package="$RUNNER_TEMP/codex-exec-goal"
- ossutil -i "$OSS_AK" -k "$OSS_SK" -e oss-cn-hongkong.aliyuncs.com \
- cp -f
oss://doris-community-ci/codex/releases/codex-exec-goal-0.156.1-a310d33f7ebb-x86_64-unknown-linux-gnu.tar.gz
"$archive"
- echo
"2ee3073479bc8c94436684b55a13d7a3f299e768d04436d90e0427814a3954bc $archive" |
sha256sum --check
- mkdir -p "$package"
- tar -xzf "$archive" -C "$package"
- (cd "$package" && sha256sum --check SHA256SUMS)
- test "$(jq -r .version "$package/codex-package.json")" = 0.156.1
- "$package/bin/codex" exec --help | grep -q -- '--output-schema'
- "$package/bin/codex" exec --help | grep -q -- '--disable'
- "$package/bin/codex" exec --help | grep -q -- '--ignore-user-config'
- "$package/bin/codex" exec --help | grep -q -- '--ephemeral'
- echo "CODEX_BIN=$package/bin/codex" >> "$GITHUB_ENV"
- - name: Load an eligible Codex auth snapshot
- if: steps.fresh.outputs.can_generate == 'true'
- id: auth
- env:
- OSS_AK: ${{ secrets.OSS_AK }}
- OSS_SK: ${{ secrets.OSS_SK }}
- run: node scripts/i18n/codex-auth.cjs load
- - name: Generate a bounded, validated batch
- if: steps.auth.outcome == 'success'
- id: translation
- env:
- I18N_CODEX_MODEL: ${{ vars.I18N_CODEX_MODEL || 'gpt-6-sol' }}
- I18N_MODE: ${{ inputs.mode || 'generate' }}
- I18N_PATH: ${{ inputs.path }}
- I18N_MAX_PAGES: ${{ inputs.mode == 'validate' && '1' || '10' }}
- GITHUB_REPOSITORY: ${{ github.repository }}
- run: node scripts/i18n/workflow-generate.cjs
"$RUNNER_TEMP/korean-translation"
- - uses: actions/create-github-app-token@v2
- if: steps.translation.outputs.created == 'true' &&
steps.config.outputs.auth_kind == 'app'
- id: app
- with:
- app-id: ${{ secrets.I18N_GITHUB_APP_ID }}
- private-key: ${{ secrets.I18N_GITHUB_APP_PRIVATE_KEY }}
- owner: apache
- repositories: doris-website
- permission-contents: write
- permission-pull-requests: write
- - name: Run shared PR admission checks and open reviewable PR
- if: steps.translation.outputs.created == 'true' &&
steps.config.outputs.auth_kind != 'none'
- env:
- GH_TOKEN: ${{ steps.app.outputs.token || secrets.I18N_GITHUB_PAT }}
- I18N_PR_AUTH_KIND: ${{ steps.config.outputs.auth_kind }}
- run: node scripts/i18n/workflow-publish.cjs
"$RUNNER_TEMP/korean-translation"
- - name: Report staged validation without GitHub write credentials
- if: steps.translation.outputs.created == 'true' &&
steps.config.outputs.auth_kind == 'none'
- run: echo 'One-page Codex candidate is in the generation artifact. A
GitHub App or personal token is required for a translation PR.' >>
"$GITHUB_STEP_SUMMARY"
- - name: Sync refreshed Codex auth without overwriting a newer snapshot
- if: always() && steps.auth.outcome == 'success'
- env:
- OSS_AK: ${{ secrets.OSS_AK }}
- OSS_SK: ${{ secrets.OSS_SK }}
- run: node scripts/i18n/codex-auth.cjs sync
- - uses: actions/upload-artifact@v4
- if: always()
- with:
- name: korean-translation-generation-${{ github.run_id }}
- path: ${{ runner.temp }}/korean-translation/
- if-no-files-found: warn
- retention-days: 30
+ translation:
+ uses: ./.github/workflows/translation-run.yml
+ with:
+ locale: ko
+ workspace: korean-translation
+ # Scheduled runs have no inputs and generate; manual runs keep the
+ # dry-run default.
+ mode: ${{ inputs.mode || 'generate' }}
+ plugin: ${{ inputs.plugin }}
+ version: ${{ inputs.version }}
+ path: ${{ inputs.path }}
+ paths: ${{ inputs.paths }}
+ secrets:
+ OSS_AK: ${{ secrets.OSS_AK }}
+ OSS_SK: ${{ secrets.OSS_SK }}
+ I18N_GITHUB_APP_ID: ${{ secrets.I18N_GITHUB_APP_ID }}
+ I18N_GITHUB_APP_PRIVATE_KEY: ${{ secrets.I18N_GITHUB_APP_PRIVATE_KEY }}
+ I18N_GITHUB_PAT: ${{ secrets.I18N_GITHUB_PAT }}
diff --git a/.github/workflows/korean-translation.yml
b/.github/workflows/translation-run.yml
similarity index 74%
copy from .github/workflows/korean-translation.yml
copy to .github/workflows/translation-run.yml
index 11487252b54..1bd26b79530 100644
--- a/.github/workflows/korean-translation.yml
+++ b/.github/workflows/translation-run.yml
@@ -1,51 +1,71 @@
-name: Korean translation discovery
+name: Translation run
+# One discovery and generation run for one managed locale. The language
+# workflows own the triggers, the manual defaults and the shared whole-run
+# lock; they pass the locale, the mode and the filters here.
on:
- workflow_dispatch:
+ workflow_call:
inputs:
+ locale:
+ description: Managed translation locale
+ type: string
+ required: true
+ workspace:
+ description: Run directory name and artifact prefix
+ type: string
+ required: true
mode:
- description: Read-only dry run, one-page live validation, or batch
generation
- type: choice
- options: [dry-run, validate, generate]
- default: dry-run
+ description: dry-run, validate or generate
+ type: string
required: true
plugin:
description: Optional plugin filter (default or community)
type: string
+ default: ''
version:
description: Optional version filter (current or 4.x)
type: string
+ default: ''
path:
description: Optional repository-relative source path prefix
type: string
+ default: ''
paths:
description: Optional JSON array of exact English .md/.mdx files
(exclusive with path)
type: string
- # Twenty minutes after the Japanese schedule; the shared lock below holds
- # this run while a Japanese run is still in progress. Scheduled runs have no
- # inputs, so the mode falls back to generate; manual runs keep the dry-run
- # default.
- schedule:
- - cron: '43 4,16 * * *'
+ default: ''
+ secrets:
+ OSS_AK:
+ required: false
+ OSS_SK:
+ required: false
+ I18N_GITHUB_APP_ID:
+ required: false
+ I18N_GITHUB_APP_PRIVATE_KEY:
+ required: false
+ I18N_GITHUB_PAT:
+ required: false
permissions:
contents: read
pull-requests: read
-concurrency:
- # One entire run across scheduled/manual triggers, locales and modes.
- # Keep this stable so an in-flight run also holds the same lock.
- group: japanese-translation-ja
- cancel-in-progress: false
- queue: max
+env:
+ I18N_LOCALE: ${{ inputs.locale }}
+ I18N_MODE: ${{ inputs.mode }}
+ I18N_PLUGIN: ${{ inputs.plugin }}
+ I18N_VERSION: ${{ inputs.version }}
+ I18N_PATH: ${{ inputs.path }}
+ I18N_PATHS: ${{ inputs.paths }}
+ I18N_LIMIT: '20'
+ I18N_MAX_PAGES: ${{ inputs.mode == 'validate' && '1' || '10' }}
+ TRANSLATION_WORKSPACE: ${{ inputs.workspace }}
jobs:
discover:
if: github.ref == 'refs/heads/master'
runs-on: ubuntu-latest
timeout-minutes: 20
- env:
- I18N_LOCALE: ko
outputs:
can_generate: ${{ steps.scan.outputs.can_generate }}
source_commit: ${{ steps.scan.outputs.source_commit }}
@@ -60,40 +80,33 @@ jobs:
node-version: 20
cache: yarn
- run: yarn --frozen-lockfile
- - name: Reconcile every open master PR with managed Korean files
+ - name: Reconcile every open master PR with managed translation files
env:
GH_TOKEN: ${{ github.token }}
GITHUB_REPOSITORY: ${{ github.repository }}
- run: node scripts/i18n/workflow-prs.cjs
"$RUNNER_TEMP/korean-translation"
+ run: node scripts/i18n/workflow-prs.cjs
"$RUNNER_TEMP/$TRANSLATION_WORKSPACE"
- name: Plan all current and historical pending documents
id: scan
- env:
- I18N_MODE: ${{ inputs.mode || 'generate' }}
- I18N_PLUGIN: ${{ inputs.plugin }}
- I18N_VERSION: ${{ inputs.version }}
- I18N_PATH: ${{ inputs.path }}
- I18N_PATHS: ${{ inputs.paths }}
- I18N_LIMIT: '20'
- run: node scripts/i18n/workflow-scan.cjs
"$RUNNER_TEMP/korean-translation"
+ run: node scripts/i18n/workflow-scan.cjs
"$RUNNER_TEMP/$TRANSLATION_WORKSPACE"
- uses: actions/upload-artifact@v4
if: always()
with:
- name: korean-translation-plan-${{ github.run_id }}
- path: ${{ runner.temp }}/korean-translation/
+ name: ${{ inputs.workspace }}-plan-${{ github.run_id }}
+ path: ${{ runner.temp }}/${{ inputs.workspace }}/
if-no-files-found: warn
retention-days: 30
generate:
needs: discover
- env:
- I18N_LOCALE: ko
if: needs.discover.outputs.can_generate == 'true'
runs-on: ubuntu-latest
timeout-minutes: 120
steps:
+ # Current master, not the discovery commit: this job can wait for a
runner
+ # long after discovery, and the replan below works from what it checks
out.
- uses: actions/checkout@v4
with:
- ref: ${{ needs.discover.outputs.source_commit }}
+ ref: master
fetch-depth: 0
persist-credentials: false
- uses: actions/setup-node@v4
@@ -106,8 +119,8 @@ jobs:
- run: yarn --frozen-lockfile
- uses: actions/download-artifact@v4
with:
- name: korean-translation-plan-${{ github.run_id }}
- path: ${{ runner.temp }}/korean-translation
+ name: ${{ inputs.workspace }}-plan-${{ github.run_id }}
+ path: ${{ runner.temp }}/${{ inputs.workspace }}
- name: Check external configuration
id: config
env:
@@ -116,7 +129,6 @@ jobs:
APP_ID: ${{ secrets.I18N_GITHUB_APP_ID }}
APP_PRIVATE_KEY: ${{ secrets.I18N_GITHUB_APP_PRIVATE_KEY }}
I18N_GITHUB_PAT: ${{ secrets.I18N_GITHUB_PAT }}
- I18N_MODE: ${{ inputs.mode || 'generate' }}
run: |
if [[ -n "$APP_ID" && -n "$APP_PRIVATE_KEY" ]]; then
echo 'auth_kind=app' >> "$GITHUB_OUTPUT"
@@ -141,23 +153,17 @@ jobs:
env:
GH_TOKEN: ${{ github.token }}
GITHUB_REPOSITORY: ${{ github.repository }}
- run: node scripts/i18n/workflow-prs.cjs
"$RUNNER_TEMP/korean-translation"
+ run: node scripts/i18n/workflow-prs.cjs
"$RUNNER_TEMP/$TRANSLATION_WORKSPACE"
- name: Replan exact source and open PRs
if: steps.config.outputs.ready == 'true'
id: fresh
- env:
- I18N_MODE: ${{ inputs.mode || 'generate' }}
- I18N_PLUGIN: ${{ inputs.plugin }}
- I18N_VERSION: ${{ inputs.version }}
- I18N_PATH: ${{ inputs.path }}
- I18N_PATHS: ${{ inputs.paths }}
- I18N_LIMIT: '20'
- run: node scripts/i18n/workflow-scan.cjs
"$RUNNER_TEMP/korean-translation"
- - name: Verify the Korean PR label before model calls
+ run: node scripts/i18n/workflow-scan.cjs
"$RUNNER_TEMP/$TRANSLATION_WORKSPACE"
+ # The publisher labels the PR only after the model calls and the branch
push.
+ - name: Verify the PR label before model calls
if: steps.fresh.outputs.can_generate == 'true'
env:
GH_TOKEN: ${{ github.token }}
- run: test "$(gh api repos/apache/doris-website/labels/translate-ko
--jq .name)" = translate-ko
+ run: test "$(gh api
"repos/apache/doris-website/labels/translate-$I18N_LOCALE" --jq .name)" =
"translate-$I18N_LOCALE"
- name: Install ossutil for the pinned Codex package and auth pool
if: steps.fresh.outputs.can_generate == 'true'
run: |
@@ -198,11 +204,8 @@ jobs:
id: translation
env:
I18N_CODEX_MODEL: ${{ vars.I18N_CODEX_MODEL || 'gpt-6-sol' }}
- I18N_MODE: ${{ inputs.mode || 'generate' }}
- I18N_PATH: ${{ inputs.path }}
- I18N_MAX_PAGES: ${{ inputs.mode == 'validate' && '1' || '10' }}
GITHUB_REPOSITORY: ${{ github.repository }}
- run: node scripts/i18n/workflow-generate.cjs
"$RUNNER_TEMP/korean-translation"
+ run: node scripts/i18n/workflow-generate.cjs
"$RUNNER_TEMP/$TRANSLATION_WORKSPACE"
- uses: actions/create-github-app-token@v2
if: steps.translation.outputs.created == 'true' &&
steps.config.outputs.auth_kind == 'app'
id: app
@@ -218,7 +221,7 @@ jobs:
env:
GH_TOKEN: ${{ steps.app.outputs.token || secrets.I18N_GITHUB_PAT }}
I18N_PR_AUTH_KIND: ${{ steps.config.outputs.auth_kind }}
- run: node scripts/i18n/workflow-publish.cjs
"$RUNNER_TEMP/korean-translation"
+ run: node scripts/i18n/workflow-publish.cjs
"$RUNNER_TEMP/$TRANSLATION_WORKSPACE"
- name: Report staged validation without GitHub write credentials
if: steps.translation.outputs.created == 'true' &&
steps.config.outputs.auth_kind == 'none'
run: echo 'One-page Codex candidate is in the generation artifact. A
GitHub App or personal token is required for a translation PR.' >>
"$GITHUB_STEP_SUMMARY"
@@ -231,7 +234,7 @@ jobs:
- uses: actions/upload-artifact@v4
if: always()
with:
- name: korean-translation-generation-${{ github.run_id }}
- path: ${{ runner.temp }}/korean-translation/
+ name: ${{ inputs.workspace }}-generation-${{ github.run_id }}
+ path: ${{ runner.temp }}/${{ inputs.workspace }}/
if-no-files-found: warn
retention-days: 30
diff --git a/i18n/ko/docusaurus-plugin-content-docs/current.json
b/i18n/ko/docusaurus-plugin-content-docs/current.json
index 1a28c733e6b..7af91ea644d 100644
--- a/i18n/ko/docusaurus-plugin-content-docs/current.json
+++ b/i18n/ko/docusaurus-plugin-content-docs/current.json
@@ -255,6 +255,10 @@
"message": "데이터 Catalog",
"description": "The label for category Data Catalogs in sidebar docs"
},
+ "sidebar.docs.category.Hive Catalog": {
+ "message": "Hive Catalog",
+ "description": "The label for category Hive Catalog in sidebar docs"
+ },
"sidebar.docs.category.Iceberg Catalog": {
"message": "Iceberg Catalog",
"description": "The label for category Iceberg Catalog in sidebar docs"
@@ -263,6 +267,10 @@
"message": "Paimon Catalog",
"description": "The label for category Paimon Catalog in sidebar docs"
},
+ "sidebar.docs.category.Lance Catalog": {
+ "message": "Lance Catalog",
+ "description": "The label for category Lance Catalog in sidebar docs"
+ },
"sidebar.docs.category.Hudi Catalog": {
"message": "Hudi Catalog",
"description": "The label for category Hudi Catalog in sidebar docs"
diff --git a/i18n/ko/docusaurus-plugin-content-docs/version-4.x.json
b/i18n/ko/docusaurus-plugin-content-docs/version-4.x.json
index 01b0fe359a8..b7e13d16787 100644
--- a/i18n/ko/docusaurus-plugin-content-docs/version-4.x.json
+++ b/i18n/ko/docusaurus-plugin-content-docs/version-4.x.json
@@ -247,10 +247,18 @@
"message": "데이터 Catalog",
"description": "The label for category Data Catalogs in sidebar docs"
},
+ "sidebar.docs.category.Hive Catalog": {
+ "message": "Hive Catalog",
+ "description": "The label for category Hive Catalog in sidebar docs"
+ },
"sidebar.docs.category.Iceberg Catalog": {
"message": "Iceberg Catalog",
"description": "The label for category Iceberg Catalog in sidebar docs"
},
+ "sidebar.docs.category.Lance Catalog": {
+ "message": "Lance Catalog",
+ "description": "The label for category Lance Catalog in sidebar docs"
+ },
"sidebar.docs.category.Paimon Catalog": {
"message": "Paimon Catalog",
"description": "The label for category Paimon Catalog in sidebar docs"
diff --git a/scripts/i18n/__tests__/korean-workflow.test.cjs
b/scripts/i18n/__tests__/korean-workflow.test.cjs
deleted file mode 100644
index 6b83352a05e..00000000000
--- a/scripts/i18n/__tests__/korean-workflow.test.cjs
+++ /dev/null
@@ -1,102 +0,0 @@
-const test = require('node:test');
-const assert = require('node:assert/strict');
-const fs = require('node:fs');
-const path = require('node:path');
-const yaml = require('js-yaml');
-const os = require('node:os');
-const {spawnSync} = require('node:child_process');
-const {allowGeneration} = require('../workflow-generate.cjs');
-const workflows = path.resolve(__dirname, '../../../.github/workflows');
-const load = name => yaml.load(fs.readFileSync(path.join(workflows, name),
'utf8'));
-const korean = load('korean-translation.yml');
-const japanese = load('japanese-translation.yml');
-
-test('one-page validation rejects an exact batch list before creating scan
artifacts', t => {
- const directory = fs.mkdtempSync(path.join(os.tmpdir(),
'ko-validate-paths-'));
- t.after(() => fs.rmSync(directory, {recursive:true,force:true}));
- const output = path.join(directory,'scan');
- const result = spawnSync(process.execPath,
[path.resolve(workflows,'../../scripts/i18n/workflow-scan.cjs'),output], {
-
encoding:'utf8',env:{...process.env,I18N_LOCALE:'ko',I18N_MODE:'validate',I18N_PATHS:'["community/subscribe-mail-list.md"]'},
- });
- assert.equal(result.status,1);
- assert.match(result.stderr,/Validation mode requires path, not a list/);
- assert(!fs.existsSync(output));
-});
-
-test('Korean generation runs on its schedule or a manual choice and shares the
existing whole-run lock', () => {
- assert.deepEqual(Object.keys(korean.on), ['workflow_dispatch',
'schedule']);
- // Twice daily, offset from the Japanese runs that hold the same lock.
- assert.deepEqual(korean.on.schedule, [{cron: '43 4,16 * * *'}]);
- assert.deepEqual(japanese.on.schedule, [{cron: '23 4,16 * * *'}]);
- assert.equal(korean.on.workflow_dispatch.inputs.mode.default, 'dry-run');
- assert.deepEqual(korean.on.workflow_dispatch.inputs.mode.options,
['dry-run', 'validate', 'generate']);
- assert.equal(korean.on.workflow_dispatch.inputs.locale, undefined);
- assert.equal(korean.on.workflow_dispatch.inputs.paths.type, 'string');
- assert.deepEqual(korean.permissions, {'contents':'read',
'pull-requests':'read'});
- assert.deepEqual(korean.concurrency, japanese.concurrency);
- assert.equal(korean.concurrency.group, 'japanese-translation-ja');
- assert.equal(korean.jobs.discover.if, "github.ref == 'refs/heads/master'");
- assert.equal(korean.jobs.generate.needs, 'discover');
- assert.equal(korean.jobs.generate.if, "needs.discover.outputs.can_generate
== 'true'");
- for (const job of Object.values(korean.jobs)) {
- assert.equal(job.env.I18N_LOCALE, 'ko');
- for (const step of job.steps) {
- if (step.env?.I18N_MODE) assert.equal(step.env.I18N_MODE, "${{
inputs.mode || 'generate' }}");
- if (step.uses?.startsWith('actions/checkout@')) {
- assert.equal(step.with['fetch-depth'], 0);
- assert.equal(step.with['persist-credentials'], false);
- }
- }
- }
-});
-
-test('a scheduled run resolves to a mode that the generation gate accepts', ()
=> {
- // Discovery marks every scheduled run eligible, so the input-less fallback
- // must be a mode that workflow-generate.cjs will run, not a dry run.
- const step = korean.jobs.generate.steps.find(item => item.id ===
'translation');
- const fallback = /\|\| '([a-z-]+)' \}\}$/.exec(step.env.I18N_MODE)[1];
- assert.equal(allowGeneration({I18N_MODE: fallback, GITHUB_EVENT_NAME:
'schedule', I18N_MAX_PAGES: '10'}), true);
- assert.equal(allowGeneration({I18N_MODE:
korean.on.workflow_dispatch.inputs.mode.default,
- GITHUB_EVENT_NAME: 'workflow_dispatch', I18N_MAX_PAGES: '10'}), false);
-});
-
-test('Korean discovery and generation exchange Korean artifacts at the exact
scanned commit', () => {
- const discover = korean.jobs.discover.steps;
- const generate = korean.jobs.generate.steps;
- assert.equal(generate.find(step =>
step.uses?.startsWith('actions/checkout@')).with.ref,
- '${{ needs.discover.outputs.source_commit }}');
- const upload = discover.find(step =>
step.uses?.startsWith('actions/upload-artifact@'));
- const download = generate.find(step =>
step.uses?.startsWith('actions/download-artifact@'));
- assert.equal(upload.with.name, 'korean-translation-plan-${{ github.run_id
}}');
- assert.equal(download.with.name, upload.with.name);
- assert.equal(download.with.path, '${{ runner.temp }}/korean-translation');
- const scripts = [...discover, ...generate].filter(step => /node
scripts\/i18n\/workflow-(prs|scan|generate|publish)/.test(step.run ?? ''));
- assert.equal(scripts.length, 6);
- assert(scripts.every(step =>
step.run.includes('"$RUNNER_TEMP/korean-translation"')));
- const execute = generate.find(step => step.id === 'translation');
- assert.equal(execute.env.I18N_MAX_PAGES, "${{ inputs.mode == 'validate' &&
'1' || '10' }}");
- assert.equal(execute.env.I18N_PATH, '${{ inputs.path }}');
- for (const steps of [discover, generate]) {
- const scan = steps.find(step =>
step.run?.includes('workflow-scan.cjs'));
- assert.equal(scan.env.I18N_PATHS, '${{ inputs.paths }}');
- }
- assert.equal(execute.env.I18N_CODEX_MODEL,
japanese.jobs.generate.steps.find(step => step.id ===
'translation').env.I18N_CODEX_MODEL);
-});
-
-test('Korean publication checks the label before auth loading and preserves
the verified runtime and refresh path', () => {
- const steps = korean.jobs.generate.steps;
- const label = steps.findIndex(step => step.name === 'Verify the Korean PR
label before model calls');
- const auth = steps.findIndex(step => step.id === 'auth');
- assert(label >= 0 && label < auth);
- assert.equal(steps[label].if, "steps.fresh.outputs.can_generate ==
'true'");
- assert.equal(steps[label].env.GH_TOKEN, '${{ github.token }}');
- assert(steps[label].run.includes('labels/translate-ko'));
- const runtime = steps.find(step => step.name === 'Install the verified
Codex runtime used by Doris review');
- assert.deepEqual(runtime, japanese.jobs.generate.steps.find(step =>
step.name === runtime.name));
- const sync = steps.find(step => step.name === 'Sync refreshed Codex auth
without overwriting a newer snapshot');
- assert.equal(sync.if, "always() && steps.auth.outcome == 'success'");
- assert.equal(sync.run, 'node scripts/i18n/codex-auth.cjs sync');
- const publish = steps.find(step =>
step.run?.includes('workflow-publish.cjs'));
- assert.equal(publish.if, "steps.translation.outputs.created == 'true' &&
steps.config.outputs.auth_kind != 'none'");
- assert.equal(publish.env.GH_TOKEN, '${{ steps.app.outputs.token ||
secrets.I18N_GITHUB_PAT }}');
-});
diff --git a/scripts/i18n/__tests__/multi-locale.test.cjs
b/scripts/i18n/__tests__/multi-locale.test.cjs
index 9b1e4c9b0a0..f048a788780 100644
--- a/scripts/i18n/__tests__/multi-locale.test.cjs
+++ b/scripts/i18n/__tests__/multi-locale.test.cjs
@@ -109,8 +109,9 @@ test('native doc metadata, browser groups and per-locale
manifests agree, includ
test('scheduled Japanese jobs select ja and PR admission checks only changed
locales with fail-fast behavior', async t => {
const workflow = file => yaml.load(fs.readFileSync(path.join(repoRoot,
'.github/workflows', file), 'utf8'));
const translation = workflow('japanese-translation.yml');
- assert.equal(translation.on.workflow_dispatch.inputs.locale.default, 'ja');
- for (const job of ['discover', 'generate'])
assert.equal(translation.jobs[job].env.I18N_LOCALE, "${{ inputs.locale || 'ja'
}}");
+ assert.equal(translation.on.workflow_dispatch.inputs.locale, undefined);
+ assert.equal(translation.jobs.translation.with.locale, 'ja');
+ assert.equal(workflow('translation-run.yml').env.I18N_LOCALE, '${{
inputs.locale }}');
assert.equal(workflow('japanese-coverage-audit.yml').jobs.audit.env.I18N_LOCALE,
'ja');
const gate =
workflow('build-check.yml').jobs['build-and-deploy'].steps.find(step =>
step.name === 'Translation gate for changed locales');
const {root, write} = await createFixture(directory(t));
diff --git a/scripts/i18n/__tests__/translation-workflows.test.cjs
b/scripts/i18n/__tests__/translation-workflows.test.cjs
new file mode 100644
index 00000000000..c7ba427aad8
--- /dev/null
+++ b/scripts/i18n/__tests__/translation-workflows.test.cjs
@@ -0,0 +1,145 @@
+const test = require('node:test');
+const assert = require('node:assert/strict');
+const fs = require('node:fs');
+const path = require('node:path');
+const yaml = require('js-yaml');
+const os = require('node:os');
+const {spawnSync} = require('node:child_process');
+const {allowGeneration} = require('../workflow-generate.cjs');
+const workflows = path.resolve(__dirname, '../../../.github/workflows');
+const load = name => yaml.load(fs.readFileSync(path.join(workflows, name),
'utf8'));
+const shared = load('translation-run.yml');
+const callers = {ja: load('japanese-translation.yml'), ko:
load('korean-translation.yml')};
+
+test('one-page validation rejects an exact batch list before creating scan
artifacts', t => {
+ const directory = fs.mkdtempSync(path.join(os.tmpdir(),
'ko-validate-paths-'));
+ t.after(() => fs.rmSync(directory, {recursive:true,force:true}));
+ const output = path.join(directory,'scan');
+ const result = spawnSync(process.execPath,
[path.resolve(workflows,'../../scripts/i18n/workflow-scan.cjs'),output], {
+
encoding:'utf8',env:{...process.env,I18N_LOCALE:'ko',I18N_MODE:'validate',I18N_PATHS:'["community/subscribe-mail-list.md"]'},
+ });
+ assert.equal(result.status,1);
+ assert.match(result.stderr,/Validation mode requires path, not a list/);
+ assert(!fs.existsSync(output));
+});
+
+test('each language workflow passes one fixed locale to the shared run under
one lock', () => {
+ const expected = {
+ ja: {workspace: 'japanese-translation', mode: 'generate', cron: '23
4,16 * * *'},
+ // Twenty minutes later; the shared lock queues it behind a Japanese
run.
+ ko: {workspace: 'korean-translation', mode: 'dry-run', cron: '43 4,16
* * *'},
+ };
+ for (const [locale, workflow] of Object.entries(callers)) {
+ assert.deepEqual(Object.keys(workflow.on), ['workflow_dispatch',
'schedule']);
+ assert.deepEqual(workflow.on.schedule, [{cron:
expected[locale].cron}]);
+ const inputs = workflow.on.workflow_dispatch.inputs;
+ // No locale input: each workflow runs only its own language.
+ assert.deepEqual(Object.keys(inputs), ['mode', 'plugin', 'version',
'path', 'paths']);
+ assert.equal(inputs.mode.default, expected[locale].mode);
+ assert.deepEqual(inputs.mode.options, ['dry-run', 'validate',
'generate']);
+ assert.deepEqual(workflow.permissions, {'contents': 'read',
'pull-requests': 'read'});
+ assert.deepEqual(workflow.concurrency, {group: 'docs-translation',
'cancel-in-progress': false, queue: 'max'});
+ assert.deepEqual(Object.keys(workflow.jobs), ['translation']);
+ const job = workflow.jobs.translation;
+ assert.equal(job.uses, './.github/workflows/translation-run.yml');
+ assert.deepEqual(job.with, {locale, workspace:
expected[locale].workspace,
+ mode: "${{ inputs.mode || 'generate' }}", plugin: '${{
inputs.plugin }}', version: '${{ inputs.version }}',
+ path: '${{ inputs.path }}', paths: '${{ inputs.paths }}'});
+ assert.deepEqual(Object.keys(job.secrets).sort(),
Object.keys(shared.on.workflow_call.secrets).sort());
+ for (const [name, value] of Object.entries(job.secrets))
assert.equal(value, `\${{ secrets.${name} }}`);
+ }
+});
+
+test('a scheduled run resolves to a mode that the generation gate accepts', ()
=> {
+ // Discovery marks every scheduled run eligible, so the input-less fallback
+ // must be a mode that workflow-generate.cjs will run, not a dry run.
+ for (const workflow of Object.values(callers)) {
+ const fallback = /\|\| '([a-z-]+)'
\}\}$/.exec(workflow.jobs.translation.with.mode)[1];
+ assert.equal(allowGeneration({I18N_MODE: fallback, GITHUB_EVENT_NAME:
'schedule', I18N_MAX_PAGES: '10'}), true);
+ }
+ assert.equal(allowGeneration({I18N_MODE:
callers.ko.on.workflow_dispatch.inputs.mode.default,
+ GITHUB_EVENT_NAME: 'workflow_dispatch', I18N_MAX_PAGES: '10'}), false);
+});
+
+test('the shared run names no language and sets mode and filters once', () => {
+ const text = fs.readFileSync(path.join(workflows, 'translation-run.yml'),
'utf8');
+ assert.doesNotMatch(text, /japanese|korean|translate-(ja|ko)\b/i);
+ assert.deepEqual(Object.keys(shared.on), ['workflow_call']);
+ assert.deepEqual(shared.permissions, {'contents': 'read', 'pull-requests':
'read'});
+ assert.equal(shared.concurrency, undefined);
+ const inputs = shared.on.workflow_call.inputs;
+ for (const name of ['locale', 'workspace', 'mode'])
assert.equal(inputs[name].required, true);
+ for (const name of ['plugin', 'version', 'path', 'paths'])
assert.equal(inputs[name].default, '');
+ assert.equal(shared.env.I18N_LOCALE, '${{ inputs.locale }}');
+ assert.equal(shared.env.I18N_MODE, '${{ inputs.mode }}');
+ for (const name of ['plugin', 'version', 'path', 'paths'])
+ assert.equal(shared.env[`I18N_${name.toUpperCase()}`], `\${{
inputs.${name} }}`);
+ assert.equal(shared.env.I18N_LIMIT, '20');
+ assert.equal(shared.env.I18N_MAX_PAGES, "${{ inputs.mode == 'validate' &&
'1' || '10' }}");
+ assert.equal(shared.jobs.discover.if, "github.ref == 'refs/heads/master'");
+ assert.equal(shared.jobs.generate.needs, 'discover');
+ assert.equal(shared.jobs.generate.if, "needs.discover.outputs.can_generate
== 'true'");
+ const local = new Set(['I18N_CODEX_MODEL', 'I18N_GITHUB_PAT',
'I18N_PR_AUTH_KIND']);
+ for (const job of Object.values(shared.jobs)) {
+ assert.equal(job.env, undefined);
+ for (const step of job.steps) {
+ for (const key of Object.keys(step.env ?? {}))
assert(!key.startsWith('I18N_') || local.has(key), key);
+ if (step.uses?.startsWith('actions/checkout@')) {
+ // Generation recounts against current master instead of the
discovery commit.
+ assert.deepEqual(step.with, {ref: 'master', 'fetch-depth': 0,
'persist-credentials': false});
+ }
+ }
+ }
+});
+
+test('discovery and generation exchange artifacts in the caller workspace', ()
=> {
+ const discover = shared.jobs.discover.steps;
+ const generate = shared.jobs.generate.steps;
+ const upload = discover.find(step =>
step.uses?.startsWith('actions/upload-artifact@'));
+ const download = generate.find(step =>
step.uses?.startsWith('actions/download-artifact@'));
+ assert.equal(upload.with.name, '${{ inputs.workspace }}-plan-${{
github.run_id }}');
+ assert.equal(download.with.name, upload.with.name);
+ assert.equal(download.with.path, '${{ runner.temp }}/${{ inputs.workspace
}}');
+ assert.equal(generate.at(-1).with.name, '${{ inputs.workspace
}}-generation-${{ github.run_id }}');
+ assert.equal(shared.env.TRANSLATION_WORKSPACE, '${{ inputs.workspace }}');
+ const scripts = [...discover, ...generate].filter(step => /node
scripts\/i18n\/workflow-(prs|scan|generate|publish)/.test(step.run ?? ''));
+ assert.equal(scripts.length, 6);
+ assert(scripts.every(step => step.run.endsWith('
"$RUNNER_TEMP/$TRANSLATION_WORKSPACE"')));
+ const index = name => generate.findIndex(step => step.name === name);
+ assert(index('Reconcile again immediately before model calls') <
index('Replan exact source and open PRs'));
+ assert(index('Replan exact source and open PRs') < index('Verify the PR
label before model calls'));
+ assert.equal(generate.find(step => step.id ===
'translation').env.I18N_CODEX_MODEL,
+ "${{ vars.I18N_CODEX_MODEL || 'gpt-6-sol' }}");
+});
+
+test('the label check runs before auth loading and fails for a missing locale
label', t => {
+ const steps = shared.jobs.generate.steps;
+ const label = steps.findIndex(step => step.name === 'Verify the PR label
before model calls');
+ const auth = steps.findIndex(step => step.id === 'auth');
+ assert(label >= 0 && label < auth);
+ assert.equal(steps[label].if, "steps.fresh.outputs.can_generate ==
'true'");
+ assert.equal(steps[label].env.GH_TOKEN, '${{ github.token }}');
+ const bin = fs.mkdtempSync(path.join(os.tmpdir(), 'translation-label-'));
+ t.after(() => fs.rmSync(bin, {recursive: true, force: true}));
+ // The mock answers like the labels API: the name, or nothing when missing.
+ fs.writeFileSync(path.join(bin, 'gh'), '#!/bin/sh\ncase "$2" in
*/labels/translate-ja) echo translate-ja;; esac\n',
+ {mode: 0o755});
+ const run = locale => spawnSync('bash', ['-e', '-c', steps[label].run],
+ {env: {...process.env, PATH: `${bin}:${process.env.PATH}`,
I18N_LOCALE: locale}}).status;
+ assert.equal(run('ja'), 0);
+ assert.notEqual(run('ko'), 0);
+});
+
+test('publication preserves the verified runtime and refresh path', () => {
+ const steps = shared.jobs.generate.steps;
+ const runtime = steps.find(step => step.name === 'Install the verified
Codex runtime used by Doris review');
+
assert(runtime.run.includes('codex-exec-goal-0.156.1-a310d33f7ebb-x86_64-unknown-linux-gnu.tar.gz'));
+
assert(runtime.run.includes('2ee3073479bc8c94436684b55a13d7a3f299e768d04436d90e0427814a3954bc
$archive'));
+ assert.equal(runtime.if, "steps.fresh.outputs.can_generate == 'true'");
+ const sync = steps.find(step => step.name === 'Sync refreshed Codex auth
without overwriting a newer snapshot');
+ assert.equal(sync.if, "always() && steps.auth.outcome == 'success'");
+ assert.equal(sync.run, 'node scripts/i18n/codex-auth.cjs sync');
+ const publish = steps.find(step =>
step.run?.includes('workflow-publish.cjs'));
+ assert.equal(publish.if, "steps.translation.outputs.created == 'true' &&
steps.config.outputs.auth_kind != 'none'");
+ assert.equal(publish.env.GH_TOKEN, '${{ steps.app.outputs.token ||
secrets.I18N_GITHUB_PAT }}');
+});
diff --git a/scripts/i18n/__tests__/workflow.test.cjs
b/scripts/i18n/__tests__/workflow.test.cjs
index 5b69fff3fe9..1f6d7f895a5 100644
--- a/scripts/i18n/__tests__/workflow.test.cjs
+++ b/scripts/i18n/__tests__/workflow.test.cjs
@@ -8,9 +8,10 @@ const {hash, loadConfig, stateFileFor, loadRecords, identity}
= require('../stat
const {structure} = require('../structure.cjs');
const {readState, applyCandidates} = require('../translate-docs.cjs');
const {reconcileOpenPrs} = require('../workflow-prs.cjs');
-const {relevantPush} = require('../workflow-scan.cjs');
+const {relevantPush, planInputs} = require('../workflow-scan.cjs');
const {pairFor, machineOwned, generate, allowGeneration} =
require('../workflow-generate.cjs');
const {expectedFiles, authorFor} = require('../workflow-publish.cjs');
+const {masterDrift, checkMaster, batchWide, pageFiles} =
require('../master-drift.cjs');
// The language facts come from the real configuration, so the tests describe
// the same automation names the pipeline publishes under.
@@ -176,59 +177,84 @@ test('personal token writes commits under its
authenticated owner', () => {
assert.throws(() => authorFor('pat', {login: 'other name', id: 1},
language), /Invalid personal/);
});
-test('one model result creates separate Dev and 4.x targets and state
records', async () => {
+// One pending Dev page whose 4.x copy shares its English source.
+function pairFixture(t) {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'ja-workflow-'));
+ t.after(() => fs.rmSync(dir, {recursive: true, force: true}));
const git = (...args) => execFileSync('git', args, {cwd: dir, encoding:
'utf8'}).trim();
- try {
- const source = '# Hello\n\nA simple guide.\n';
- const relative = 'guide.md';
- const devSource = `docs/${relative}`;
- const x4Source = `versioned_docs/version-4.x/${relative}`;
- for (const file of [devSource, x4Source]) {
- fs.mkdirSync(path.dirname(path.join(dir, file)), {recursive:
true});
- fs.writeFileSync(path.join(dir, file), source);
- }
- fs.mkdirSync(path.join(dir, 'translations/scope'), {recursive: true});
- fs.mkdirSync(path.join(dir, 'translations/state/ja'), {recursive:
true});
- fs.writeFileSync(path.join(dir, 'translations/scope/ja.json'),
JSON.stringify(scope));
- fs.writeFileSync(path.join(dir, 'translations/state/ja/docs.json'),
'{"schemaVersion":1,"pages":[]}\n');
- git('init', '-q');
- git('config', 'user.name', 'Test');
- git('config', 'user.email', '[email protected]');
- git('add', '.');
- git('commit', '-qm', 'source');
- const sourceCommit = git('rev-parse', 'HEAD');
- const task = (id, version, sourcePath, targetPath) => ({id, groupId:
version === 'current' ? 'DEV' : '4X',
- pluginId: 'default', version, docId: 'guide', locale: 'ja', kind:
'translate_backlog',
- sourcePath, targetPath, sourceHash: hash(source), sourceCommit,
- requiresHumanReview: false, activePr: null});
- const dev = task('T06-DEV-000000000000001', 'current', devSource,
- 'i18n/ja/docusaurus-plugin-content-docs/current/guide.md');
- const x4 = task('T06-4X-000000000000002', '4.x', x4Source,
- 'i18n/ja/docusaurus-plugin-content-docs/version-4.x/guide.md');
- let calls = 0;
- const adapter = {model: 'fixture', usedTokens: 0, translate: async
({units}) => {
- calls++;
- return {translations: units.map(unit => ({id: unit.id,
- text: unit.text === 'Hello' ? 'こんにちは' : '簡単なガイドです。'}))};
- }};
- const manifest = await generate({rootDir: dir, target, output:
path.join(dir, 'output'),
- ensureFresh: commit => assert.equal(commit, sourceCommit),
- plan: {schemaVersion: 1, locale: 'ja', sourceCommit, selected:
[dev.id], tasks: [dev, x4], exceptions: []},
- reconciliation: {botPrs: [], botDetails: [], blockedTargets: []},
- maxPages: 2, adapter, glossary:
require('../../../translations/glossary/ja.json')});
- assert.equal(calls, 1);
- assert.equal(manifest.candidates.length, 2);
- assert.equal(manifest.provider, 'codex');
- assert.equal(manifest.candidates[1].reusedFrom, dev.id);
- assert.equal(fs.readFileSync(path.join(dir, dev.targetPath), 'utf8'),
- fs.readFileSync(path.join(dir, x4.targetPath), 'utf8'));
- assert.equal(fs.readFileSync(path.join(dir, 'output/candidates',
dev.targetPath), 'utf8'),
- fs.readFileSync(path.join(dir, dev.targetPath), 'utf8'));
- const state = JSON.parse(fs.readFileSync(path.join(dir,
'translations/state/ja/docs.json'), 'utf8'));
- assert.deepEqual(state.pages.map(page => page.version), ['4.x',
'current']);
- assert(state.pages.every(page => page.reviewStatus === 'machine' &&
page.sourceHash === hash(source)));
- } finally {fs.rmSync(dir, {recursive: true, force: true});}
+ const source = '# Hello\n\nA simple guide.\n';
+ const relative = 'guide.md';
+ const devSource = `docs/${relative}`;
+ const x4Source = `versioned_docs/version-4.x/${relative}`;
+ for (const file of [devSource, x4Source]) {
+ fs.mkdirSync(path.dirname(path.join(dir, file)), {recursive: true});
+ fs.writeFileSync(path.join(dir, file), source);
+ }
+ fs.mkdirSync(path.join(dir, 'translations/scope'), {recursive: true});
+ fs.mkdirSync(path.join(dir, 'translations/state/ja'), {recursive: true});
+ fs.writeFileSync(path.join(dir, 'translations/scope/ja.json'),
JSON.stringify(scope));
+ fs.writeFileSync(path.join(dir, 'translations/state/ja/docs.json'),
'{"schemaVersion":1,"pages":[]}\n');
+ git('init', '-q');
+ git('config', 'user.name', 'Test');
+ git('config', 'user.email', '[email protected]');
+ git('add', '.');
+ git('commit', '-qm', 'source');
+ const sourceCommit = git('rev-parse', 'HEAD');
+ const task = (id, version, sourcePath, targetPath) => ({id, groupId:
version === 'current' ? 'DEV' : '4X',
+ pluginId: 'default', version, docId: 'guide', locale: 'ja', kind:
'translate_backlog',
+ sourcePath, targetPath, sourceHash: hash(source), sourceCommit,
+ requiresHumanReview: false, activePr: null});
+ const dev = task('T06-DEV-000000000000001', 'current', devSource,
+ 'i18n/ja/docusaurus-plugin-content-docs/current/guide.md');
+ const x4 = task('T06-4X-000000000000002', '4.x', x4Source,
+ 'i18n/ja/docusaurus-plugin-content-docs/version-4.x/guide.md');
+ let calls = 0;
+ const adapter = {model: 'fixture', usedTokens: 0, translate: async
({units}) => {
+ calls++;
+ return {translations: units.map(unit => ({id: unit.id,
+ text: unit.text === 'Hello' ? 'こんにちは' : '簡単なガイドです。'}))};
+ }};
+ const run = ensureFresh => generate({rootDir: dir, target, output:
path.join(dir, 'output'), ensureFresh,
+ plan: {schemaVersion: 1, locale: 'ja', sourceCommit, selected:
[dev.id], tasks: [dev, x4], exceptions: []},
+ reconciliation: {botPrs: [], botDetails: [], blockedTargets: []},
+ maxPages: 2, adapter, glossary:
require('../../../translations/glossary/ja.json')});
+ return {dir, source, sourceCommit, dev, x4, run, calls: () => calls};
+}
+
+test('one model result creates separate Dev and 4.x targets and state
records', async t => {
+ const {dir, source, sourceCommit, dev, x4, run, calls} = pairFixture(t);
+ const manifest = await run(commit => assert.equal(commit, sourceCommit));
+ assert.equal(calls(), 1);
+ assert.equal(manifest.candidates.length, 2);
+ assert.equal(manifest.provider, 'codex');
+ assert.equal(manifest.candidates[1].reusedFrom, dev.id);
+ assert.equal(fs.readFileSync(path.join(dir, dev.targetPath), 'utf8'),
+ fs.readFileSync(path.join(dir, x4.targetPath), 'utf8'));
+ assert.equal(fs.readFileSync(path.join(dir, 'output/candidates',
dev.targetPath), 'utf8'),
+ fs.readFileSync(path.join(dir, dev.targetPath), 'utf8'));
+ const state = JSON.parse(fs.readFileSync(path.join(dir,
'translations/state/ja/docs.json'), 'utf8'));
+ assert.deepEqual(state.pages.map(page => page.version), ['4.x',
'current']);
+ assert(state.pages.every(page => page.reviewStatus === 'machine' &&
page.sourceHash === hash(source)));
+});
+
+test('a page changed on master after its model call is dropped together with
its Dev/4.x copy', async t => {
+ const {dir, dev, x4, run, calls} = pairFixture(t);
+ const seen = [];
+ // Before the batch, before the page, then after it: only the last check
sees the change.
+ const manifest = await run((commit, pages) => {
+ seen.push(pages);
+ if (seen.length === 3) return {head: 'b'.repeat(40), moved: true,
files: [dev.sourcePath],
+ batchWide: [], affected: [dev.id]};
+ });
+ assert.equal(calls(), 1);
+ assert.deepEqual([...seen[0].keys()].sort(), [dev.id, x4.id].sort());
+ assert.deepEqual(seen[0].get(x4.id), [x4.sourcePath, x4.targetPath,
stateFileFor(x4)]);
+ assert.deepEqual(manifest.candidates, []);
+ assert.deepEqual(manifest.failures.map(item => item.id), [dev.id, x4.id]);
+ assert.match(manifest.failures[0].error, /master changed this page/);
+ assert.deepEqual(manifest.master, {head: 'b'.repeat(40), moved: true,
changedFiles: 1});
+ for (const task of [dev, x4]) assert(!fs.existsSync(path.join(dir,
task.targetPath)));
+ assert(!fs.existsSync(path.join(dir, 'output/pr-body.md')));
});
async function updateFixture(t, {sharded = true, human = false,
@@ -513,7 +539,9 @@ test('publication deduplicates shared state and rejects an
unrelated page shard'
[record.targetPath, second.targetPath,
'translations/state/ja/docs.json'].sort());
});
-test('normal generation skips open PRs and blocked targets, then generates ten
other pages', async t => {
+// Fourteen pending pages; the first is in an open PR, the second is reserved
+// by another PR's candidate and the third is blocked.
+function pendingPagesFixture(t) {
const root = fs.mkdtempSync(path.join(os.tmpdir(),
'translation-ten-page-batch-'));
t.after(() => fs.rmSync(root, {recursive: true, force: true}));
const put = (file, text) => {
@@ -543,15 +571,21 @@ test('normal generation skips open PRs and blocked
targets, then generates ten o
calls++;
return {translations: units.map(unit => ({id: unit.id, text: '訳' +
unit.text}))};
}};
- const manifest = await generate({rootDir: root, target, output:
path.join(root, 'output'),
+ const run = ensureFresh => generate({rootDir: root, target, output:
path.join(root, 'output'),
plan: {schemaVersion: 1, locale: 'ja', sourceCommit, selected:
tasks.map(task => task.id), tasks, exceptions: []},
reconciliation: {botPrs: [21, 22], botDetails: [{number: 21,
untouched: false}, {number: 22, untouched: true}],
active: {candidates: [{...tasks[1], sourceHash: hash('older
source'), pr: '#22'}]},
- blockedTargets: [tasks[2].targetPath]}, ensureFresh: () => {},
+ blockedTargets: [tasks[2].targetPath]}, ensureFresh,
adapter, glossary: require('../../../translations/glossary/ja.json')});
+ return {root, tasks, run, calls: () => calls};
+}
+
+test('normal generation skips open PRs and blocked targets, then generates ten
other pages', async t => {
+ const {root, tasks, run, calls} = pendingPagesFixture(t);
+ const manifest = await run(() => {});
assert.equal(manifest.candidates.length, 10);
assert.equal(manifest.failures.length, 0);
- assert.equal(calls, 10);
+ assert.equal(calls(), 10);
assert.equal(loadRecords(root, 'ja').size, 10);
assert.deepEqual(manifest.candidates.map(item => item.id), tasks.slice(3,
13).map(item => item.id));
for (const task of [tasks[0], tasks[1], tasks[2], tasks[13]]) {
@@ -559,4 +593,155 @@ test('normal generation skips open PRs and blocked
targets, then generates ten o
assert(!fs.existsSync(path.join(root, stateFileFor(task))));
}
assert.equal(manifest.replacesPr, undefined);
+ assert.equal(manifest.master, undefined);
+});
+
+test('a page master changed before its turn is skipped without a model call
and the batch continues', async t => {
+ const {root, tasks, run, calls} = pendingPagesFixture(t);
+ let checks = 0;
+ const drift = {head: 'a'.repeat(40), moved: true, files:
[tasks[4].sourcePath, 'blog/post.md'],
+ batchWide: [], affected: [tasks[4].id]};
+ // One check before the batch, one before each page that reaches the model
and
+ // one after the last; the check before page 4 is the first to see the
change.
+ const manifest = await run(() => ++checks >= 3 ? drift : undefined);
+ assert.equal(calls(), 10);
+ assert.equal(checks, 13);
+ assert.deepEqual(manifest.candidates.map(item => item.id), [tasks[3],
...tasks.slice(5, 14)].map(item => item.id));
+ assert.deepEqual(manifest.failures.map(item => item.id), [tasks[4].id]);
+ assert(!fs.existsSync(path.join(root, tasks[4].targetPath)));
+ assert.deepEqual(manifest.master, {head: drift.head, moved: true,
changedFiles: 2});
+ assert.match(fs.readFileSync(path.join(root, 'output/pr-body.md'), 'utf8'),
+ new RegExp(`Master moved to \`${drift.head}\` during generation; none
of its 2 changed files affect this batch\\.`));
+});
+
+test('a batch-wide master change stops generation before any model call',
async t => {
+ // Seen before the batch, or before the page: not a page failure either
way.
+ for (const failingCheck of [1, 2]) {
+ const fixture = await updateFixture(t);
+ let checks = 0;
+ await assert.rejects(fixture.run({ensureFresh: () => {
+ if (++checks === failingCheck) throw new Error('master changed
translations/glossary/ja.json after the source commit');
+ }}), /translations\/glossary\/ja\.json/);
+ assert.equal(checks, failingCheck);
+ assert.equal(fixture.calls(), 0);
+ assert(!fs.existsSync(path.join(fixture.root,
'output/generation.json')));
+ assert.equal(fs.readFileSync(path.join(fixture.root,
record.targetPath), 'utf8'), fixture.previousTarget);
+ }
+});
+
+test('a batch-wide master change after the last page discards the batch and
keeps its manifest', async t => {
+ const fixture = await updateFixture(t);
+ let checks = 0;
+ await assert.rejects(fixture.run({ensureFresh: () => {
+ if (++checks === 3) throw new Error('master changed
scripts/i18n/translate-docs.cjs after the source commit');
+ }}), /scripts\/i18n\/translate-docs\.cjs/);
+ assert.equal(fixture.calls(), 1);
+ const manifest = JSON.parse(fs.readFileSync(path.join(fixture.root,
'output/generation.json'), 'utf8'));
+ assert.match(manifest.discarded, /scripts\/i18n\/translate-docs\.cjs/);
+ assert.equal(fs.readFileSync(path.join(fixture.root, record.targetPath),
'utf8'), fixture.previousTarget);
+ assert(!fs.existsSync(path.join(fixture.root, 'output/pr-body.md')));
+});
+
+// A source commit on a bare "origin" plus a second clone that advances master.
+function driftFixture(t) {
+ const dir = fs.mkdtempSync(path.join(os.tmpdir(),
'translation-master-drift-'));
+ t.after(() => fs.rmSync(dir, {recursive: true, force: true}));
+ const remote = path.join(dir, 'remote.git'), root = path.join(dir,
'root'), other = path.join(dir, 'other');
+ const git = (cwd, ...args) => execFileSync('git', ['-c', 'user.name=Test',
'-c', '[email protected]',
+ '-c', 'init.defaultBranch=master', ...args], {cwd, encoding: 'utf8',
stdio: ['ignore', 'pipe', 'pipe']}).trim();
+ const write = (cwd, file, text) => {
+ fs.mkdirSync(path.dirname(path.join(cwd, file)), {recursive: true});
+ fs.writeFileSync(path.join(cwd, file), text);
+ };
+ git(dir, 'init', '-q', '--bare', remote);
+ git(dir, 'init', '-q', root);
+ for (const [file, text] of [['docs/guide.md', '# Source\n'],
[record.targetPath, japanese],
+ [stateFileFor(record), '{}\n'], ['translations/glossary/ja.json',
'{}\n'], ['blog/post.md', '# Post\n']])
+ write(root, file, text);
+ git(root, 'add', '.');
+ git(root, 'commit', '-qm', 'source');
+ git(root, 'push', '-q', remote, 'HEAD:refs/heads/master');
+ const sourceCommit = git(root, 'rev-parse', 'HEAD');
+ git(dir, 'clone', '-q', remote, other);
+ const advance = (file, text) => {
+ if (text === null) git(other, 'rm', '-q', file);
+ else write(other, file, text);
+ git(other, 'add', '-A');
+ git(other, 'commit', '-qm', `change ${file}`);
+ git(other, 'push', '-q', 'origin', 'HEAD:master');
+ };
+ const pages = new Map([['T1', pageFiles(record, 'ja')]]);
+ const options = {root, sourceCommit, locale: 'ja', pages, remote};
+ return {git, other, remote, advance, options};
+}
+
+test('master drift allows unrelated merges and reports a changed or moved
batch page', t => {
+ const {git, other, advance, options} = driftFixture(t);
+ assert.deepEqual(masterDrift(options), {head: options.sourceCommit, moved:
false, files: [], batchWide: [], affected: []});
+ advance('blog/post.md', '# Post 2\n');
+ let drift = checkMaster(options);
+ assert.equal(drift.moved, true);
+ assert.deepEqual([drift.files, drift.batchWide, drift.affected],
[['blog/post.md'], [], []]);
+ git(other, 'mv', 'docs/guide.md', 'docs/moved.md');
+ advance('docs/moved.md', '# Source\n');
+ drift = masterDrift(options);
+ // A move reports its old path, so the page that lost its source is
affected.
+ assert.deepEqual(drift.files, ['blog/post.md', 'docs/guide.md',
'docs/moved.md']);
+ assert.deepEqual(drift.affected, ['T1']);
+});
+
+test('master drift discards the batch for translation configuration or tooling
changes', t => {
+ const {advance, options} = driftFixture(t);
+ advance('translations/glossary/ja.json', '{"version": "2"}\n');
+ assert.deepEqual(masterDrift(options).batchWide,
['translations/glossary/ja.json']);
+ assert.throws(() => checkMaster(options), /master changed
translations\/glossary\/ja\.json after the source commit/);
+ const wide = batchWide('ja');
+ for (const file of ['translations/config.json', 'config/i18n.js',
'docusaurus.config.js', 'versions.json',
+ 'translations/scope/ja.json',
'translations/protection-exceptions/ja.json', 'translations/state/ja/docs.json',
+ 'translations/state/ja/shards.schema.json',
'translations/inventory/ja/2026-10-12/summary.json',
+ 'scripts/i18n/workflow-generate.cjs']) assert(wide(file), file);
+ for (const file of [stateFileFor(record), 'translations/glossary/ko.json',
'translations/state/ko/docs.json',
+ 'i18n/ja/code.json', 'sidebars.ts', 'blog/post.md',
'scripts/check_move.js']) assert(!wide(file), file);
+});
+
+test('master drift stops when master no longer contains the source commit', t
=> {
+ const {git, other, options} = driftFixture(t);
+ git(other, 'checkout', '-q', '--orphan', 'rewritten');
+ git(other, 'commit', '-qm', 'rewritten history');
+ git(other, 'push', '-q', '--force', 'origin', 'HEAD:master');
+ assert.throws(() => masterDrift(options), /master no longer contains the
source commit/);
+});
+
+test('a plan is reused only for the same checkout, reservations and filters',
() => {
+ const head = 'a'.repeat(40), active =
'{"schemaVersion":1,"candidates":[],"blockedTargets":[]}\n';
+ const base = planInputs(head, active, {});
+ assert.equal(base, planInputs(head, active, {I18N_LIMIT: '20', I18N_MODE:
'generate'}));
+ for (const changed of [planInputs('b'.repeat(40), active, {}),
+ planInputs(head, active.replace('[]}', '["i18n/ja/x.md"]}'), {}),
+ planInputs(head, active, {I18N_PATHS: '["docs/guide.md"]'}),
planInputs(head, active, {I18N_PLUGIN: 'community'}),
+ planInputs(head, active, {I18N_LIMIT: '5'})]) assert.notEqual(changed,
base);
+});
+
+test('the generation replan reuses an unchanged discovery plan without
recounting', t => {
+ const output = fs.mkdtempSync(path.join(os.tmpdir(),
'translation-plan-reuse-'));
+ t.after(() => fs.rmSync(output, {recursive: true, force: true}));
+ const head = execFileSync('git', ['rev-parse', 'HEAD'], {cwd: repoRoot,
encoding: 'utf8'}).trim();
+ const active = '{"schemaVersion":1,"candidates":[],"blockedTargets":[]}\n';
+ fs.writeFileSync(path.join(output, 'active-candidates.json'), active);
+ fs.writeFileSync(path.join(output, 'open-pr-reconciliation.json'),
JSON.stringify({schemaVersion: 1,
+ blockedTargets: [], botPrs: [], openPrCount: 3, relevantPrs: [],
exceptions: []}));
+ fs.writeFileSync(path.join(output, 'coverage.json'), '{}\n');
+ const env = {...process.env, I18N_LOCALE: 'ko', I18N_MODE: 'generate',
GITHUB_EVENT_NAME: 'schedule',
+ GITHUB_OUTPUT: path.join(output, 'github-output'),
GITHUB_STEP_SUMMARY: path.join(output, 'summary.md')};
+ for (const key of ['I18N_PLUGIN', 'I18N_VERSION', 'I18N_PATH',
'I18N_PATHS', 'I18N_LIMIT']) delete env[key];
+ // A plan the planner could not have produced proves that nothing
recounted it.
+ fs.writeFileSync(path.join(output, 'plan.json'),
JSON.stringify({schemaVersion: 1, locale: 'ko', sourceCommit: head,
+ counts: {required: 1, mergedCurrent: 0, pending: 1, activeCandidates:
0}, selected: ['T-REUSED'],
+ tasks: [{id: 'T-REUSED'}], exceptions: [], planInputs:
planInputs(head, active, env)}));
+ const started = Date.now();
+ execFileSync(process.execPath, [path.join(repoRoot,
'scripts/i18n/workflow-scan.cjs'), output], {env});
+ assert(Date.now() - started < 30000);
+ assert.equal(fs.readFileSync(env.GITHUB_OUTPUT, 'utf8'),
`can_generate=true\nsource_commit=${head}\n`);
+ assert.match(fs.readFileSync(env.GITHUB_STEP_SUMMARY, 'utf8'), /Plan:
reused from discovery/);
+ assert.deepEqual(JSON.parse(fs.readFileSync(path.join(output,
'plan.json'), 'utf8')).selected, ['T-REUSED']);
});
diff --git a/scripts/i18n/master-drift.cjs b/scripts/i18n/master-drift.cjs
new file mode 100644
index 00000000000..cf5bdf442dd
--- /dev/null
+++ b/scripts/i18n/master-drift.cjs
@@ -0,0 +1,57 @@
+// Whether master moved since a batch's source commit, and whether the move
+// touches anything the batch depends on. Runner queues and model calls take
+// long enough that unrelated merges are common, so only these decide:
+// a changed batch page drops that page, while a change to translation
+// configuration, routing identity or the pipeline itself invalidates every
+// candidate and discards the batch.
+const assert = require('node:assert/strict');
+const {execFileSync} = require('node:child_process');
+const {stateFileFor} = require('./state.cjs');
+
+function repositoryUrl(repository = process.env.GITHUB_REPOSITORY ||
'apache/doris-website') {
+ assert.equal(repository, 'apache/doris-website', 'Unexpected repository');
+ return `https://github.com/${repository}.git`;
+}
+
+function batchWide(locale) {
+ const files = new Set(['translations/config.json', 'config/i18n.js',
'docusaurus.config.js', 'versions.json',
+ `translations/scope/${locale}.json`,
`translations/glossary/${locale}.json`,
+ `translations/protection-exceptions/${locale}.json`]);
+ // Top-level state files: the legacy ledger, resolved deletions and the
shard schema.
+ const state = new RegExp(`^translations/state/${locale}/[^/]+$`);
+ return file => files.has(file) || state.test(file) ||
file.startsWith('scripts/i18n/')
+ || file.startsWith(`translations/inventory/${locale}/`);
+}
+
+// Every file one page of a batch reads or writes.
+function pageFiles(task, locale) {
+ return [task.sourcePath, task.targetPath, stateFileFor({...task, locale})];
+}
+
+// `pages` maps a task or candidate ID to its page files.
+function masterDrift({root, sourceCommit, locale, pages = new Map(), remote =
repositoryUrl()}) {
+ const git = (...args) => execFileSync('git', args, {cwd: root, encoding:
'utf8',
+ stdio: ['ignore', 'pipe', 'pipe'], maxBuffer: 64 * 1024 *
1024}).trim();
+ const head = git('ls-remote', remote, 'refs/heads/master').split(/\s+/)[0];
+ assert(/^[a-f0-9]{40}$/.test(head), 'Cannot read the remote master');
+ if (head === sourceCommit) return {head, moved: false, files: [],
batchWide: [], affected: []};
+ git('fetch', '--no-tags', '--quiet', remote, 'refs/heads/master');
+ const fetched = git('rev-parse', 'FETCH_HEAD');
+ try {git('merge-base', '--is-ancestor', sourceCommit, fetched);}
+ catch {throw new Error('master no longer contains the source commit;
discard this batch and rescan');}
+ // Without rename detection a moved file reports both its old and new path.
+ const files = git('diff', '--name-only', '--no-renames', '-z',
sourceCommit, fetched).split('\0').filter(Boolean);
+ const changed = new Set(files);
+ return {head: fetched, moved: true, files, batchWide:
files.filter(batchWide(locale)),
+ affected: [...pages].filter(([, paths]) => paths.some(file =>
changed.has(file))).map(([id]) => id)};
+}
+
+// Generation and publication stop on batch-wide changes and report the rest.
+function checkMaster(options) {
+ const drift = masterDrift(options);
+ assert(!drift.batchWide.length, `master changed ${drift.batchWide.slice(0,
3).join(', ')}`
+ + (drift.batchWide.length > 3 ? ' and more' : '') + ' after the source
commit; discard this batch and rescan');
+ return drift;
+}
+
+module.exports = {repositoryUrl, batchWide, pageFiles, masterDrift,
checkMaster};
diff --git a/scripts/i18n/simulate-workflow.cjs
b/scripts/i18n/simulate-workflow.cjs
index 6695c7d9335..9870775dab8 100644
--- a/scripts/i18n/simulate-workflow.cjs
+++ b/scripts/i18n/simulate-workflow.cjs
@@ -20,6 +20,11 @@ const scenario = process.env.I18N_SIM_SCENARIO || 'retry';
assert([1, 10].includes(pages), 'Simulation supports one or ten pages');
assert(['success', 'retry', 'fallback', 'incremental'].includes(scenario),
'Unknown simulation scenario');
assert(scenario !== 'incremental' || pages === 1, 'Incremental simulation uses
one page');
+// A merge that lands on master after discovery: unrelated, to the selected
page,
+// or to the translation configuration.
+const drift = process.env.I18N_SIM_MASTER_DRIFT || '';
+assert(['', 'unrelated', 'page', 'config'].includes(drift), 'Unknown master
drift');
+assert(!drift || pages === 1, 'Master drift simulation uses one page');
function command(binary, args, options = {}) {
return execFileSync(binary, args, {encoding: 'utf8', ...options}).trim();
@@ -50,6 +55,8 @@ const {extractUnits} =
require(${JSON.stringify(path.join(repo, 'scripts/i18n/tr
const units = await extractUnits(translation,
${JSON.stringify(path.extname(sourcePath))});
for (const unit of [...units].reverse()) translation =
translation.slice(0, unit.start)
+ '訳' + unit.text + translation.slice(unit.end);
+ // Like a real model, localize a declared front matter language.
+ translation = translation.replace(/("language":\\s*")en(")/, '$1ja$2');
if (${JSON.stringify(mode)} === 'invalid') translation = '';
if (${JSON.stringify(mode)} === 'retry' && payload.feedback.length === 0)
translation = translation.replace(/^#{1,6} .*$/gm, '');
@@ -179,6 +186,44 @@ async function main() {
runNode(repo, 'scripts/i18n/workflow-scan.cjs', [scan], env, logs);
const plan = JSON.parse(fs.readFileSync(path.join(scan, 'plan.json'),
'utf8'));
assert.equal(plan.selected.length, pages);
+ if (drift) {
+ const other = path.join(output, 'other');
+ command('git', ['clone', '--shared', '-q', remote, other]);
+ command('git', ['config', '--local', 'user.name', 'Local Simulator'],
{cwd: other});
+ command('git', ['config', '--local', 'user.email',
'[email protected]'], {cwd: other});
+ const file = {unrelated: 'blog/translation-simulation-drift.md',
config: 'translations/glossary/ja.json',
+ page: plan.tasks.find(task => task.id ===
plan.selected[0]).sourcePath}[drift];
+ fs.mkdirSync(path.dirname(path.join(other, file)), {recursive: true});
+ fs.appendFileSync(path.join(other, file), drift === 'config' ? '\n' :
'\nMaster drift simulation.\n');
+ command('git', ['add', '.'], {cwd: other});
+ command('git', ['commit', '-qm', `Master drift simulation: ${drift}`],
{cwd: other});
+ command('git', ['push', '-q', 'origin', 'HEAD:master'], {cwd: other});
+ }
+ if (drift === 'page' || drift === 'config') {
+ // Neither model sees a request: the page is skipped, or the batch
stops first.
+ const results = ['invalid', 'valid'].map(kind => {
+ const result = runNode(repo, 'scripts/i18n/workflow-generate.cjs',
[scan],
+ {...env, CODEX_BIN: path.join(bin, `codex-${kind}`),
I18N_SIM_PHASE: `-${kind}`}, logs, 1);
+ const file = path.join(scan, 'generation.json');
+ if (drift === 'config') {
+ assert.match(result.stderr, /master changed
translations\/glossary\/ja\.json after the source commit/);
+ assert(!fs.existsSync(file));
+ return {exitCode: result.status, error: result.stderr.trim()};
+ }
+ const manifest = JSON.parse(fs.readFileSync(file, 'utf8'));
+ assert.equal(manifest.candidates.length, 0);
+ assert.equal(manifest.usedTokens, 0);
+ assert.match(manifest.failures[0].error, /master changed this page
after the source commit/);
+ assert.equal(manifest.master.moved, true);
+ return {exitCode: result.status, error:
manifest.failures[0].error, master: manifest.master};
+ });
+ const report = {kind: 'local simulation; no external model or GitHub
writes', output, drift,
+ sourceCommit: plan.sourceCommit, selected: plan.selected,
sourcePath: selectedPath, generation: results,
+ pullRequest: null};
+ fs.writeFileSync(path.join(output, 'report.json'),
JSON.stringify(report, null, 2) + '\n');
+ process.stdout.write(JSON.stringify(report, null, 2) + '\n');
+ return;
+ }
const failed = {...env, CODEX_BIN: path.join(bin, 'codex-invalid'),
I18N_SIM_PHASE: '-invalid'};
runNode(repo, 'scripts/i18n/workflow-generate.cjs', [scan], failed, logs,
pages === 1 ? 1 : 0);
const failure = JSON.parse(fs.readFileSync(path.join(scan,
'generation.json'), 'utf8'));
@@ -201,6 +246,12 @@ async function main() {
runNode(repo, 'scripts/i18n/workflow-publish.cjs', [scan], env, logs);
const pr = JSON.parse(fs.readFileSync(env.I18N_SIM_PR_RECORD, 'utf8'));
assert.match(pr.body, /Shared PR admission checks: passed/);
+ if (drift === 'unrelated') {
+ // Published from the source commit; the PR checks run against current
master.
+ assert.equal(generation.master.moved, true);
+ assert.match(pr.body, /Master moved to `[a-f0-9]{40}` during
generation; none of its 1 changed files affect this batch/);
+ assert.match(pr.body, /Master was at `[a-f0-9]{40}` when publishing;
none of its 1 changed files/);
+ } else assert.equal(generation.master.moved, false);
assert.equal(pr.head, `i18n/ja-automated-${runId}`);
const branch = command('git', ['rev-parse', `refs/heads/${pr.head}`],
{cwd: remote});
const candidate = generation.candidates[0];
@@ -217,7 +268,8 @@ async function main() {
const expectedFiles = [...new Set(generation.candidates.flatMap(item =>
[item.targetPath, item.statePath]))].sort();
const committedFiles = command('git', ['diff', '--name-only',
plan.sourceCommit, branch], {cwd: repo}).split('\n').sort();
assert.deepEqual(committedFiles, expectedFiles);
- const report = {kind: 'local simulation; no external model or GitHub
writes', output,
+ const report = {kind: 'local simulation; no external model or GitHub
writes', output, drift: drift || null,
+ master: generation.master,
sourceCommit: plan.sourceCommit, selected: plan.selected, sourcePath:
selectedPath, scenario,
invalidModel: {exitCode: pages === 1 ? 1 : 0, error:
failure.failures[0].error, usedTokens: failure.usedTokens},
generation: {candidates: generation.candidates.length, failures:
generation.failures.length,
diff --git a/scripts/i18n/workflow-generate.cjs
b/scripts/i18n/workflow-generate.cjs
index 9a6b4bde2f9..635726e3e87 100644
--- a/scripts/i18n/workflow-generate.cjs
+++ b/scripts/i18n/workflow-generate.cjs
@@ -9,6 +9,7 @@ const {checkedTask, managedPath, safeFile} =
require('./translation-engine.cjs')
const {codexAdapter} = require('./codex-adapter.cjs');
const {prepare, applyCandidates, readState, stagedBaseline,
statePathForRecord, targetFor} = require('./translate-docs.cjs');
const {markerPrefixFor} = require('./automation-naming.cjs');
+const {checkMaster, pageFiles} = require('./master-drift.cjs');
const root = path.resolve(__dirname, '../..');
const allowedKinds = new Set(['translate_new', 'translate_backlog', 'update']);
@@ -55,14 +56,6 @@ function copyPair(primary, partner, scope, state, rootDir =
root) {
requests: 0, reusedFrom: primary.task.id};
}
-function ensureMasterFresh(sourceCommit) {
- const repository = process.env.GITHUB_REPOSITORY || 'apache/doris-website';
- assert.equal(repository, 'apache/doris-website', 'Unexpected repository');
- const actual = execFileSync('git', ['ls-remote',
`https://github.com/${repository}.git`, 'refs/heads/master'],
- {encoding: 'utf8'}).trim().split(/\s+/)[0];
- assert.equal(actual, sourceCommit, 'master advanced during generation;
discard this batch and rescan');
-}
-
function bodyFor(manifest) {
const rows = manifest.candidates.map(item => {
const review = item.generation;
@@ -79,8 +72,10 @@ function bodyFor(manifest) {
return `- \`${item.sourcePath}\`: third attempt retained for
review. `
+ detail.slice(0, 1500) + (detail.length > 1500 ? '…' : '');
});
+ const master = manifest.master?.moved ? [`Master moved to
\`${manifest.master.head}\` during generation; `
+ + `none of its ${manifest.master.changedFiles} changed files affect
this batch.`] : [];
return [`## ${manifest.language} translation candidate`, '',
- `Source commit: \`${manifest.sourceCommit}\``,
+ `Source commit: \`${manifest.sourceCommit}\``, ...master,
`Model: \`${manifest.model}\`; glossary:
\`${manifest.glossaryVersion}\``,
`Review status: machine. ${manifest.language} proofreading and the
normal PR checks are still required.`,
'', '| Action | Version | English source | SHA-256 prefix | Generation
|',
@@ -92,14 +87,16 @@ function bodyFor(manifest) {
'Generation uses at most three attempts per document. Deletions and
code-only updates may need no model call.',
'Each retry receives the earlier validation errors. Third-attempt
content issues are retained as warnings above.',
'Full attempt history and diagnostics are in generation.json and the
page state records.',
- 'Committed source hashes, managed paths, target/state hashes and the
master SHA are checked before applying candidates.',
+ 'Committed source hashes, managed paths and target/state hashes are
checked before applying candidates.',
+ 'Master is checked before each model call and before publishing:
unrelated merges are allowed, a page changed on master is dropped, and a
translation configuration or tooling change discards the batch.',
'The publisher runs the shared PR translation-state, syntax and
internal-link checks before pushing.',
'The normal PR build and human proofreading are still required before
merging.',
'', `${markerPrefixFor(manifest.language)}PENDING -->`, ''].join('\n');
}
-async function generate({plan, reconciliation, maxPages = 10, target,
- adapter, glossary, rootDir = root, ensureFresh = ensureMasterFresh,
output}) {
+async function generate({plan, reconciliation, maxPages = 10, target, adapter,
glossary, rootDir = root,
+ ensureFresh = (sourceCommit, pages) => checkMaster({root: rootDir,
sourceCommit, locale: target.locale, pages}),
+ output}) {
assert(Number.isInteger(maxPages) && maxPages > 0 && maxPages <= 20,
'Invalid page limit');
assert(plan.schemaVersion === 1 && plan.sourceCommit ===
execFileSync('git', ['rev-parse', 'HEAD'],
{cwd: rootDir, encoding: 'utf8'}).trim(), 'Plan does not match trusted
checkout');
@@ -108,6 +105,22 @@ async function generate({plan, reconciliation, maxPages =
10, target,
const blocked = new Set(reconciliation.blockedTargets);
const active = new Set((reconciliation.active?.candidates ??
[]).map(identity));
const candidates = [], failures = [], consumed = new Set();
+ // Master is checked before the first model call, before each page and
after
+ // the last one, so no model call goes to a page master has already
changed.
+ const pages = new Map();
+ for (const task of plan.selected.map(id => plan.tasks.find(item => item.id
=== id)).filter(Boolean)) {
+ for (const item of [task, pairFor(task, plan.tasks,
scope)].filter(Boolean))
+ pages.set(item.id, pageFiles(item, target.locale));
+ }
+ const stale = new Set();
+ let drift = {moved: false};
+ const refresh = () => {
+ drift = ensureFresh(plan.sourceCommit, pages) ?? drift;
+ for (const id of drift.affected ?? []) stale.add(id);
+ };
+ const staleFailure = task => ({id: task.id, sourcePath: task.sourcePath,
+ error: 'master changed this page after the source commit; it stays
pending for the next run'});
+ refresh();
for (const id of plan.selected) {
if (candidates.length >= maxPages) break;
const task = plan.tasks.find(item => item.id === id);
@@ -122,11 +135,17 @@ async function generate({plan, reconciliation, maxPages =
10, target,
if (partner && (active.has(identity(partner)) ||
blocked.has(partner.targetPath) || !machineOwned(partner, state)
|| state.data.pages.some(record => identity(record) ===
identity(partner))
|| candidates.length + 2 > maxPages)) partner = null;
+ // Outside the per-page catch: a batch-wide change must stop the run.
+ refresh();
+ if (stale.has(id)) {
+ failures.push(staleFailure(task));
+ continue;
+ }
try {
checkedTask(plan, id, scope, rootDir);
const primary = await prepare({root: rootDir, plan, taskId: id,
scope, state, adapter, glossary, target});
candidates.push(primary);
- if (partner) try {
+ if (partner && !stale.has(partner.id)) try {
const copy = copyPair(primary, partner, scope, state, rootDir);
candidates.push(copy);
consumed.add(partner.id);
@@ -137,10 +156,20 @@ async function generate({plan, reconciliation, maxPages =
10, target,
failures.push({id, sourcePath: task.sourcePath, error:
error.message});
}
}
+ let discarded = null;
+ if (candidates.length) {
+ try {refresh();} catch (error) {discarded = error;}
+ // A copy shares its primary's translation, so it goes with it.
+ const dropped = candidates.filter(item => stale.has(item.task.id) ||
stale.has(item.reusedFrom));
+ for (const item of dropped) failures.push(staleFailure(item.task));
+ candidates.splice(0, candidates.length, ...candidates.filter(item =>
!dropped.includes(item)));
+ }
const manifest = {schemaVersion: 1, sourceCommit: plan.sourceCommit,
provider: 'codex',
locale: target.locale, language: target.language,
model: adapter.model, glossaryVersion: glossary.version, liveCall:
adapter.usedTokens > 0,
liveVerified: false, usedTokens: adapter.usedTokens,
+ ...(drift.head ? {master: {head: drift.head, moved: drift.moved,
changedFiles: drift.files.length}} : {}),
+ ...(discarded ? {discarded: discarded.message} : {}),
candidates: candidates.map(item => ({id: item.task.id, kind:
item.task.kind,
pluginId: item.task.pluginId, version: item.task.version, docId:
item.task.docId,
sourcePath: item.task.sourcePath, sourceHash: item.task.sourceHash,
@@ -150,8 +179,8 @@ async function generate({plan, reconciliation, maxPages =
10, target,
...(item.record.generation ? {generation: item.record.generation}
: {})})), failures};
fs.mkdirSync(output, {recursive: true});
fs.writeFileSync(path.join(output, 'generation.json'),
JSON.stringify(manifest, null, 2) + '\n');
+ if (discarded) throw discarded;
if (candidates.length) {
- ensureFresh(plan.sourceCommit);
for (const item of candidates) checkedTask({...plan, selected:
[...plan.selected, item.task.id]},
item.task.id, scope, rootDir);
for (const item of candidates) {
@@ -195,7 +224,9 @@ async function main() {
`created=${manifest.candidates.length > 0}\n`);
if (process.env.GITHUB_STEP_SUMMARY)
fs.appendFileSync(process.env.GITHUB_STEP_SUMMARY,
`\nGenerated ${manifest.candidates.length} ${manifest.language} page
candidates at ${manifest.sourceCommit}; `
- + `${manifest.failures.length} failed or need manual handling. No
translation is merged yet.\n`);
+ + `${manifest.failures.length} failed or need manual handling. No
translation is merged yet.\n`
+ + (manifest.master?.moved ? `Master moved to ${manifest.master.head}
meanwhile; none of its `
+ + `${manifest.master.changedFiles} changed files affect the
remaining candidates.\n` : ''));
process.stdout.write(JSON.stringify({candidates:
manifest.candidates.length,
failures: manifest.failures.length, usedTokens: manifest.usedTokens})
+ '\n');
if (process.env.I18N_MODE === 'validate' && manifest.candidates.length ===
0)
diff --git a/scripts/i18n/workflow-publish.cjs
b/scripts/i18n/workflow-publish.cjs
index 2a3fc7bdad6..0897f8f6f8f 100644
--- a/scripts/i18n/workflow-publish.cjs
+++ b/scripts/i18n/workflow-publish.cjs
@@ -7,6 +7,7 @@ const {languageName, pipelineLocale, stateFileFor} =
require('./state.cjs');
const {managedPath} = require('./translation-engine.cjs');
const {statePathFor} = require('./workflow-prs.cjs');
const {botBranchFor, botBranchPattern} = require('./automation-naming.cjs');
+const {checkMaster} = require('./master-drift.cjs');
const root = path.resolve(__dirname, '../..');
const git = (...args) => execFileSync('git', args, {cwd: root, encoding:
'utf8'}).trim();
@@ -74,10 +75,15 @@ async function main() {
const body = fs.readFileSync(bodyFile, 'utf8');
assert(body.includes('bot-head=PENDING'), 'Missing PR ownership marker');
await require('./pr-preflight.cjs').preflight({root, base:
manifest.sourceCommit, locale});
+ // The branch stays on the source commit; the PR checks run against
current master.
+ const drift = checkMaster({root, sourceCommit: manifest.sourceCommit,
locale,
+ pages: new Map(manifest.candidates.map(item => [item.id,
+ [item.sourcePath, item.targetPath, item.statePath ??
statePathFor(locale)]]))});
+ assert(!drift.affected.length, `master changed ${drift.affected.join(',
')} after the source commit; rescan required`);
fs.writeFileSync(bodyFile, body.replace('bot-head=PENDING',
`bot-head=${newHead}`)
- + `\nShared PR admission checks: passed (translation state, syntax,
internal links).\n`);
- const remoteMaster = git('ls-remote', 'origin',
'refs/heads/master').split(/\s+/)[0];
- assert.equal(remoteMaster, manifest.sourceCommit, 'master advanced before
publish; rescan required');
+ + `\nShared PR admission checks: passed (translation state, syntax,
internal links).\n`
+ + (drift.moved ? `Master was at \`${drift.head}\` when publishing;
none of its ${drift.files.length} `
+ + 'changed files since the source commit affect this batch.\n' :
''));
// gh configures Git's credential helper from GH_TOKEN. The token is never
// placed in a remote URL, command argument, log or generated file.
execFileSync('gh', ['auth', 'setup-git'], {cwd: root, stdio: 'inherit'});
diff --git a/scripts/i18n/workflow-scan.cjs b/scripts/i18n/workflow-scan.cjs
index 32adc3d5a2a..68721231157 100644
--- a/scripts/i18n/workflow-scan.cjs
+++ b/scripts/i18n/workflow-scan.cjs
@@ -4,7 +4,7 @@ const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
const {execFileSync} = require('node:child_process');
-const {languageName, pipelineLocale} = require('./state.cjs');
+const {hash, languageName, pipelineLocale} = require('./state.cjs');
const root = path.resolve(__dirname, '../..');
const configInputs = ['docusaurus.config.js', 'config/i18n.js', 'sidebars.ts',
'sidebarsCommunity.json',
@@ -34,7 +34,14 @@ function changedPaths(before, head) {
catch {return null;}
}
-function summarize(plan, reconciliation, eligible, mode, language) {
+// Everything a plan depends on besides the planner itself: the checkout, the
+// open-PR reservations (blocked targets included) and the requested filters.
+function planInputs(head, activeCandidates, env = process.env) {
+ return hash(JSON.stringify({head, activeCandidates, filters:
['I18N_PLUGIN', 'I18N_VERSION', 'I18N_PATH', 'I18N_PATHS']
+ .map(key => env[key] || '').concat(env.I18N_LIMIT || '20')}));
+}
+
+function summarize(plan, reconciliation, eligible, mode, language, reused =
false) {
const exceptionKinds = new Map();
for (const item of [...plan.exceptions, ...reconciliation.exceptions])
exceptionKinds.set(item.kind, (exceptionKinds.get(item.kind) ?? 0) +
1);
@@ -42,6 +49,8 @@ function summarize(plan, reconciliation, eligible, mode,
language) {
`## ${language} translation discovery`,
'',
`- Source commit: \`${plan.sourceCommit}\``,
+ `- Plan: ${reused ? 'reused from discovery; master and open-PR
reservations are unchanged'
+ : 'recounted from the native documents'}`,
`- Mode: \`${mode}\`; generation eligible: **${eligible}**`,
`- Required: ${plan.counts.required}; merged current:
${plan.counts.mergedCurrent}; pending: ${plan.counts.pending}`,
`- Open PRs inspected: ${reconciliation.openPrCount}; relevant:
${reconciliation.relevantPrs.length}; active candidates:
${plan.counts.activeCandidates}`,
@@ -63,25 +72,37 @@ function main() {
const reconciliation = JSON.parse(fs.readFileSync(path.join(output,
'open-pr-reconciliation.json'), 'utf8'));
const planFile = path.join(output, 'plan.json');
const auditFile = path.join(output, 'coverage.json');
- const args = [path.join(__dirname, 'plan-translations.cjs'),
'--source-ref', 'HEAD',
- '--active-candidates', path.join(output, 'active-candidates.json'),
'--output', planFile,
- '--audit-output', auditFile, '--limit', process.env.I18N_LIMIT ||
'20'];
- for (const [key, option] of [['I18N_PLUGIN', '--plugin'], ['I18N_VERSION',
'--version'],
- ['I18N_PATH', '--path'], ['I18N_PATHS', '--paths']]) {
- if (process.env[key]) args.push(option, process.env[key]);
+ const activeFile = path.join(output, 'active-candidates.json');
+ const event = process.env.GITHUB_EVENT_NAME || 'workflow_dispatch';
+ const inputs = planInputs(execFileSync('git', ['rev-parse', 'HEAD'], {cwd:
root, encoding: 'utf8'}).trim(),
+ fs.readFileSync(activeFile, 'utf8'));
+ // The generation job downloads the discovery plan. Recounting every native
+ // document takes minutes, so an unchanged plan is reused.
+ const previous = fs.existsSync(planFile) && fs.existsSync(auditFile)
+ ? JSON.parse(fs.readFileSync(planFile, 'utf8')) : null;
+ const reused = event !== 'push' && previous?.planInputs === inputs;
+ let plan = previous;
+ if (!reused) {
+ const args = [path.join(__dirname, 'plan-translations.cjs'),
'--source-ref', 'HEAD',
+ '--active-candidates', activeFile, '--output', planFile,
+ '--audit-output', auditFile, '--limit', process.env.I18N_LIMIT ||
'20'];
+ for (const [key, option] of [['I18N_PLUGIN', '--plugin'],
['I18N_VERSION', '--version'],
+ ['I18N_PATH', '--path'], ['I18N_PATHS', '--paths']]) {
+ if (process.env[key]) args.push(option, process.env[key]);
+ }
+ execFileSync(process.execPath, args, {cwd: root, stdio: 'inherit'});
+ plan = JSON.parse(fs.readFileSync(planFile, 'utf8'));
+ const blocked = new Set(reconciliation.blockedTargets);
+ const originalSelected = plan.selected.length;
+ plan.selected = plan.selected.filter(id =>
!blocked.has(plan.tasks.find(task => task.id === id)?.targetPath));
+ plan.selectedCount = plan.selected.length;
+ plan.selectedEstimatedSourceTokens = plan.selected.reduce((total, id)
=>
+ total + (plan.tasks.find(task => task.id ===
id)?.estimatedSourceTokens || 0), 0);
+ plan.blockedSelected = originalSelected - plan.selected.length;
+ plan.planInputs = inputs;
+ fs.writeFileSync(planFile, JSON.stringify(plan, null, 2) + '\n');
}
- execFileSync(process.execPath, args, {cwd: root, stdio: 'inherit'});
- const plan = JSON.parse(fs.readFileSync(planFile, 'utf8'));
const audit = JSON.parse(fs.readFileSync(auditFile, 'utf8'));
- const blocked = new Set(reconciliation.blockedTargets);
- const originalSelected = plan.selected.length;
- plan.selected = plan.selected.filter(id =>
!blocked.has(plan.tasks.find(task => task.id === id)?.targetPath));
- plan.selectedCount = plan.selected.length;
- plan.selectedEstimatedSourceTokens = plan.selected.reduce((total, id) =>
- total + (plan.tasks.find(task => task.id ===
id)?.estimatedSourceTokens || 0), 0);
- plan.blockedSelected = originalSelected - plan.selected.length;
- fs.writeFileSync(planFile, JSON.stringify(plan, null, 2) + '\n');
- const event = process.env.GITHUB_EVENT_NAME || 'workflow_dispatch';
const mode = process.env.I18N_MODE || 'dry-run';
const changed = event === 'push' ?
changedPaths(process.env.GITHUB_EVENT_BEFORE, plan.sourceCommit) : [];
const locale = pipelineLocale(root);
@@ -94,7 +115,7 @@ function main() {
}
const eligible = event === 'schedule' || event === 'workflow_dispatch' &&
['generate', 'validate'].includes(mode)
|| event === 'push' && changed !== null && relevantPush(changed,
audit, scope);
- const summary = summarize(plan, reconciliation, eligible, mode,
languageName(root, locale));
+ const summary = summarize(plan, reconciliation, eligible, mode,
languageName(root, locale), reused);
if (process.env.GITHUB_STEP_SUMMARY)
fs.appendFileSync(process.env.GITHUB_STEP_SUMMARY, summary);
else process.stdout.write(summary);
if (process.env.GITHUB_OUTPUT) fs.appendFileSync(process.env.GITHUB_OUTPUT,
@@ -105,4 +126,4 @@ function main() {
if (require.main === module) {
try {main();} catch (error) {console.error(error.message);
process.exitCode = 1;}
}
-module.exports = {relevantPush, changedPaths, summarize, rulesFor};
+module.exports = {relevantPush, changedPaths, summarize, rulesFor, planInputs};
diff --git a/translations/README.md b/translations/README.md
index 75806b97e50..b318b8f4a28 100644
--- a/translations/README.md
+++ b/translations/README.md
@@ -104,10 +104,11 @@ exercise these boundaries without real credentials or
model calls; actual
single-page generation is a separate validation step.
The `Korean translation discovery` workflow runs at 04:43 and 16:43 UTC daily
-and on manual dispatch. It fixes the locale to `ko`, uses separate Korean
-artifacts, and shares the existing `japanese-translation-ja` lock with Japanese
-runs. The schedule starts twenty minutes after the Japanese one, so a Korean
run
-waits while a Japanese run is still in progress. A scheduled run generates a
+and on manual dispatch. Like the Japanese workflow, it calls the shared
+`.github/workflows/translation-run.yml`, here with the locale fixed to `ko` and
+its own `korean-translation-*` artifacts; both workflows hold the same
+`docs-translation` lock. The schedule starts twenty minutes after the Japanese
+one, so a Korean run waits while a Japanese run is still in progress. A
scheduled run generates a
batch when eligible Korean documents are pending, with the same limits as a
manual `generate` run: the planner considers at most 20 targets and a batch has
at most 10 pages. It opens a normal labeled PR that maintainers review and
merge
@@ -367,9 +368,9 @@ The translation pipeline works on one locale per run.
`scripts/i18n/state.cjs`
exposes `pipelineLocale(root)`: a configuration that declares a single locale
needs no selector and behaves exactly as before, and `I18N_LOCALE` picks one as
soon as several are managed. An absent or unconfigured selector then fails
-before discovery or generation. The Japanese translation workflow defaults its
-manual input and scheduled runs to `ja`; the Japanese coverage audit always
-selects `ja`. A manual translation run can select another configured locale.
+before discovery or generation. The Japanese coverage audit always
+selects `ja`. Neither translation workflow accepts a locale input: the Japanese
+workflow passes `ja` and the Korean workflow `ko` to the shared run.
Build Check runs PR admission for every configured translation locale,
including
locales disabled for the site build, when a shared input changes. Otherwise it
@@ -471,10 +472,15 @@ be applied to managed translation files. The offline
fixture does not establish
live model quality, latency, cost or availability. A real limited call and
review are recommended before running batch generation.
-## Asynchronous Japanese discovery and generation
+## Asynchronous discovery and generation
`.github/workflows/japanese-translation.yml` runs at 04:23 and 16:23 UTC daily
-and on manual dispatch. It has no push trigger. Every run first recounts
+and on manual dispatch; `.github/workflows/korean-translation.yml` runs twenty
+minutes later. Each sets only its triggers, manual defaults and the lock, then
+calls `.github/workflows/translation-run.yml` with its locale and artifact
+prefix, so both languages run the same steps. Neither has a push trigger.
+The description below uses Japanese; Korean differs only in its locale, label,
+artifact names and manual default. Every run first recounts
the current native Dev, 4.x and community documents, reads every open master
PR's changed files, and checks Japanese target files against that PR's state
records. Each run uploads `plan.json`, `coverage.json`,
@@ -486,22 +492,28 @@ attention and neither closes nor overwrites the existing
PR. A PR whose
candidate cannot be verified blocks that target from robot generation and
appears in the exception report.
-Manual dispatch defaults to `generate` and accepts optional plugin, version and
-source path filters. `validate` mode requires one exact English source path
+Japanese manual dispatch defaults to `generate` (Korean to `dry-run`) and
+accepts optional plugin, version and source path prefix filters, or an exact
+`paths` list. `validate` mode requires one exact English source path
ending in `.md` or `.mdx`; it can call the model and create a regular PR for
one
page. It provides a single-page live check after OSS model access is available.
With OSS access but no GitHub write credential, `validate` still writes the
candidate to the generation artifact and makes no GitHub write.
`dry-run` remains available for read-only discovery. Full scans recover
pending work from new or changed source documents and earlier failed runs.
-One workflow-level concurrency group covers every trigger, mode and locale.
+One concurrency group, `docs-translation`, covers both workflows, every
+trigger and every mode. Rename it only while no translation run is queued or
+running; otherwise the run in flight and the next one hold different locks.
Only one complete run can execute at a time; `cancel-in-progress: false`
preserves it when another run is requested. `queue: max` lets up to 100
pending runs wait instead of replacing the previous pending run; see
[GitHub's concurrency
documentation](https://docs.github.com/en/actions/how-tos/write-workflows/choose-when-workflows-run/control-workflow-concurrency).
Each queued run takes a fresh master checkout and recounts pending documents.
-Translation PR
-merges do not trigger this workflow. It does not participate in website
+The generation job checks out master again instead of the discovery commit,
+because it can wait for a runner long after discovery, then reconciles open PRs
+and replans. When master and the open-PR reservations are unchanged since
+discovery, it reuses the discovery plan instead of recounting every native
+document. Translation PR merges do not trigger this workflow. It does not
participate in website
builds or release deployment.
The mode controls generation directly: manual `generate` runs and scheduled
@@ -545,7 +557,9 @@ The generator calls the model once for exact Dev/4.x pairs
whose doc ID,
relative source path and normalized source hash match. It writes two separate
state records. Human-owned or changed targets, moves, deletions and other
protected tasks stay in the manual queue. The robot opens a regular PR with
-machine review status and the `translate-ja` label, and never merges it.
+machine review status and the `translate-ja` label, and never merges it. The
+run checks that the label exists before loading Codex auth, because the
+publisher applies it only after the model calls and the branch push.
Maintainers review and merge PRs when ready. Multiple batches can remain open:
their documents are excluded, while unrelated pending documents can form the
next batch. This exclusion includes stale candidates and manually edited
@@ -570,6 +584,21 @@ warnings to the exact source and target hashes. The PR
translation gate recogniz
that recorded fallback, and the PR body lists its warnings for review. Three
requests that return no usable document remain an operational failure.
+Master usually moves while a batch waits for a runner or for the model.
+Generation checks it before the first model call, before each page and after
the
+last one, and the publisher checks it again before pushing
+(`scripts/i18n/master-drift.cjs`). A merge that touches none of the batch's
files
+leaves the batch valid: the PR branch stays on the source commit, the PR body
+notes the newer master, and the normal PR checks run against current master. A
+merge that changes a batch page's English source, target or state shard drops
+that page, which stays pending for the next run; the publisher stops instead.
+A merge that changes translation configuration, routing identity or the
+pipeline itself discards the batch before any further model call: the
+locale's scope, glossary, protection exceptions, inventory or top-level state
+files, `translations/config.json`, `config/i18n.js`, `docusaurus.config.js`,
+`versions.json` or `scripts/i18n/`. So does a master that no longer contains
+the source commit.
+
Before pushing, the publisher runs `scripts/i18n/pr-preflight.cjs`, the same
entry
point used by the PR build workflow. It checks translation state, Markdown/MDX
syntax and internal links using the PR link checker's resolver. Invalid syntax
@@ -591,6 +620,10 @@ I18N_SIM_PAGES=10 I18N_SIM_SCENARIO=fallback \
# Translate a real page exceeding the former 60,000-character limit in one
request.
I18N_SIM_SOURCE_PATH=docs/admin-manual/config/be-config.md
I18N_SIM_SCENARIO=success \
node scripts/i18n/simulate-workflow.cjs /tmp/doris-ja-long-page-sim
+
+# A merge lands on master after discovery: unrelated, to the selected page,
+# or to the Japanese glossary.
+I18N_SIM_MASTER_DRIFT=unrelated node scripts/i18n/simulate-workflow.cjs
/tmp/doris-ja-drift-sim
```
The output directory must be empty. The simulator copies the current scripts,
@@ -600,6 +633,10 @@ source fixtures there. Real open-PR reconciliation and
full planning run against
a GitHub CLI mock, followed by the production engine and a schema-shaped Codex
mock. `I18N_SIM_SCENARIO` accepts `success` (one attempt), `retry` (two
attempts,
the default), or `fallback` (three attempts retained with warnings).
+`I18N_SIM_MASTER_DRIFT` pushes one more commit to the local remote after
+discovery: `unrelated` publishes normally and records the newer master in the
PR
+body, `page` skips the selected page without a model request, and `config`
stops
+the batch before any model request.
The simulator also verifies three empty responses, token accounting, the shared
PR admission checks, and publication of only managed Japanese targets/state to
@@ -609,10 +646,10 @@ No real model call or GitHub write occurs.
## Validation
-### Selecting an exact Korean manual batch
+### Selecting an exact manual batch
-Manual Korean dispatch defaults to `dry-run`. Its optional
-`paths` input accepts a JSON array of 1–20 exact English source files, so a
bounded
+Both translation workflows take an optional `paths` input on manual dispatch.
+It accepts a JSON array of 1–20 exact English source files, so a bounded
batch can select documents from different folders without including every page
under a shared prefix:
diff --git a/translations/scope/ko.json b/translations/scope/ko.json
index 1aae217ddac..cb6dd2d4391 100644
--- a/translations/scope/ko.json
+++ b/translations/scope/ko.json
@@ -30,6 +30,12 @@
"key": "sidebar.docs.category.S3",
"reason": "Storage service name."
},
+ {
+ "pluginId": "default",
+ "version": "current",
+ "key": "sidebar.docs.category.Hive Catalog",
+ "reason": "Named Doris Catalog connector; preserve its documented
connector name."
+ },
{
"pluginId": "default",
"version": "current",
@@ -42,6 +48,12 @@
"key": "sidebar.docs.category.Paimon Catalog",
"reason": "Named Doris Catalog connector; preserve its documented
connector name."
},
+ {
+ "pluginId": "default",
+ "version": "current",
+ "key": "sidebar.docs.category.Lance Catalog",
+ "reason": "Named Doris Catalog connector; preserve its documented
connector name."
+ },
{
"pluginId": "default",
"version": "current",
@@ -114,12 +126,24 @@
"key": "sidebar.docs.category.S3",
"reason": "Storage service name."
},
+ {
+ "pluginId": "default",
+ "version": "4.x",
+ "key": "sidebar.docs.category.Hive Catalog",
+ "reason": "Named Doris Catalog connector; preserve its documented
connector name."
+ },
{
"pluginId": "default",
"version": "4.x",
"key": "sidebar.docs.category.Iceberg Catalog",
"reason": "Named Doris Catalog connector; preserve its documented
connector name."
},
+ {
+ "pluginId": "default",
+ "version": "4.x",
+ "key": "sidebar.docs.category.Lance Catalog",
+ "reason": "Named Doris Catalog connector; preserve its documented
connector name."
+ },
{
"pluginId": "default",
"version": "4.x",
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
