This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new c1dc0ada582 [fix](i18n) Prepare pipelines for multiple managed locales
(#4352)
c1dc0ada582 is described below
commit c1dc0ada58258822db4f31d9f05b11cf3e011b7b
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Sun Oct 18 12:32:17 2026 +0100
[fix](i18n) Prepare pipelines for multiple managed locales (#4352)
Adding a second managed translation locale currently breaks scheduled
Japanese runs, makes native inventory reconcile each document twice, and
lets one language overwrite another in SEO alternates.
This change defaults Japanese automation and simulation to `ja`, checks
every configured translation locale in Build Check, and forwards the
publisher's selected locale into admission checks. Native inventory
isolates pages and reports by locale. Document metadata, manifests, and
browser routes share complete alternate groups keyed by plugin, version,
and doc ID, including differing Chinese slugs on fallback pages.
Validation on Node 20.20.2:
- 158 i18n/publish tests passed; one existing Apache integration test
skipped by its environment condition.
- Real Docusaurus metadata fixtures cover both managed languages,
repeated doc IDs across plugins/versions, stale/missing translations,
drafts/unlisted pages, and disabled Korean.
- Small four-language and Korean-disabled builds pass static SEO checks
and Playwright hydration/navigation checks.
- The existing Japanese status fixture passes its build, static checks,
and Playwright checks.
- Japanese PR admission passes on the committed branch against the
latest master.
Korean is declared only inside fixtures. Production locale registration
and translation content are handled in subsequent changes.
Co-authored-by: morningman <[email protected]>
---
.github/workflows/build-check.yml | 11 +-
.github/workflows/japanese-coverage-audit.yml | 2 +
.github/workflows/japanese-translation.yml | 7 +-
scripts/i18n/__tests__/fixtures/multi-locale.cjs | 85 ++++++++++++++
scripts/i18n/__tests__/locale-registry.test.cjs | 4 +-
scripts/i18n/__tests__/multi-locale.test.cjs | 141 +++++++++++++++++++++++
scripts/i18n/alternates.cjs | 18 +++
scripts/i18n/docs-plugin.cjs | 2 +
scripts/i18n/inventory.cjs | 11 +-
scripts/i18n/managed-locales.cjs | 8 +-
scripts/i18n/pr-preflight.cjs | 5 +-
scripts/i18n/simulate-workflow.cjs | 2 +-
scripts/i18n/status-plugin.cjs | 13 ++-
scripts/i18n/verify-multi-locale-fixture.cjs | 68 +++++++++++
scripts/i18n/workflow-publish.cjs | 2 +-
translations/README.md | 30 ++++-
16 files changed, 379 insertions(+), 30 deletions(-)
diff --git a/.github/workflows/build-check.yml
b/.github/workflows/build-check.yml
index e928217d469..3528ea8df34 100644
--- a/.github/workflows/build-check.yml
+++ b/.github/workflows/build-check.yml
@@ -94,7 +94,7 @@ jobs:
yarn
yarn key-features:generate
- - name: Japanese translation unit tests
+ - name: Translation unit tests
env:
NODE_OPTIONS: --max-old-space-size=8192
run: |
@@ -103,6 +103,8 @@ jobs:
node --test scripts/i18n/__tests__/search-input.test.cjs
node --test scripts/i18n/__tests__/search-contexts.test.cjs
node --test scripts/i18n/__tests__/locale-registry.test.cjs
+ node --test scripts/i18n/__tests__/multi-locale.test.cjs
+ node --test scripts/i18n/__tests__/inventory.test.cjs
node --test scripts/i18n/__tests__/planner.test.cjs
node --test
scripts/i18n/__tests__/translation-engine.test.cjs
node --test scripts/i18n/__tests__/workflow.test.cjs
@@ -110,10 +112,13 @@ jobs:
node --test
scripts/i18n/__tests__/translate-document.test.cjs
node --test
scripts/i18n/__tests__/incremental-document.test.cjs
- - name: Japanese translation gate
+ - name: Translation gate for every managed locale
run: |
BASE_SHA=$(git merge-base origin/${{ github.base_ref }} HEAD)
- node scripts/i18n/pr-preflight.cjs --base "$BASE_SHA"
+ MANAGED_LOCALES=$(node scripts/i18n/managed-locales.cjs
--all)
+ for LOCALE in $MANAGED_LOCALES; do
+ I18N_LOCALE="$LOCALE" node scripts/i18n/pr-preflight.cjs
--base "$BASE_SHA"
+ done
- name: Publish artifact optimization tests
env:
diff --git a/.github/workflows/japanese-coverage-audit.yml
b/.github/workflows/japanese-coverage-audit.yml
index 1801c055f34..3719f4ce94e 100644
--- a/.github/workflows/japanese-coverage-audit.yml
+++ b/.github/workflows/japanese-coverage-audit.yml
@@ -23,6 +23,8 @@ permissions:
jobs:
audit:
+ env:
+ I18N_LOCALE: ja
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
diff --git a/.github/workflows/japanese-translation.yml
b/.github/workflows/japanese-translation.yml
index 7c7c9e9f02f..1e5b856030e 100644
--- a/.github/workflows/japanese-translation.yml
+++ b/.github/workflows/japanese-translation.yml
@@ -19,7 +19,8 @@ on:
description: Optional repository-relative source path prefix
type: string
locale:
- description: Optional translation locale; required once several are
managed
+ default: ja
+ description: Configured translation locale (defaults to ja)
type: string
schedule:
- cron: '23 4,16 * * *'
@@ -41,7 +42,7 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 20
env:
- I18N_LOCALE: ${{ inputs.locale }}
+ I18N_LOCALE: ${{ inputs.locale || 'ja' }}
outputs:
can_generate: ${{ steps.scan.outputs.can_generate }}
source_commit: ${{ steps.scan.outputs.source_commit }}
@@ -81,7 +82,7 @@ jobs:
generate:
needs: discover
env:
- I18N_LOCALE: ${{ inputs.locale }}
+ I18N_LOCALE: ${{ inputs.locale || 'ja' }}
if: needs.discover.outputs.can_generate == 'true'
runs-on: ubuntu-latest
timeout-minutes: 120
diff --git a/scripts/i18n/__tests__/fixtures/multi-locale.cjs
b/scripts/i18n/__tests__/fixtures/multi-locale.cjs
new file mode 100644
index 00000000000..18184553dd1
--- /dev/null
+++ b/scripts/i18n/__tests__/fixtures/multi-locale.cjs
@@ -0,0 +1,85 @@
+// Real native metadata shared by the offline tests and the small SEO build.
+const fs = require('node:fs');
+const path = require('node:path');
+const {hash} = require('../../state.cjs');
+const {structure} = require('../../structure.cjs');
+const repoRoot = path.resolve(__dirname, '../../../..');
+
+async function createFixture(root) {
+ const write = (file, content) => {
+ const target = path.join(root, file);
+ fs.mkdirSync(path.dirname(target), {recursive: true});
+ fs.writeFileSync(target, content);
+ };
+ fs.mkdirSync(root, {recursive: true});
+ fs.symlinkSync(path.join(repoRoot, 'node_modules'), path.join(root,
'node_modules'));
+ const settings = locale => ({sourceLocale: 'en', sourceOverrides: [],
+ languageName: locale === 'ja' ? 'Japanese' : 'Korean', scriptPattern:
locale === 'ja' ? '[\\u3040-\\u30ff]' : '[\\uac00-\\ud7af]'});
+ write('translations/config.json', JSON.stringify({schemaVersion: 1,
locales: {ja: settings('ja'), ko: settings('ko')}}));
+ write('versions.json', '["4.x"]');
+ write('versioned_sidebars/version-4.x-sidebars.json', '{}');
+ const records = {ja: [], ko: []};
+ for (const [pluginId, version, docIds] of [
+ ['default', 'current', ['shared', 'ja-only', 'ko-only', 'missing',
'hidden', 'draft']],
+ ['default', '4.x', ['shared']], ['community', 'current', ['shared']],
+ ]) {
+ const sourceRoot = pluginId === 'community' ? 'community' : version
=== 'current' ? 'docs' : `versioned_docs/version-${version}`;
+ const plugin = `docusaurus-plugin-content-docs${pluginId === 'default'
? '' : '-community'}`;
+ const versionDir = version === 'current' ? 'current' :
`version-${version}`;
+ for (const docId of docIds) {
+ const filename = `01-${docId}.md`;
+ const sourcePath = `${sourceRoot}/${filename}`;
+ const source = `---\ntitle: English ${docId}\nlast_update:\n
date: 2026-01-01\nslug: /english-${docId}\n${docId === 'hidden' ? 'unlisted:
true\n' : docId === 'draft' ? 'draft: true\n' : ''}---\n\n## Usage\n\nUse
Doris.\n\n\`\`\`sql\nSELECT 1;\n\`\`\`\n${pluginId === 'default' && version ===
'current' ? '\n[ja-only](./01-ja-only.md)\n\n[ko-only](./01-ko-only.md)\n' :
''}`;
+ write(sourcePath, source);
+ write(`i18n/zh-CN/${plugin}/${versionDir}/${filename}`,
source.replace('English', '中文').replace('/english-', '/chinese-'));
+ for (const locale of ['ja', 'ko']) {
+ if (docId === 'missing') continue;
+ const targetPath =
`i18n/${locale}/${plugin}/${versionDir}/${filename}`;
+ const target = source.replace('English', locale === 'ja' ?
'日本語' : '한국어')
+ .replace('## Usage', locale === 'ja' ? '## 使用方法' : '##
사용법').replace('Use Doris.', locale === 'ja' ? 'Doris を使用します。' : 'Doris를
사용합니다.');
+ write(targetPath, target);
+ const stale = docId === 'ja-only' && locale === 'ko' || docId
=== 'ko-only' && locale === 'ja';
+ const recordedSource = stale ? source.replace('SELECT 1',
'SELECT 0') : source;
+ records[locale].push({pluginId, version, docId, locale,
sourceLocale: 'en', sourcePath, targetPath,
+ sourceHash: hash(recordedSource), targetHash: hash(target),
+ sourceTechnicalHash: (await structure(recordedSource,
'.md')).technicalHash,
+ sourceCommit: 'a'.repeat(40), reviewStatus: 'machine',
ownership: 'machine',
+ translatedAt: '2026-10-18T00:00:00Z', glossaryVersion:
'0'});
+ }
+ }
+ }
+ write('i18n/zh-CN/docusaurus-plugin-content-docs/current/chinese-only.md',
'# Chinese-only source\n');
+ for (const locale of ['ja', 'ko']) {
+ write(`translations/state/${locale}/docs.json`,
JSON.stringify({schemaVersion: 1, pages: records[locale]}));
+
write(`i18n/${locale}/docusaurus-plugin-content-docs/current/orphan.md`, '#
Orphan\n');
+ }
+ write('src/pages/index.tsx', "import React from 'react'; export default
function Home(){return <main>Fixture</main>;}\n");
+ for (const file of ['src/utils/translation.ts',
'src/utils/localization.ts', 'src/utils/locale.ts',
'src/theme/SiteMetadata/index.tsx',
+ 'src/theme/DocItem/Metadata/index.tsx',
'src/theme/DocItem/Content/index.tsx',
+ 'src/theme/DocItem/Layout/TranslationStatus.tsx',
'src/theme/DocItem/Layout/LocalizedNotice.module.css']) {
+ write(file, fs.readFileSync(path.join(repoRoot, file), 'utf8'));
+ }
+ write('src/theme/DocItem/Layout/index.tsx', `import React from 'react';
+import Original from '@theme-original/DocItem/Layout';
+import {useDoc} from '@docusaurus/plugin-content-docs/client';
+import {docTranslation} from '@site/src/utils/translation';
+import TranslationStatus from './TranslationStatus';
+export default function Layout(props) {
+ const state = docTranslation(useDoc().metadata);
+ return <>{state && <TranslationStatus state={state}/>}<Original
{...props}/></>;
+}\n`);
+ write('docusaurus.config.js', `module.exports = {
+ title: 'Multi-locale fixture', url: 'https://doris.apache.org', baseUrl: '/',
trailingSlash: true,
+ i18n: {defaultLocale: 'en', locales: ['en','zh-CN','ja',
...(process.env.FIXTURE_DISABLE_KO ? [] : ['ko'])],
+ localeConfigs: {en: {htmlLang:'en-US'}, 'zh-CN': {htmlLang:'zh-Hans-CN'},
ja:{htmlLang:'ja'}, ko:{htmlLang:'ko'}}},
+ customFields: {translatedLocales: ['ja','ko'], defaultLocalePageLocales:
['ja','ko']},
+ plugins: [[require(${JSON.stringify(path.join(repoRoot,
'scripts/i18n/docs-plugin.cjs'))}), {
+ path:'docs', sidebarPath:false, lastVersion:'4.x',
versions:{current:{path:'dev'},'4.x':{path:'4.x'}}}],
+ [require(${JSON.stringify(path.join(repoRoot,
'scripts/i18n/docs-plugin.cjs'))}), {id:'community', path:'community',
routeBasePath:'community',sidebarPath:false}],
+ require(${JSON.stringify(path.join(repoRoot,
'scripts/i18n/status-plugin.cjs'))}),
+ function fixtureCache(){return
{name:'fixture-cache',configureWebpack:()=>({cache:false})}}],
+ presets: [['classic',{docs:false,blog:false}]],
+};\n`);
+ return {root, write};
+}
+module.exports = {createFixture};
diff --git a/scripts/i18n/__tests__/locale-registry.test.cjs
b/scripts/i18n/__tests__/locale-registry.test.cjs
index 3571711146b..219a4d2211f 100644
--- a/scripts/i18n/__tests__/locale-registry.test.cjs
+++ b/scripts/i18n/__tests__/locale-registry.test.cjs
@@ -61,7 +61,7 @@ test('route groups span every managed locale while a manifest
carries one', asyn
const page = (locale, docId, indexable = true) => ({pluginId: 'default',
version: 'current', docId, locale,
status: 'current', useTarget: true, indexable,
permalink: `/${locale}/docs/current/${docId}/`, englishPermalink:
`/docs/current/${docId}/`,
- alternates: {en: `/docs/current/${docId}/`, ...(indexable ? {[locale]:
`/${locale}/docs/current/${docId}/`} : {})}});
+ alternates: {en: `/docs/current/${docId}/`, 'zh-CN':
`/zh-CN/docs/current/${docId}/`, ...(indexable ? {[locale]:
`/${locale}/docs/current/${docId}/`} : {})}});
const allContent = {'docusaurus-plugin-content-docs': {
default: {translationManifest: {pages: [page('ja', 'a'), page('ja',
'b', false)], reports: [
{kind: 'unsupported_source_route', sourcePath:
'i18n/zh-CN/x.md'},
@@ -75,7 +75,7 @@ test('route groups span every managed locale while a manifest
carries one', asyn
assert.equal(manifest.locale, locale);
// Both locales group the indexable page; the non-indexable one
contributes none.
assert.deepEqual(Object.keys(globalData.alternates).sort(),
- ['/docs/current/a/', '/ja/docs/current/a/']);
+ ['/docs/current/a/', '/ja/docs/current/a/',
'/zh-CN/docs/current/a/']);
assert.deepEqual(manifest.pages.map(item => item.docId), locale ===
'ja' ? ['a', 'b'] : []);
// Repository-wide reports stay, another locale's records do not.
assert.deepEqual(manifest.reports.filter(report => report.kind ===
'unsupported_source_route').length, 1);
diff --git a/scripts/i18n/__tests__/multi-locale.test.cjs
b/scripts/i18n/__tests__/multi-locale.test.cjs
new file mode 100644
index 00000000000..153c215919e
--- /dev/null
+++ b/scripts/i18n/__tests__/multi-locale.test.cjs
@@ -0,0 +1,141 @@
+const test = require('node:test');
+const assert = require('node:assert/strict');
+const fs = require('node:fs');
+const os = require('node:os');
+const path = require('node:path');
+const {execFileSync, spawnSync} = require('node:child_process');
+const yaml = require('js-yaml');
+const {pipelineLocale, managedLocales, identity} = require('../state.cjs');
+const {collect, buildGroup} = require('../inventory.cjs');
+const statusPlugin = require('../status-plugin.cjs');
+const {createFixture} = require('./fixtures/multi-locale.cjs');
+const repoRoot = path.resolve(__dirname, '../../..');
+
+function directory(t) {
+ const root = fs.mkdtempSync(path.join(os.tmpdir(), 'doris-multi-locale-'));
+ t.after(() => fs.rmSync(root, {recursive: true, force: true}));
+ return root;
+}
+async function withEnv(values, run) {
+ const old = {...process.env};
+ Object.entries(values).forEach(([key, value]) => value === undefined ?
delete process.env[key] : process.env[key] = value);
+ try {return await run();}
+ finally {
+ for (const key of Object.keys(process.env)) if (!(key in old)) delete
process.env[key];
+ Object.assign(process.env, old);
+ }
+}
+
+test('two managed locales require a configured selector even when the site
disables them', async t => {
+ const {root} = await createFixture(directory(t));
+ assert.throws(() => pipelineLocale(root, ''), /Set I18N_LOCALE/);
+ assert.throws(() => pipelineLocale(root, 'fr'), /Unconfigured translation
locale/);
+ for (const locale of ['ja', 'ko']) assert.equal(pipelineLocale(root,
locale), locale);
+ assert.deepEqual(managedLocales(root), ['ja', 'ko']);
+ const cli = all => execFileSync(process.execPath, [path.join(repoRoot,
'scripts/i18n/managed-locales.cjs'), ...all],
+ {cwd: root, encoding: 'utf8', env: {...process.env, DORIS_ENABLE_JA:
'', DORIS_ENABLE_LOCALES: ''}});
+ assert.equal(cli([]), '');
+ assert.equal(cli(['--all']), 'ja ko');
+});
+
+test('native inventory isolates locales across repeated doc IDs, plugins,
versions and reports', async t => {
+ const {root} = await createFixture(directory(t));
+ for (const locale of ['ja', 'ko']) await withEnv({I18N_LOCALE: locale},
async () => {
+ const native = await collect(root);
+ assert.equal(native.pages.length, 8);
+ assert(native.pages.every(page => page.locale === locale));
+ assert.equal(new Set(native.pages.map(identity)).size, 8);
+ assert.equal(native.pages.filter(page => page.docId ===
'shared').length, 3);
+ assert.deepEqual(native.versions.map(({pluginId, version}) =>
[pluginId, version]),
+ [['default', 'current'], ['default', '4.x'], ['community',
'current']]);
+ assert(native.reports.every(report => !report.locale || report.locale
=== locale));
+ assert.equal(native.reports.filter(report => report.kind ===
'unsupported_source_route').length, 1);
+ assert.equal(native.reports.filter(report => report.kind ===
'orphan_target').length, 1);
+ const group = {id: 'DEV', pluginId: 'default', version: 'current',
sourceRoot: 'docs'};
+ const result = buildGroup({group, pages: [...native.pages,
{...native.pages[0], locale: 'fr'}],
+ files: native.pages.filter(page => page.pluginId === 'default' &&
page.version === 'current').map(page => page.englishPath),
+ scope: {locale, groups: [group], retainEnglish: [], maxBatchPages:
10}, order: new Map(),
+ committedHash: () => null, committedRecords: new Map()});
+ assert.equal(result.counts.registered, 6);
+ assert.equal(result.counts.required, 5);
+ assert.equal(result.counts.drafts, 1);
+ const shared = native.pages.find(page => page.docId === 'shared' &&
page.version === 'current' && page.pluginId === 'default');
+ assert.deepEqual(shared.alternates, {en: '/docs/dev/english-shared/',
'zh-CN': '/zh-CN/docs/dev/chinese-shared/',
+ ja: '/ja/docs/dev/english-shared/', ko:
'/ko/docs/dev/english-shared/'});
+ for (const [id, expected] of [['ja-only', ['en', 'zh-CN', 'ja']],
['ko-only', ['en', 'zh-CN', 'ko']],
+ ['missing', ['en', 'zh-CN']], ['hidden', ['en', 'zh-CN']],
['draft', ['en']]]) {
+ assert.deepEqual(Object.keys(native.pages.find(page => page.docId
=== id).alternates), expected, id);
+ }
+ });
+});
+
+test('native doc metadata, browser groups and per-locale manifests agree,
including disabled ko', async t => {
+ const {loadContext} = require('@docusaurus/core/lib/server/site.js');
+ const {loadPluginConfigs} =
require('@docusaurus/core/lib/server/plugins/configs.js');
+ const {initPluginsConfigs} =
require('@docusaurus/core/lib/server/plugins/init.js');
+ for (const disabled of [false, true]) {
+ const {root} = await createFixture(directory(t));
+ await withEnv({NODE_ENV: 'production', FIXTURE_DISABLE_KO: disabled ?
'true' : undefined}, async () => {
+ const locales = ['en', 'zh-CN', 'ja', ...(disabled ? [] : ['ko'])];
+ for (const locale of locales) {
+ const context = await loadContext({siteDir: root, locale});
+ const configs = (await
loadPluginConfigs(context)).filter(config =>
config.plugin.isTranslationDocsPlugin);
+ const plugins = (await initPluginsConfigs(context,
configs)).map(result => result.plugin);
+ const allContent = {'docusaurus-plugin-content-docs': {}};
+ for (const plugin of plugins)
allContent['docusaurus-plugin-content-docs'][plugin.options.id] = await
plugin.loadContent();
+ let data;
+ await statusPlugin(context).allContentLoaded({allContent,
actions: {setGlobalData: value => {data = value;}}});
+ const pages =
Object.values(allContent['docusaurus-plugin-content-docs']).flatMap(content =>
content.translationManifest.pages);
+ const shared = pages.find(page => page.docId === 'shared' &&
page.version === 'current' && page.pluginId === 'default');
+ assert.deepEqual(Object.keys(shared.alternates), locales);
+ for (const permalink of Object.values(shared.alternates))
assert.deepEqual(data.alternates[permalink], shared.alternates);
+ for (const page of pages)
assert.deepEqual(data.alternates[page.permalink], page.alternates);
+ const output = JSON.parse(fs.readFileSync(path.join(root,
'.i18n-build/manifests', `${locale}.json`)));
+ assert(output.pages.every(page => page.locale === locale));
+ assert(output.reports.every(report => !report.locale ||
report.locale === locale));
+ if (['ja', 'ko'].includes(locale)) {
+ const metadata =
allContent['docusaurus-plugin-content-docs'].default.loadedVersions
+ .find(version => version.versionName ===
'current').docs.find(doc => doc.id === 'shared').translation;
+ assert.deepEqual(metadata.alternates, shared.alternates);
+ assert.deepEqual(output.pages.find(page => page.docId ===
'shared' && page.version === 'current' && page.pluginId ===
'default').alternates,
+ shared.alternates);
+ }
+ if (disabled)
assert(!JSON.stringify(data.alternates).includes('/ko/'));
+ }
+ });
+ }
+});
+
+test('scheduled Japanese jobs select ja and PR admission checks every
configured locale with fail-fast behavior', async t => {
+ const workflow = file => yaml.load(fs.readFileSync(path.join(repoRoot,
'.github/workflows', file), 'utf8'));
+ const translation = workflow('japanese-translation.yml');
+ assert.equal(translation.on.workflow_dispatch.inputs.locale.default, 'ja');
+ for (const job of ['discover', 'generate'])
assert.equal(translation.jobs[job].env.I18N_LOCALE, "${{ inputs.locale || 'ja'
}}");
+
assert.equal(workflow('japanese-coverage-audit.yml').jobs.audit.env.I18N_LOCALE,
'ja');
+ const gate =
workflow('build-check.yml').jobs['build-and-deploy'].steps.find(step =>
step.name === 'Translation gate for every managed locale');
+ const {root, write} = await createFixture(directory(t));
+ for (const file of ['scripts/i18n/managed-locales.cjs',
'scripts/i18n/state.cjs', 'config/i18n.js']) write(file,
fs.readFileSync(path.join(repoRoot, file), 'utf8'));
+ write('scripts/i18n/pr-preflight.cjs',
"require('node:fs').appendFileSync('calls', process.env.I18N_LOCALE + '\\n');
process.exit(process.env.I18N_LOCALE === process.env.FAIL_LOCALE ? 1 : 0);\n");
+ execFileSync('git', ['init', '-q'], {cwd: root});
+ execFileSync('git', ['-c', 'user.name=Test', '-c',
'[email protected]', 'commit', '--allow-empty', '-qm', 'base'], {cwd:
root});
+ execFileSync('git', ['update-ref', 'refs/remotes/origin/master', 'HEAD'],
{cwd: root});
+ for (const fail of ['', 'ko', 'ja']) {
+ write('calls', '');
+ const result = spawnSync('bash', ['-e', '-c', gate.run.replace('${{
github.base_ref }}', 'master')],
+ {cwd: root, encoding: 'utf8', env: {...process.env, FAIL_LOCALE:
fail, DORIS_ENABLE_JA: '', DORIS_ENABLE_LOCALES: ''}});
+ assert.equal(result.status, fail ? 1 : 0, result.stderr);
+ assert.equal(fs.readFileSync(path.join(root, 'calls'), 'utf8'), fail
=== 'ja' ? 'ja\n' : 'ja\nko\n');
+ }
+});
+
+test('the publisher-selected locale is forwarded to the admission subprocess',
async t => {
+ const {preflight} = require('../pr-preflight.cjs');
+ const {root, write} = await createFixture(directory(t));
+ write('scripts/i18n/check-translations.cjs',
"require('node:fs').writeFileSync('selected-locale',
process.env.I18N_LOCALE);\n");
+ execFileSync('git', ['init', '-q'], {cwd: root});
+ execFileSync('git', ['-c', 'user.name=Test', '-c',
'[email protected]', 'commit', '--allow-empty', '-qm', 'base'], {cwd:
root});
+ for (const locale of ['ja', 'ko']) {
+ await withEnv({I18N_LOCALE: 'unconfigured'}, () => preflight({root,
base: 'HEAD', locale}));
+ assert.equal(fs.readFileSync(path.join(root, 'selected-locale'),
'utf8'), locale);
+ }
+});
diff --git a/scripts/i18n/alternates.cjs b/scripts/i18n/alternates.cjs
new file mode 100644
index 00000000000..66ab931ff07
--- /dev/null
+++ b/scripts/i18n/alternates.cjs
@@ -0,0 +1,18 @@
+// A native document is identified by plugin, version and doc ID. Each managed
+// locale contributes only its own indexable route to that document's group.
+// Use the same complete group in doc metadata, manifests and browser data.
+function mergeAlternates(pages) {
+ const groups = new Map();
+ const key = page => JSON.stringify([page.pluginId, page.version,
page.docId]);
+ for (const page of pages) {
+ const id = key(page);
+ if (!groups.has(id)) groups.set(id, {});
+ const group = groups.get(id);
+ for (const locale of ['en', 'zh-CN']) {
+ if (page.alternates[locale]) group[locale] =
page.alternates[locale];
+ }
+ if (page.indexable && page.alternates[page.locale]) group[page.locale]
= page.alternates[page.locale];
+ }
+ for (const page of pages) page.alternates = groups.get(key(page));
+}
+module.exports = {mergeAlternates};
diff --git a/scripts/i18n/docs-plugin.cjs b/scripts/i18n/docs-plugin.cjs
index da743e6aacf..99bcd84d0ef 100644
--- a/scripts/i18n/docs-plugin.cjs
+++ b/scripts/i18n/docs-plugin.cjs
@@ -6,6 +6,7 @@ const {processDocMetadata} =
require('@docusaurus/plugin-content-docs/lib/docs.j
const {Globby, getPluginI18nPath, normalizeUrl, readLastUpdateData,
aliasedSitePath, resolveMarkdownLinkPathname} = require('@docusaurus/utils');
const {applyTrailingSlash} = require('@docusaurus/utils-common');
const {hash, relative, identity, loadConfig, managedLocales, loadRecords,
sourceLocaleFor, decide} = require('./state.cjs');
+const {mergeAlternates} = require('./alternates.cjs');
const {structure, alignHeadings} = require('./structure.cjs');
async function exists(file) { try { await fs.access(file); return true; }
catch { return false; } }
@@ -157,6 +158,7 @@ module.exports = async function
translationDocsPlugin(context, options) {
sourcePath: relative(context.siteDir,
path.join(versions['zh-CN'][index].contentPathLocalized, target))});
}
}
+ mergeAlternates(pages);
const pluginContext = activeLocale ? {...context, localizationDir:
viewRoot(activeLocale)} : context;
const plugin = await native.default(pluginContext, options);
const byId = new Map(pages.map(page => [identity(page), page]));
diff --git a/scripts/i18n/inventory.cjs b/scripts/i18n/inventory.cjs
index 4a5b3cd0a30..3ebbf72a86e 100644
--- a/scripts/i18n/inventory.cjs
+++ b/scripts/i18n/inventory.cjs
@@ -57,7 +57,7 @@ function mergedCurrent(page, committedHash) {
function buildGroup({group, pages, files, order, scope, committedHash,
committedRecords}) {
const key = keyFor(scope.locale);
- const relevant = pages.filter(page => page.pluginId === group.pluginId &&
page.version === group.version);
+ const relevant = pages.filter(page => page.pluginId === group.pluginId &&
page.version === group.version && page.locale === scope.locale);
assert(relevant.length, `Missing native plugin/version: ${group.id}`);
const identities = new Set();
const routes = new Set();
@@ -147,12 +147,13 @@ async function collect(root) {
for (const plugin of plugins) {
const content = await plugin.loadContent();
assert(content.translationManifest, 'Missing T02 native translation
manifest');
- pages.push(...content.translationManifest.pages);
- reports.push(...content.translationManifest.reports);
+ const selectedPages = content.translationManifest.pages.filter(page =>
page.locale === locale);
+ pages.push(...selectedPages);
+ reports.push(...content.translationManifest.reports.filter(report =>
!report.locale || report.locale === locale));
for (const version of content.loadedVersions) {
versions.push({pluginId: plugin.options.id, version:
version.versionName, order: sidebarOrder(version.sidebars)});
const registered = new Map([...version.docs,
...version.drafts].map(doc => [doc.id, doc]));
- for (const page of content.translationManifest.pages.filter(page
=> page.version === version.versionName)) {
+ for (const page of selectedPages.filter(page => page.version ===
version.versionName)) {
const doc = registered.get(page.docId);
assert(doc && applyTrailingSlash(doc.permalink,
{trailingSlash: context.siteConfig.trailingSlash, baseUrl: context.baseUrl})
=== page.englishPermalink, `Native route mismatch:
${key(page)}`);
@@ -162,7 +163,7 @@ async function collect(root) {
}
labels.push({pluginId: plugin.options.id, files: await
plugin.getTranslationFiles({content})});
}
- return {pages, reports, versions, labels};
+ return {pages, reports: [...new Map(reports.map(report =>
[JSON.stringify(report), report])).values()], versions, labels};
}
function renderBatches(snapshot) {
diff --git a/scripts/i18n/managed-locales.cjs b/scripts/i18n/managed-locales.cjs
index a676cfdf0f8..79d9991bf91 100644
--- a/scripts/i18n/managed-locales.cjs
+++ b/scripts/i18n/managed-locales.cjs
@@ -1,8 +1,6 @@
-// Print the locales whose translation state this checkout manages and whose
-// build is enabled, so a workflow can verify every one of them without naming
-// a language. The site configuration decides what a build serves; the
-// translation configuration decides what state exists.
+// Publication verifies enabled locales; PR admission must also check state for
+// disabled locales. --all selects the translation configuration alone.
const {enabledLocales} = require('../../config/i18n');
const {managedLocales} = require('./state.cjs');
-process.stdout.write(managedLocales(process.cwd(), enabledLocales()).join('
'));
+process.stdout.write(managedLocales(process.cwd(),
process.argv.includes('--all') ? undefined : enabledLocales()).join(' '));
diff --git a/scripts/i18n/pr-preflight.cjs b/scripts/i18n/pr-preflight.cjs
index 1703c2fdb6c..6f0d1ed6439 100644
--- a/scripts/i18n/pr-preflight.cjs
+++ b/scripts/i18n/pr-preflight.cjs
@@ -3,11 +3,12 @@
const fs = require('node:fs');
const path = require('node:path');
const {execFileSync} = require('node:child_process');
+const {pipelineLocale} = require('./state.cjs');
const {checkLinks, checkSyntax} = require('./document-checks.cjs');
-async function preflight({root, base}) {
+async function preflight({root, base, locale = pipelineLocale(root)}) {
execFileSync(process.execPath, [path.join(root,
'scripts/i18n/check-translations.cjs'), '--base', base],
- {cwd: root, stdio: 'inherit'});
+ {cwd: root, stdio: 'inherit', env: {...process.env, I18N_LOCALE:
locale}});
const files = execFileSync('git', ['diff', '--name-only',
'--diff-filter=AM', '-z', base, 'HEAD'],
{cwd: root, encoding: 'utf8'}).split('\0').filter(Boolean)
.filter(file =>
/^i18n\/[^/]+\/docusaurus-plugin-content-docs[^/]*\/.+\.mdx?$/.test(file));
diff --git a/scripts/i18n/simulate-workflow.cjs
b/scripts/i18n/simulate-workflow.cjs
index 234e834586f..6695c7d9335 100644
--- a/scripts/i18n/simulate-workflow.cjs
+++ b/scripts/i18n/simulate-workflow.cjs
@@ -171,7 +171,7 @@ async function main() {
writeExecutable(path.join(bin, 'codex-invalid'), codexMock('invalid',
repo));
const env = {...process.env, PATH: `${bin}:${process.env.PATH}`, GH_TOKEN:
'local-simulation',
GITHUB_REPOSITORY: 'apache/doris-website', GITHUB_EVENT_NAME:
'workflow_dispatch',
- GITHUB_RUN_ID: runId, I18N_PR_AUTH_KIND: 'pat', I18N_MODE: pages === 1
? 'validate' : 'generate',
+ GITHUB_RUN_ID: runId, I18N_LOCALE: 'ja', I18N_PR_AUTH_KIND: 'pat',
I18N_MODE: pages === 1 ? 'validate' : 'generate',
I18N_PATH: selectedPath, I18N_MAX_PAGES: String(pages),
I18N_LIMIT: '20', DORIS_ENABLE_JA: 'true', CODEX_HOME: auth,
I18N_SIM_PR_RECORD: path.join(output, 'pull-request.json')};
diff --git a/scripts/i18n/status-plugin.cjs b/scripts/i18n/status-plugin.cjs
index 3da3238ec54..ca92fb057cb 100644
--- a/scripts/i18n/status-plugin.cjs
+++ b/scripts/i18n/status-plugin.cjs
@@ -1,6 +1,7 @@
const fs = require('node:fs/promises');
const path = require('node:path');
const {managedLocales} = require('./state.cjs');
+const {mergeAlternates} = require('./alternates.cjs');
module.exports = function translationStatusPlugin(context) {
const locale = context.i18n.currentLocale;
@@ -15,14 +16,14 @@ module.exports = function translationStatusPlugin(context) {
// route groups need all of them: an English page must advertise
the
// localized alternate just as the localized page advertises
English.
const allPages = manifests.flatMap(manifest => manifest.pages);
- // Only current translations need a cross-locale route group in the
- // browser. Missing/stale pages use the existing en/zh behavior
with
- // this locale removed; their detailed status lives in the doc's
own
- // metadata.
+ mergeAlternates(allPages);
+ // Keep translated groups and source routes that prefix replacement
+ // cannot infer, including groups whose targets all fall back.
const alternates = {};
for (const page of allPages) {
- if (page.alternates[page.locale]) {
- for (const permalink of Object.values(page.alternates))
alternates[permalink] = page.alternates;
+ const sourceRoutesDiffer =
page.alternates['zh-CN']?.replace('/zh-CN/', '/') !== page.englishPermalink;
+ if (sourceRoutesDiffer ||
Object.keys(page.alternates).some(lang => !['en', 'zh-CN'].includes(lang))) {
+ for (const permalink of [page.permalink,
...Object.values(page.alternates)]) alternates[permalink] = page.alternates;
}
}
// The manifest describes one locale, so it carries only its pages.
diff --git a/scripts/i18n/verify-multi-locale-fixture.cjs
b/scripts/i18n/verify-multi-locale-fixture.cjs
new file mode 100644
index 00000000000..5e7289c7545
--- /dev/null
+++ b/scripts/i18n/verify-multi-locale-fixture.cjs
@@ -0,0 +1,68 @@
+// Small real builds: no production ko registration or full-version site build.
+const assert = require('node:assert/strict');
+const fs = require('node:fs/promises');
+const path = require('node:path');
+const {spawnSync} = require('node:child_process');
+const {createFixture} = require('./__tests__/fixtures/multi-locale.cjs');
+const {verify} = require('./verify-static.cjs');
+const root = path.resolve(__dirname, '../..');
+
+async function verifyBrowser(site, locales) {
+ const http = require('node:http');
+ const {chromium} = require('playwright');
+ const server = http.createServer(async (request, response) => {
+ try {
+ const pathname = decodeURIComponent(new URL(request.url,
'http://local').pathname);
+ const file = path.join(site, pathname, path.extname(pathname) ? ''
: 'index.html');
+ response.setHeader('Content-Type',
{'.js':'application/javascript', '.css':'text/css', '.html':'text/html;
charset=utf-8'}[path.extname(file)] ?? 'application/octet-stream');
+ response.end(await fs.readFile(file));
+ } catch {response.writeHead(404); response.end('Not found');}
+ });
+ await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));
+ const browser = await chromium.launch({headless: true});
+ try {
+ const page = await browser.newPage();
+ const errors = [];
+ page.on('pageerror', error => errors.push(error.message));
+ const origin = `http://127.0.0.1:${server.address().port}`;
+ for (const locale of locales) {
+ await page.goto(origin +
`/${locale}/docs/dev/english-shared/?fixture=1#usage`, {waitUntil:
'networkidle'});
+ assert.equal(await
page.locator('.theme-doc-markdown').getAttribute('lang'), locale);
+ for (const lang of ['en-US', 'zh-Hans-CN', ...locales])
assert.equal(await
page.locator(`link[rel="alternate"][hreflang="${lang}"]`).count(), 1);
+ const other = locale === 'ja' ? 'ko-only' : 'ja-only';
+ await page.locator(`.theme-doc-markdown
a[href="/${locale}/docs/dev/english-${other}/"]`).click();
+ await
page.waitForSelector('[data-translation-status="stale_technical"]');
+ assert.equal(await
page.locator('.theme-doc-markdown').getAttribute('lang'), 'en');
+ assert.equal(await
page.locator(`link[rel="alternate"][hreflang="${locale}"]`).count(), 0);
+ assert.equal(await
page.locator('link[rel="alternate"][hreflang="zh-Hans-CN"]').getAttribute('href'),
+ `https://doris.apache.org/zh-CN/docs/dev/chinese-${other}/`);
+ await page.goBack({waitUntil: 'networkidle'});
+ assert.equal(await
page.locator('.theme-doc-markdown').getAttribute('lang'), locale);
+ await page.reload({waitUntil: 'networkidle'});
+ assert.equal(await
page.locator('.theme-doc-markdown').getAttribute('lang'), locale);
+ }
+ assert.deepEqual(errors, []);
+ return {passed: true, locales, checks: ['hydration', 'complete
alternates', 'SPA stale fallback', 'Chinese slug', 'back', 'refresh']};
+ } finally {await browser.close(); await new Promise(resolve =>
server.close(resolve));}
+}
+
+async function main() {
+ const results = [];
+ for (const disabled of [false, true]) {
+ const fixture = path.join(root, '.i18n-build', disabled ?
'multi-locale-disabled-fixture' : 'multi-locale-fixture');
+ await fs.rm(fixture, {recursive: true, force: true});
+ await createFixture(fixture);
+ const result = spawnSync(process.execPath, [path.join(root,
'scripts/build-locales.cjs'), '--site-dir', fixture],
+ {stdio: 'inherit', env: {...process.env, FIXTURE_DISABLE_KO:
disabled ? 'true' : '', NODE_OPTIONS: '--max-old-space-size=4096'}});
+ assert.equal(result.status, 0, 'Multi-locale fixture build failed');
+ const locales = disabled ? ['ja'] : ['ja', 'ko'];
+ const site = path.join(fixture, 'build');
+ const verification = Object.fromEntries(locales.map(locale => [locale,
verify({site,
+ manifest: path.join(fixture, '.i18n-build/manifests',
`${locale}.json`)})]));
+ const browser = process.argv.includes('--browser') ? await
verifyBrowser(site, locales) : undefined;
+ results.push({fixture, disabled, verification, browser});
+ }
+ await fs.writeFile(path.join(root,
'.i18n-build/multi-locale-verification.json'), JSON.stringify(results, null, 2)
+ '\n');
+ console.log(JSON.stringify(results, null, 2));
+}
+if (require.main === module) main().catch(error => {console.error(error);
process.exitCode = 1;});
diff --git a/scripts/i18n/workflow-publish.cjs
b/scripts/i18n/workflow-publish.cjs
index 537a74dbef3..2a3fc7bdad6 100644
--- a/scripts/i18n/workflow-publish.cjs
+++ b/scripts/i18n/workflow-publish.cjs
@@ -73,7 +73,7 @@ async function main() {
const bodyFile = path.join(output, 'pr-body.md');
const body = fs.readFileSync(bodyFile, 'utf8');
assert(body.includes('bot-head=PENDING'), 'Missing PR ownership marker');
- await require('./pr-preflight.cjs').preflight({root, base:
manifest.sourceCommit});
+ await require('./pr-preflight.cjs').preflight({root, base:
manifest.sourceCommit, locale});
fs.writeFileSync(bodyFile, body.replace('bot-head=PENDING',
`bot-head=${newHead}`)
+ `\nShared PR admission checks: passed (translation state, syntax,
internal links).\n`);
const remoteMaster = git('ls-remote', 'origin',
'refs/heads/master').split(/\s+/)[0];
diff --git a/translations/README.md b/translations/README.md
index 8d5ff6880e8..9412bc0a3f7 100644
--- a/translations/README.md
+++ b/translations/README.md
@@ -226,8 +226,34 @@ reintroduced page returns to normal native enumeration.
The translation pipeline works on one locale per run. `scripts/i18n/state.cjs`
exposes `pipelineLocale(root)`: a configuration that declares a single locale
needs no selector and behaves exactly as before, and `I18N_LOCALE` picks one as
-soon as several are managed. The two discovery workflows take a `locale` input
-and pass it through as that variable.
+soon as several are managed. An absent or unconfigured selector then fails
+before discovery or generation. The Japanese translation workflow defaults its
+manual input and scheduled runs to `ja`; the Japanese coverage audit always
+selects `ja`. A manual translation run can select another configured locale.
+
+Build Check runs PR admission once per configured translation locale, including
+locales disabled for the site build:
+
+```bash
+MANAGED_LOCALES=$(node scripts/i18n/managed-locales.cjs --all)
+for LOCALE in $MANAGED_LOCALES; do
+ I18N_LOCALE="$LOCALE" node scripts/i18n/pr-preflight.cjs --base "$BASE_SHA"
+done
+```
+
+Without `--all`, `managed-locales.cjs` lists only enabled managed locales for
+publication checks. The publisher passes its selected locale to PR admission.
+Native inventory returns only that selected locale's pages and reports, plus
+repository-wide reports and all configured plugin/version metadata. SEO groups
+merge indexable translations by native plugin, version and doc ID, preserving
+actual source-language slugs and excluding stale or disabled translations.
+
+Run the offline native-metadata regression with
+`node --test scripts/i18n/__tests__/multi-locale.test.cjs`. For small real
builds
+with both managed locales and with Korean disabled, run
+`node scripts/i18n/verify-multi-locale-fixture.cjs`; add `--browser` when
+Playwright and Chromium are installed. These fixtures declare Korean locally
+without registering or enabling it on the website.
Each locale also declares, in `translations/config.json`:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
