This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 0b96854eb9b [feat](i18n) Add Japanese document gates and search (#4210)
0b96854eb9b is described below
commit 0b96854eb9bb0b1638e5bf560d43831b7a296f3e
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Tue Oct 13 03:31:42 2026 +0800
[feat](i18n) Add Japanese document gates and search (#4210)
Japanese document PRs now receive source-hash, version,
protected-content, link, and real `/ja/` static-route checks while
English-source changes report stale Japanese content without blocking
the source PR. A scheduled and manually runnable audit recounts the
current native Dev, 4.x, and community scope, including sidebar labels;
the T03 inventory remains a comparison baseline rather than an
allowlist. The existing required Node dead-link gate also resolves
Japanese relative links through the same-version English fallback; its
original CI failure is covered by a regression test.
Japanese search indexes only current translated documents in those three
groups. The SearchBar and SearchPage use the active locale index,
distinguish Dev, 4.x, and community results, handle Japanese text and
SQL identifiers, and link empty results to the matching English search.
Apache rewrite rules retain the existing English/Chinese Dev redirects
while returning locale-appropriate 404 responses for missing Japanese
paths. Two existing Japanese overview links are corrected and their
target hashes updated.
Validation: 37 i18n/build tests, 37 governance tests, 2 Python link
tests, a three-language Dev build, 1,778 static status checks, 1,690
Dev/community navigation routes, 26 browser pages with 63 checks against
a freshly built T03 English/Chinese baseline, Japanese
SearchPage/SearchBar and mobile search tests, and real Apache 301/404
checks. The local full TypeScript check still reports existing
repository diagnostics; no TypeScript source is changed. GitHub CI will
validate Dev and 4.x together before this PR is ready to merge.
Japanese publication and automatic translation remain disabled. No model
calls or deployment are part of this PR.
---------
Co-authored-by: morningman <[email protected]>
---
.github/workflows/build-check.yml | 316 ++-------------------
.github/workflows/check-md-links.yml | 6 +-
.github/workflows/japanese-coverage-audit.yml | 49 ++++
.github/workflows/japanese-documents.yml | 14 +-
config/i18n.js | 4 +
docusaurus.config.js | 6 +-
i18n/ja/code.json | 20 +-
.../getting-started/what-is-apache-doris.md | 2 +-
.../getting-started/what-is-apache-doris.md | 2 +-
scripts/__tests__/test_japanese_links.py | 43 +++
scripts/build-locales.cjs | 2 +-
scripts/check_md_links_single.py | 29 +-
scripts/check_move.js | 27 +-
scripts/docs-governance/lint-links.js | 14 +-
scripts/docs-governance/manifest.js | 34 ++-
scripts/i18n/__tests__/t04.test.cjs | 128 +++++++++
scripts/i18n/audit-coverage.cjs | 122 ++++++++
scripts/i18n/build-scope.cjs | 61 ++++
scripts/i18n/check-translations.cjs | 82 ++++++
scripts/i18n/inventory.cjs | 2 +-
scripts/i18n/paths.cjs | 10 +
scripts/i18n/protection.cjs | 32 +++
scripts/i18n/search-plugin.cjs | 102 +++++++
scripts/i18n/structure.cjs | 2 +-
scripts/i18n/verify-search.cjs | 98 +++++++
src/theme/SearchBar/SearchBar.jsx | 12 +-
src/theme/SearchPage/SearchPage.jsx | 10 +-
src/utils/searchInput.js | 16 ++
static/.htaccess | 17 +-
translations/protection-exceptions/ja.json | 43 +++
translations/scope/README.md | 19 ++
translations/scope/ja.json | 205 ++++++++++++-
translations/state/ja/docs.json | 4 +-
33 files changed, 1179 insertions(+), 354 deletions(-)
diff --git a/.github/workflows/build-check.yml
b/.github/workflows/build-check.yml
index 773158e8d52..f3c6afaf408 100644
--- a/.github/workflows/build-check.yml
+++ b/.github/workflows/build-check.yml
@@ -74,307 +74,28 @@ jobs:
id: detect
run: |
BASE_SHA=$(git merge-base origin/${{ github.base_ref }} HEAD)
- CHANGED_FILES=$(git diff --name-only "$BASE_SHA" HEAD)
+ node scripts/i18n/build-scope.cjs "$BASE_SHA"
- echo "=== Changed files ==="
- echo "$CHANGED_FILES"
-
- #
---------------------------------------------------------------
- # The build scope is three independent questions. Answering
them
- # separately is what keeps a website-only PR from paying for a
- # 4-version x 2-locale build (~1h, and enough memory pressure
to
- # kill the runner outright).
- #
- # TIER how much has to be compiled at all
- # skip nothing that reaches the bundler changed
- # build compile the version set computed below
- # full the *set of versions itself* changed, so every
- # active version has to be re-validated
- # VERSIONS which doc versions have content changes
- # LOCALES which locales have *content* changes
- #
- # Site code (src/**, docusaurus.config.js, deps) deliberately
does
- # NOT force `full`: the React/TS/SCSS compile is identical for
- # every doc version, so one representative version catches
exactly
- # the same errors at a quarter of the cost. Only versions.json
and
- # config/versions-plugin.js — the files that define which
versions
- # exist — earn a full build.
- #
- # Unrecognised paths fall through to `build` (minimal, but
real),
- # so a new kind of file is never silently skipped.
- #
---------------------------------------------------------------
-
- VERSIONS=""
- LOCALES=""
- TIER="skip"
- JSON_TO_VALIDATE=""
-
- # Representative version for compile-only checks: the latest
- # released version, i.e. the stable one the navbar defaults to.
- PROBE_VERSION=$(jq -r '.[0] // "4.x"' versions.json)
-
- # skip < build < full; never lower an already-raised tier.
- raise_tier() {
- case "$1" in
- full)
- TIER="full"
- ;;
- build)
- if [ "$TIER" = "skip" ]; then TIER="build"; fi
- ;;
- esac
- }
-
- add_version() {
- VERSIONS="$1,$VERSIONS"
- raise_tier build
- }
-
- add_locale() {
- case ",$LOCALES," in
- *",$1,"*) ;;
- *) LOCALES="${LOCALES:+$LOCALES,}$1" ;;
- esac
- raise_tier build
- }
-
- # Translation JSON only needs a syntax check (see below).
- validate_json_later() {
- JSON_TO_VALIDATE="${JSON_TO_VALIDATE}${1}"$'\n'
- }
-
- # First match wins, so order matters: the most specific paths
and
- # the version-defining files come before the broad catch-alls.
- while IFS= read -r file; do
- [ -n "$file" ] || continue
- case "$file" in
- # ---- Files that define which versions exist
----------
- # These change the version set itself, so every active
- # version has to be rebuilt to stay honest.
- versions.json|config/versions-plugin.js)
- raise_tier full
- ;;
-
- # ---- English doc content
-----------------------------
- # Dev / unreleased version.
- docs/*)
- add_locale en
- add_version current
- ;;
- versioned_docs/version-*/*)
- ver=$(echo "$file" | sed -n
's|versioned_docs/version-\([^/]*\)/.*|\1|p')
- add_locale en
- add_version "$ver"
- ;;
-
- # ---- Chinese doc content
-----------------------------
- # Only zh-CN is built: an en page cannot break because
a
- # zh-CN page changed. With a single --locale,
Docusaurus
- # drops the locale path prefix (baseUrl becomes /
instead
- # of /zh-CN/) — the same path `yarn start:zh-CN` takes.
- # That only shifts output paths, and onBrokenLinks is
- # 'warn', so it cannot manufacture a failure.
- i18n/zh-CN/docusaurus-plugin-content-docs/current/*)
- add_locale zh-CN
- add_version current
- ;;
- i18n/zh-CN/docusaurus-plugin-content-docs/version-*/*)
- ver=$(echo "$file" | sed -n
's|i18n/zh-CN/docusaurus-plugin-content-docs/version-\([^/]*\)/.*|\1|p')
- add_locale zh-CN
- add_version "$ver"
- ;;
-
- # ---- Translation tables (JSON)
-----------------------
- # code.json and the per-version label files are lookup
- # tables, not content: Docusaurus falls back to the
- # source string and warns on a missing/stale key, it
does
- # not fail. The single build-breaking failure mode is
- # malformed JSON, which `jq` catches in milliseconds —
so
- # these no longer drag a whole zh-CN site build along.
-
i18n/*/code.json|i18n/*/docusaurus-plugin-content-docs*/*.json)
- validate_json_later "$file"
- ;;
-
- # ---- Version-scoped sidebars
-------------------------
- # e.g. versioned_sidebars/version-4.x-sidebars.json →
4.x
- versioned_sidebars/version-*-sidebars.json)
- ver=$(echo "$file" | sed -n
's|versioned_sidebars/version-\(.*\)-sidebars\.json|\1|p')
- add_locale en
- add_version "$ver"
- ;;
- sidebars.ts)
- add_locale en
- add_version current
- ;;
-
- # ---- Always-built, version-independent plugins
-------
- # Blog, community, releases and course are separate
- # plugin instances that Docusaurus compiles regardless
of
- # DOCS_VERSIONS, so they need a build but no extra
- # versions. Their sidebars belong here for the same
- # reason — they used to force a full multi-version
build
- # for nothing.
- blog/*|community/*|releasenotes/*|course/*)
- add_locale en
- raise_tier build
- ;;
-
sidebarsCommunity.json|sidebarsReleases.json|sidebarsCourse.ts)
- add_locale en
- raise_tier build
- ;;
-
i18n/zh-CN/docusaurus-plugin-content-docs-community/*|i18n/zh-CN/docusaurus-plugin-content-docs-releases/*)
- add_locale zh-CN
- raise_tier build
- ;;
-
- # ---- Nothing the bundler ever sees
-------------------
- # static/ is copied verbatim into the output. A change
- # there cannot fail (or be validated by) a build, so it
- # must not widen the scope on its own. A doc that
- # references a new asset triggers its own build above.
- static/*)
- ;;
- # CI config, governance tooling, repo-level prose and
- # helper scripts. No Python or shell file in this repo
is
- # compiled into the site.
- .github/*|.claude/*|.agents/*|.vscode/*|.idea/*)
- ;;
- scripts/docs-governance/*|website-quality-governance/*)
- ;;
- *.py|*.sh)
- ;;
-
README.md|AGENTS.md|TOC.md|LICENSE|NOTICE|store_format_v3.md|cspell.json|author.yml|buildVersions.json|.gitignore|.prettierrc|.prettierignore|.eslintrc|.eslintrc.js)
- ;;
-
- # ---- Site code, config and deps
----------------------
- # Compiled once and shared by every version and locale,
- # so the representative version is a complete check.
- # scripts/ entries listed here are genuine build
inputs:
- # key-features/ and last-update/ are read by
- # docusaurus.config.js, patch-search-tokenize.js runs
on
- # postinstall.
-
src/*|config/*|docusaurus.config.js|package.json|yarn.lock|tailwind.config.js|tsconfig.json|scripts/key-features/*|scripts/last-update/*|scripts/patch-search-tokenize.js)
- add_locale en
- raise_tier build
- ;;
-
- # ---- Anything else: fail safe
------------------------
- *)
- echo "Unclassified path '$file' — falling back to
a minimal build."
- add_locale en
- raise_tier build
- ;;
- esac
- done <<< "$CHANGED_FILES"
-
- # Syntax-check the translation tables we chose not to build.
- if [ -n "$JSON_TO_VALIDATE" ]; then
- echo ""
- echo "=== Validating changed i18n JSON ==="
- while IFS= read -r f; do
- [ -n "$f" ] || continue
- # Deleted in this PR — nothing left to parse.
- [ -f "$f" ] || { echo " skip (deleted) $f";
continue; }
- if jq empty "$f" >/dev/null 2>&1; then
- echo " ok $f"
- else
- echo " INVALID JSON $f"
- jq empty "$f" || true
- exit 1
- fi
- done <<< "$JSON_TO_VALIDATE"
- fi
-
- # The versions Docusaurus can actually build: `current` (dev)
- # plus everything in versions.json. Older dirs (version-1.2,
- # version-2.0) may still sit on disk while being absent from
- # versions.json — feeding those to onlyIncludeVersions fails
the
- # build with "unknown versions".
- ACTIVE_VERSIONS=$(jq -r '.[]' versions.json | tr '\n' ',' |
sed 's/,$//')
- ACTIVE_VERSIONS="current,${ACTIVE_VERSIONS}"
- echo ""
- echo "Active versions (current + versions.json):
$ACTIVE_VERSIONS"
-
- SKIP_BUILD="false"
- NEED_FULL_BUILD="false"
- DOCS_VERSIONS=""
-
- if [ "$TIER" = "skip" ]; then
- SKIP_BUILD="true"
- LOCALES="en"
- # An empty DOCS_VERSIONS means "every version" to
- # docusaurus.config.js. Nothing should read it while
- # skip_build is set, but pin it to the probe version
anyway so
- # a future edit can't turn a skipped build into the most
- # expensive one.
- DOCS_VERSIONS="$PROBE_VERSION"
- echo "Nothing that reaches the bundler changed — skipping
the site build."
- elif [ "$TIER" = "full" ]; then
- NEED_FULL_BUILD="true"
- DOCS_VERSIONS=""
- echo "The version set itself changed — building ALL active
versions."
- else
- RAW_VERSIONS=$(echo "$VERSIONS" | tr ',' '\n' | sort -u |
grep -v '^$' || true)
- KEPT=""
- DROPPED=""
- for v in $RAW_VERSIONS; do
- if echo ",$ACTIVE_VERSIONS," | grep -q ",$v,"; then
- KEPT="${v},$KEPT"
- else
- DROPPED="${v},$DROPPED"
- fi
- done
- DOCS_VERSIONS=$(echo "$KEPT" | sed 's/,$//')
- DROPPED=$(echo "$DROPPED" | sed 's/,$//')
- if [ -n "$DROPPED" ]; then
- echo "Ignoring changes to retired/unknown versions:
$DROPPED"
- fi
- if [ -z "$DOCS_VERSIONS" ]; then
- # Either no versioned content changed (site code, blog,
- # community, course...) or everything we found was
- # retired. One representative version is enough.
- DOCS_VERSIONS="$PROBE_VERSION"
- echo "No active versioned content changed — probing
with '$PROBE_VERSION' only."
- else
- echo "Versioned content changed for: $DOCS_VERSIONS"
- fi
- fi
-
- # Default the locale when a build is needed but no
locale-specific
- # content changed (site code, deps, blog, ...).
- if [ "$SKIP_BUILD" != "true" ] && [ -z "$LOCALES" ]; then
- LOCALES="en"
- fi
-
- LOCALE_ARGS=""
- IFS=',' read -ra LOCALE_ARR <<< "$LOCALES"
- for locale in "${LOCALE_ARR[@]}"; do
- [ -n "$locale" ] || continue
- LOCALE_ARGS="$LOCALE_ARGS --locale $locale"
- done
-
- echo "docs_versions=$DOCS_VERSIONS" >> "$GITHUB_OUTPUT"
- echo "locale_args=$LOCALE_ARGS" >> "$GITHUB_OUTPUT"
- echo "need_full_build=$NEED_FULL_BUILD" >> "$GITHUB_OUTPUT"
- echo "skip_build=$SKIP_BUILD" >> "$GITHUB_OUTPUT"
-
- echo ""
- echo "=== Build plan ==="
- echo " TIER: $TIER"
- echo " SKIP_BUILD: $SKIP_BUILD"
- echo " DOCS_VERSIONS: ${DOCS_VERSIONS:-all}"
- echo " LOCALE_ARGS: ${LOCALE_ARGS:- (none)}"
- echo " NEED_FULL_BUILD: $NEED_FULL_BUILD"
+ - name: Prepare swap for Japanese build
+ if: steps.detect.outputs.enable_ja == 'true' &&
steps.detect.outputs.skip_build != 'true'
+ run: |
+ sudo fallocate -l 8G /mnt/japanese-build-check-swap
+ sudo chmod 600 /mnt/japanese-build-check-swap
+ sudo mkswap /mnt/japanese-build-check-swap
+ sudo swapon /mnt/japanese-build-check-swap
- name: Build
env:
DOCS_VERSIONS: ${{ steps.detect.outputs.docs_versions }}
+ DORIS_ENABLE_JA: ${{ steps.detect.outputs.enable_ja }}
+ PWA_SERVICE_WORKER_URL: https://doris.apache.org/sw.js
run: |
npm install -g yarn
yarn cache clean
export NODE_OPTIONS=--max-old-space-size=8192
yarn
yarn key-features:generate
+ node --test scripts/i18n/__tests__/t04.test.cjs
# Dependencies are installed unconditionally above because
the
# governance steps below need node_modules; only the site
build
@@ -385,9 +106,22 @@ jobs:
fi
echo "Building with DOCS_VERSIONS=${DOCS_VERSIONS:-all}"
- PWA_SERVICE_WORKER_URL=https://doris.apache.org/sw.js yarn
docusaurus build ${{ steps.detect.outputs.locale_args }}
+ if [ "${{ steps.detect.outputs.locale_count }}" = "1" ] && [
"${{ steps.detect.outputs.enable_ja }}" != "true" ]; then
+ PWA_SERVICE_WORKER_URL=https://doris.apache.org/sw.js
yarn docusaurus build ${{ steps.detect.outputs.locale_args }}
+ else
+ # Every locale gets a separate Node process and its real
prefix.
+ yarn build:locales
+ fi
+ if [ "${{ steps.detect.outputs.enable_ja }}" = "true" ]; then
+ node scripts/i18n/verify-static.cjs --site build
--manifest .i18n-build/manifests/ja.json
+ fi
rm -rf build
+ - name: Japanese translation gate
+ run: |
+ BASE_SHA=$(git merge-base origin/${{ github.base_ref }} HEAD)
+ node scripts/i18n/check-translations.cjs --base "$BASE_SHA"
+
- name: Docs changed link gate
continue-on-error: true
run: |
diff --git a/.github/workflows/check-md-links.yml
b/.github/workflows/check-md-links.yml
index 4dc0984a00c..d345d7cc2d4 100644
--- a/.github/workflows/check-md-links.yml
+++ b/.github/workflows/check-md-links.yml
@@ -18,6 +18,9 @@ jobs:
with:
python-version: '3.11'
+ - name: Verify Japanese source fallback resolver
+ run: python3 -m unittest scripts/__tests__/test_japanese_links.py
+
- name: Collect added/modified Markdown files
id: changed
env:
@@ -36,7 +39,8 @@ jobs:
"$BASE_SHA" HEAD \
-- 'docs/**/*.md' 'docs/**/*.mdx' \
'i18n/zh-CN/docusaurus-plugin-content-docs/current/**/*.md' \
- 'i18n/zh-CN/docusaurus-plugin-content-docs/current/**/*.mdx'
+ 'i18n/zh-CN/docusaurus-plugin-content-docs/current/**/*.mdx' \
+ 'i18n/ja/**/*.md' 'i18n/ja/**/*.mdx'
echo EOF
} >> "$GITHUB_OUTPUT"
diff --git a/.github/workflows/japanese-coverage-audit.yml
b/.github/workflows/japanese-coverage-audit.yml
new file mode 100644
index 00000000000..8acbf687dde
--- /dev/null
+++ b/.github/workflows/japanese-coverage-audit.yml
@@ -0,0 +1,49 @@
+name: Japanese coverage audit
+
+on:
+ workflow_dispatch:
+ schedule:
+ - cron: '17 5 * * 1'
+ pull_request:
+ branches: [master]
+ paths:
+ - 'docs/**'
+ - 'versioned_docs/version-4.x/**'
+ - 'community/**'
+ - 'i18n/ja/**'
+ - 'translations/**'
+ - 'sidebars.ts'
+ - 'sidebarsCommunity.json'
+ - 'versioned_sidebars/version-4.x-sidebars.json'
+ - 'scripts/i18n/**'
+ - '.github/workflows/japanese-coverage-audit.yml'
+
+permissions:
+ contents: read
+
+jobs:
+ audit:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ persist-credentials: false
+ - uses: actions/setup-node@v4
+ with:
+ node-version: 20
+ cache: yarn
+ - run: yarn --frozen-lockfile
+ - name: Recount native scope at checked-out source SHA
+ env:
+ DORIS_ENABLE_JA: 'true'
+ run: |
+ mkdir -p "$RUNNER_TEMP/japanese-audit"
+ node scripts/i18n/audit-coverage.cjs --source-ref HEAD --output
"$RUNNER_TEMP/japanese-audit/coverage.json"
+ - uses: actions/upload-artifact@v4
+ if: always()
+ with:
+ name: japanese-coverage-${{ github.sha }}
+ path: ${{ runner.temp }}/japanese-audit/coverage.json
+ if-no-files-found: warn
+ retention-days: 30
diff --git a/.github/workflows/japanese-documents.yml
b/.github/workflows/japanese-documents.yml
index 8ba4398a1f6..8c31ded7dde 100644
--- a/.github/workflows/japanese-documents.yml
+++ b/.github/workflows/japanese-documents.yml
@@ -14,6 +14,10 @@ on:
- 'translations/**'
- 'i18n/ja/**'
- 'docusaurus.config.js'
+ - 'src/theme/SearchBar/**'
+ - 'src/theme/SearchPage/**'
+ - 'src/utils/tokenize.ts'
+ - 'static/.htaccess'
- '.github/workflows/japanese-documents.yml'
permissions:
@@ -27,7 +31,7 @@ jobs:
document-scope:
# T03 validates real Dev/4.x content, without repeating the full memory
# experiment. The complete production matrix remains in japanese-poc.yml.
- if: github.event_name == 'workflow_dispatch' ||
startsWith(github.head_ref, 'feat/japanese-doc-inventory')
+ if: github.event_name == 'workflow_dispatch' ||
startsWith(github.head_ref, 'feat/japanese-doc-inventory') ||
startsWith(github.head_ref, 'feat/japanese-i18n-ci-search')
runs-on: ubuntu-latest
timeout-minutes: 120
env:
@@ -68,9 +72,7 @@ jobs:
yarn key-features:generate
mkdir -p ../evidence
node --test scripts/i18n/__tests__/*.test.cjs
scripts/i18n/poc/*.test.cjs scripts/__tests__/build-locales.test.cjs
- SOURCE_COMMIT=$(node -p
"require('./translations/inventory/ja/2026-10-12/summary.json').sourceCommit")
- node scripts/i18n/inventory.cjs --source-ref "$SOURCE_COMMIT"
--output ../evidence/inventory
- diff -r translations/inventory/ja/2026-10-12 ../evidence/inventory
+ node scripts/i18n/audit-coverage.cjs --source-ref HEAD --output
../evidence/coverage.json
- name: Build Dev and 4.x English/Chinese baseline
working-directory: baseline
env:
@@ -105,7 +107,9 @@ jobs:
doc_navigation_exit=$?
node candidate/scripts/i18n/poc/verify-static.cjs --site
candidate/build --output evidence/candidate-browser --expect-ja --baseline
evidence/baseline-browser/signatures.json
doc_browser_exit=$?
- if (( doc_status_exit || doc_navigation_exit || doc_browser_exit ));
then exit 1; fi
+ node candidate/scripts/i18n/verify-search.cjs --site candidate/build
> evidence/search.json
+ search_exit=$?
+ if (( doc_status_exit || doc_navigation_exit || doc_browser_exit ||
search_exit )); then exit 1; fi
- name: Preserve CI-built site for failed validation replay
uses: actions/upload-artifact@v4
if: failure()
diff --git a/config/i18n.js b/config/i18n.js
new file mode 100644
index 00000000000..dfddeb4150b
--- /dev/null
+++ b/config/i18n.js
@@ -0,0 +1,4 @@
+// Keep CI locale discovery independent of Docusaurus and installed
dependencies.
+const locales = ['en', 'zh-CN', 'ja'];
+const enabledLocales = () => locales.filter(locale => locale !== 'ja' ||
process.env.DORIS_ENABLE_JA === 'true');
+module.exports = {locales, enabledLocales};
diff --git a/docusaurus.config.js b/docusaurus.config.js
index 4fa33608ae4..da0d23b1cd3 100644
--- a/docusaurus.config.js
+++ b/docusaurus.config.js
@@ -9,7 +9,9 @@ const REDIRECTS_4X = require('./config/redirects-4.x.json');
const path = require('path');
const translationDocsPlugin = require('./scripts/i18n/docs-plugin.cjs');
const translationStatusPlugin = require('./scripts/i18n/status-plugin.cjs');
+const searchPlugin = require('./scripts/i18n/search-plugin.cjs');
const ENABLE_JA = process.env.DORIS_ENABLE_JA === 'true';
+const {enabledLocales} = require('./config/i18n');
const BRAND_THEME_BOOTSTRAP = `(function () {
var theme = 'doris';
@@ -289,7 +291,7 @@ const config = {
trailingSlash: true,
i18n: {
defaultLocale: 'en',
- locales: ENABLE_JA ? ['en', 'zh-CN', 'ja'] : ['en', 'zh-CN'],
+ locales: enabledLocales(),
localeConfigs: {
en: {
label: 'English',
@@ -528,7 +530,7 @@ const config = {
],
themes: SKIP_SEARCH ? [] : [
[
- '@yang1666204/docusaurus-search-local',
+ searchPlugin,
{
hashed: true,
language: ['en', 'zh'],
diff --git a/i18n/ja/code.json b/i18n/ja/code.json
index fbb01b88a85..4badae893ee 100644
--- a/i18n/ja/code.json
+++ b/i18n/ja/code.json
@@ -825,38 +825,38 @@
"description": "The label for the navbar versions dropdown on mobile view"
},
"theme.SearchBar.noResultsText": {
- "message": "No results"
+ "message": "結果が見つかりません"
},
"theme.SearchBar.seeAllOutsideContext": {
- "message": "See all results outside \"{context}\""
+ "message": "「{context}」以外のすべての結果を表示"
},
"theme.SearchBar.searchInContext": {
- "message": "See all results within \"{context}\""
+ "message": "「{context}」の結果をすべて表示"
},
"theme.SearchBar.seeAll": {
- "message": "See all results"
+ "message": "すべての結果を表示"
},
"theme.SearchBar.label": {
- "message": "Search",
+ "message": "検索",
"description": "The ARIA label and placeholder for search button"
},
"theme.SearchPage.existingResultsTitle": {
- "message": "Search results for \"{query}\"",
+ "message": "「{query}」の検索結果",
"description": "The search page title for non-empty query"
},
"theme.SearchPage.emptyResultsTitle": {
- "message": "Search the documentation",
+ "message": "ドキュメントを検索",
"description": "The search page title for empty query"
},
"theme.SearchPage.searchContext.everywhere": {
- "message": "Everywhere"
+ "message": "全体"
},
"theme.SearchPage.documentsFound.plurals": {
- "message": "1 document found|{count} documents found",
+ "message": "{count} 件の文書が見つかりました",
"description": "Pluralized label for \"{count} documents found\". Use as
much plural forms (separated by \"|\") as your language support (see
https://www.unicode.org/cldr/cldr-aux/charts/34/supplemental/language_plural_rules.html)"
},
"theme.SearchPage.noResultsText": {
- "message": "No documents were found",
+ "message": "該当する文書はありません",
"description": "The paragraph for empty search result"
},
"theme.AnnouncementBar.closeButtonAriaLabel": {
diff --git
a/i18n/ja/docusaurus-plugin-content-docs/current/getting-started/what-is-apache-doris.md
b/i18n/ja/docusaurus-plugin-content-docs/current/getting-started/what-is-apache-doris.md
index 9565599277d..0235bfdd637 100644
---
a/i18n/ja/docusaurus-plugin-content-docs/current/getting-started/what-is-apache-doris.md
+++
b/i18n/ja/docusaurus-plugin-content-docs/current/getting-started/what-is-apache-doris.md
@@ -112,7 +112,7 @@ Apache Doris は列指向ストレージを採用し、列ごとにデータを
- **ストレージモデル**：Duplicate モデル、Aggregate モデル、Unique モデル（行単位の更新に対応）
- **マテリアライズドビュー**：強い整合性を持つ単一テーブルのビューと、非同期に更新する複数テーブルのビュー
-> **ベクトルインデックスと転置インデックスは、ハイブリッド検索と分析を支える主要な技術です**。詳細は [AI
の概要](ai/ai-overview.md)を参照してください。
+> **ベクトルインデックスと転置インデックスは、ハイブリッド検索と分析を支える主要な技術です**。詳細は [AI
の概要](../ai/ai-overview.md)を参照してください。
### クエリエンジン {#query-engine}
diff --git
a/i18n/ja/docusaurus-plugin-content-docs/version-4.x/getting-started/what-is-apache-doris.md
b/i18n/ja/docusaurus-plugin-content-docs/version-4.x/getting-started/what-is-apache-doris.md
index 9565599277d..0235bfdd637 100644
---
a/i18n/ja/docusaurus-plugin-content-docs/version-4.x/getting-started/what-is-apache-doris.md
+++
b/i18n/ja/docusaurus-plugin-content-docs/version-4.x/getting-started/what-is-apache-doris.md
@@ -112,7 +112,7 @@ Apache Doris は列指向ストレージを採用し、列ごとにデータを
- **ストレージモデル**：Duplicate モデル、Aggregate モデル、Unique モデル（行単位の更新に対応）
- **マテリアライズドビュー**：強い整合性を持つ単一テーブルのビューと、非同期に更新する複数テーブルのビュー
-> **ベクトルインデックスと転置インデックスは、ハイブリッド検索と分析を支える主要な技術です**。詳細は [AI
の概要](ai/ai-overview.md)を参照してください。
+> **ベクトルインデックスと転置インデックスは、ハイブリッド検索と分析を支える主要な技術です**。詳細は [AI
の概要](../ai/ai-overview.md)を参照してください。
### クエリエンジン {#query-engine}
diff --git a/scripts/__tests__/test_japanese_links.py
b/scripts/__tests__/test_japanese_links.py
new file mode 100644
index 00000000000..4ddb8151362
--- /dev/null
+++ b/scripts/__tests__/test_japanese_links.py
@@ -0,0 +1,43 @@
+import os
+import tempfile
+import unittest
+from pathlib import Path
+from scripts.check_md_links_single import resolve_internal
+
+
+class JapaneseLinkResolution(unittest.TestCase):
+ def test_missing_translation_resolves_to_same_version_source(self):
+ with tempfile.TemporaryDirectory() as temp:
+ root = Path(temp)
+ (root / 'static').mkdir()
+ current = root /
'i18n/ja/docusaurus-plugin-content-docs/current/guide'
+ current.mkdir(parents=True)
+ source = root / 'docs/reference/topic.md'
+ source.parent.mkdir(parents=True)
+ source.write_text('# Topic\n')
+ other = root / 'versioned_docs/version-4.x/reference/topic.md'
+ other.parent.mkdir(parents=True)
+ other.write_text('# 4.x topic\n')
+ versioned = root /
'i18n/ja/docusaurus-plugin-content-docs/version-4.x/guide'
+ versioned.mkdir(parents=True)
+ self.assertEqual(resolve_internal(str(current),
'../reference/topic.md'), str(source))
+ self.assertEqual(resolve_internal(str(versioned),
'../reference/topic.md'), str(other))
+ self.assertEqual(resolve_internal(str(current),
'/ja/docs/dev/reference/topic'), str(source))
+ self.assertEqual(resolve_internal(str(versioned),
'/ja/docs/4.x/reference/topic'), str(other))
+ self.assertIsNone(resolve_internal(str(current),
'/ja/docs/dev/reference/missing'))
+
+ def test_community_fallback_uses_community_plugin(self):
+ with tempfile.TemporaryDirectory() as temp:
+ root = Path(temp)
+ (root / 'static').mkdir()
+ localized = root /
'i18n/ja/docusaurus-plugin-content-docs-community/current/guide'
+ localized.mkdir(parents=True)
+ source = root / 'community/guide/next.mdx'
+ source.parent.mkdir(parents=True)
+ source.write_text('# Next\n')
+ self.assertEqual(resolve_internal(str(localized), 'next.mdx'),
str(source))
+ self.assertEqual(resolve_internal(str(localized),
'/ja/community/guide/next'), str(source))
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/scripts/build-locales.cjs b/scripts/build-locales.cjs
index ca818998a9b..e906fbf9e37 100644
--- a/scripts/build-locales.cjs
+++ b/scripts/build-locales.cjs
@@ -89,7 +89,7 @@ async function buildLocales({siteDir = process.cwd(), outDir
= 'build', run, sta
throw error;
} finally {
fs.writeFileSync(reportPath,JSON.stringify(report,null,2)+'\n');
- fs.rmSync(stage,{recursive:true,force:true});
+
fs.rmSync(stage,{recursive:true,force:true,maxRetries:8,retryDelay:250});
}
}
diff --git a/scripts/check_md_links_single.py b/scripts/check_md_links_single.py
index 2016834adf9..c32acad7535 100755
--- a/scripts/check_md_links_single.py
+++ b/scripts/check_md_links_single.py
@@ -29,7 +29,7 @@ import re
import sys
from concurrent.futures import ThreadPoolExecutor, as_completed
from functools import lru_cache
-from urllib.parse import urlparse
+from urllib.parse import urlparse, unquote
try:
import urllib.request
@@ -113,7 +113,7 @@ def _candidates_for(full: str):
def resolve_internal(base_dir: str, target: str):
"""Return the resolved file path if target exists, else None."""
- path_part = target.split("#", 1)[0]
+ path_part = unquote(target.split("#", 1)[0].split("?", 1)[0])
if path_part == "":
return "anchor-only"
@@ -129,6 +129,19 @@ def resolve_internal(base_dir: str, target: str):
if site_root is None:
return None
rel = path_part.lstrip("/")
+ localized =
re.match(r"ja/(docs/(dev|[^/]+)|community|releases|course)/(.*)", rel)
+ if localized:
+ route, version, rest = localized.groups()
+ if route.startswith("docs/"):
+ source = "docs" if version == "dev" else
"versioned_docs/version-" + version
+ localized_root = "i18n/ja/docusaurus-plugin-content-docs/" +
("current" if version == "dev" else "version-" + version)
+ else:
+ source = ROUTE_PREFIX_REMAP.get(route, route)
+ localized_root = "i18n/ja/docusaurus-plugin-content-docs-" +
route + "/current"
+ for root in (localized_root, source):
+ for candidate in _candidates_for(os.path.join(site_root, root,
rest.rstrip("/"))):
+ if os.path.isfile(candidate):
+ return candidate
head, _, rest = rel.partition("/")
if head in ROUTE_PREFIX_REMAP and rest:
remapped = os.path.normpath(
@@ -147,6 +160,18 @@ def resolve_internal(base_dir: str, target: str):
for c in _candidates_for(full):
if os.path.isfile(c):
return c
+ site_root, _ = _find_site_roots(base_dir)
+ if site_root:
+ relative = os.path.relpath(full, site_root).replace(os.sep, "/")
+ localized =
re.match(r"i18n/ja/docusaurus-plugin-content-docs(-[^/]+)?/(current|version-([^/]+))/(.*)",
relative)
+ if localized:
+ suffix, _, version, rest = localized.groups()
+ roots = {None: "docs", "-community": "community", "-releases":
"releasenotes", "-course": "course"}
+ source = "versioned_docs/version-" + version if version and suffix
is None else roots.get(suffix) if not version else None
+ if source:
+ for candidate in _candidates_for(os.path.join(site_root,
source, rest)):
+ if os.path.isfile(candidate):
+ return candidate
return None
diff --git a/scripts/check_move.js b/scripts/check_move.js
index 3e4520c043d..5e5b2f59987 100644
--- a/scripts/check_move.js
+++ b/scripts/check_move.js
@@ -7,10 +7,11 @@
const { execSync } = require("child_process");
const fs = require("fs");
const path = require("path");
+const { sourceForLocalized } = require("./i18n/paths.cjs");
const commitHash = process.argv[2];
-if (!commitHash) {
+if (require.main === module && !commitHash) {
console.error("❌ Please provide the commit hash, such as: node
check-dead-links.js <commit-hash>");
process.exit(1);
}
@@ -52,6 +53,19 @@ function isLocalLink(link) {
!/.*@.*\..*/.test(link);
}
+function existingLocalTarget(filePath, rawLink) {
+ const candidates = [path.resolve(path.dirname(filePath), rawLink)];
+ // The Japanese docs plugin intentionally serves the English page when a
+ // translation is absent. Resolve its relative links from the same plugin
+ // and version of the English source; keep the EN/ZH gate unchanged.
+ if (filePath.startsWith("i18n/ja/")) {
+ const source = sourceForLocalized(filePath);
+ if (source) candidates.push(path.resolve(path.dirname(source), rawLink));
+ }
+ return candidates.some(candidate => [candidate, `${candidate}.md`,
`${candidate}.mdx`]
+ .some(file => fs.existsSync(file)));
+}
+
// Strip Markdown code regions so the link scanner ignores their contents.
//
// The previous implementation used `/```[\s\S]*?```/g` followed by ``
/`[^`]*`/g ``,
@@ -196,7 +210,6 @@ function removeMdxEsmBlocks(content) {
// Check links in files
function checkFileLinks(filePath) {
const content = fs.readFileSync(filePath, "utf-8");
- const dir = path.dirname(filePath);
const cleanedContent = removeMdxEsmBlocks(removeCodeBlocks(content));
const matches = [...cleanedContent.matchAll(linkRegex)];
@@ -205,12 +218,7 @@ function checkFileLinks(filePath) {
if (!isLocalLink(rawLink)) continue;
console.log('rawLink',rawLink);
- let fullPath = path.resolve(dir, rawLink);
- if (!fs.existsSync(fullPath)) {
- // Try adding a .md/.mdx suffix and try again
- if (fs.existsSync(fullPath + ".md")) continue;
- if (fs.existsSync(fullPath + ".mdx")) continue;
-
+ if (!existingLocalTarget(filePath, rawLink)) {
console.error(`❌ ${filePath}: Broken link -> ${rawLink}`);
hasBrokenLinks = true;
}
@@ -240,4 +248,5 @@ function main() {
}
}
-main();
+if (require.main === module) main();
+module.exports = { existingLocalTarget };
diff --git a/scripts/docs-governance/lint-links.js
b/scripts/docs-governance/lint-links.js
index 1b6af0c77e1..41972cb5b53 100644
--- a/scripts/docs-governance/lint-links.js
+++ b/scripts/docs-governance/lint-links.js
@@ -16,6 +16,7 @@ const {
stripMarkdownExtension,
} = require('./lib');
const { loadSidebarRefs } = require('./lint-sidebar');
+const {sourceForLocalized} = require('../i18n/paths.cjs');
const GOVERNED_MARKDOWN_PREFIXES = [
'blog/',
@@ -220,6 +221,10 @@ function buildIndexes(rootDir, manifest, fileCache) {
}
}
+ // Japanese heading IDs always follow the actual version's English source.
+ for (const entry of manifest.entries.filter(entry => entry.locale === 'ja'))
{
+ if (entry.source_counterpart) anchorsBySource.set(entry.source_path,
anchorsBySource.get(entry.source_counterpart) || new Set());
+ }
return { bySource, byRoute, anchorsBySource };
}
@@ -280,7 +285,8 @@ function resolveInternalTarget(rootDir, sourcePath,
rawTarget, indexes) {
if (fs.existsSync(path.join(rootDir, staticCandidate))) {
return { kind: 'asset', sourcePath: staticCandidate, hash };
}
- const entry = indexes.byRoute.get(pathname.replace(/\/+$/, ''));
+ const entry = indexes.byRoute.get(pathname.replace(/\/+$/, ''))
+ || (pathname.startsWith('/ja/') ?
indexes.byRoute.get(pathname.slice(3).replace(/\/+$/, '')) : null);
if (entry) {
return { kind: 'route', sourcePath: entry.source_path, hash };
}
@@ -295,7 +301,11 @@ function resolveInternalTarget(rootDir, sourcePath,
rawTarget, indexes) {
return { kind: 'missing-route', pathname, hash };
}
- const files = candidateFiles(rootDir, sourcePath, pathname);
+ let files = candidateFiles(rootDir, sourcePath, pathname);
+ if (!files.length && sourcePath.startsWith('i18n/ja/')) {
+ const source = sourceForLocalized(sourcePath);
+ if (source) files = candidateFiles(rootDir, source, pathname);
+ }
if (files.length > 0) {
return { kind: 'file', sourcePath: files[0], hash };
}
diff --git a/scripts/docs-governance/manifest.js
b/scripts/docs-governance/manifest.js
index b80925c2942..70c9e7e5934 100644
--- a/scripts/docs-governance/manifest.js
+++ b/scripts/docs-governance/manifest.js
@@ -18,15 +18,15 @@ const {
walkMarkdownFiles,
} = require('./lib');
+const {locales} = require('../../config/i18n');
const CONTENT_ROOTS = [
- 'docs',
- 'versioned_docs',
- 'i18n/zh-CN/docusaurus-plugin-content-docs',
- 'community',
- 'i18n/zh-CN/docusaurus-plugin-content-docs-community/current',
- 'releasenotes',
- 'i18n/zh-CN/docusaurus-plugin-content-docs-releases/current',
- 'blog',
+ 'docs', 'versioned_docs', 'community', 'releasenotes', 'blog',
+ ...locales.filter(locale => locale !== 'en').flatMap(locale => [
+ `i18n/${locale}/docusaurus-plugin-content-docs`,
+ `i18n/${locale}/docusaurus-plugin-content-docs-community/current`,
+ `i18n/${locale}/docusaurus-plugin-content-docs-releases/current`,
+ `i18n/${locale}/docusaurus-plugin-content-docs-course/current`,
+ ]),
];
function parseFrontMatter(absPath) {
@@ -47,6 +47,20 @@ function parseFrontMatter(absPath) {
function getContentInfo(relativePath, rules) {
const currentRoute = rules.currentRouteVersion;
+ const localized =
relativePath.match(/^i18n\/([^/]+)\/docusaurus-plugin-content-docs(-[^/]+)?\/(current|version-([^/]+))\/(.+)$/);
+ if (localized && locales.includes(localized[1])) {
+ const [, locale, suffix = '', , archivedVersion, docRelative] = localized;
+ const version = archivedVersion || 'current';
+ const plugins = {'': ['main_docs', 'docs', 'sidebars.ts'], '-community':
['community', 'community', 'sidebarsCommunity.json'],
+ '-releases': ['releases', 'releases', 'sidebarsReleases.json'],
'-course': ['course', 'course', 'sidebarsCourse.ts']};
+ const plugin = plugins[suffix];
+ if (!plugin) return null;
+ return {contentRoot: `${locale === 'zh-CN' ? 'zh' : locale}_${suffix ?
plugin[0] : archivedVersion ? 'versioned_docs' : 'docs'}`,
+ plugin: plugin[0], locale, version, docRelative,
+ routeBase: `/${locale}/${plugin[1]}${suffix ? '' : `/${archivedVersion
|| currentRoute}`}`,
+ sidebarSource: suffix ? plugin[2] : archivedVersion ?
`versioned_sidebars/version-${version}-sidebars.json` : plugin[2]};
+ }
+
let match = relativePath.match(/^versioned_docs\/version-([^/]+)\/(.+)$/);
if (match) {
const version = match[1];
@@ -198,7 +212,7 @@ function resolveCanonicalRoute(entry, rules) {
return entry.route_path;
}
const docPart = routePartFromEntry(entry.doc_id, entry.plugin);
- const localePrefix = entry.locale === 'zh-CN' ? '/zh-CN' : '';
+ const localePrefix = entry.locale === 'en' ? '' : `/${entry.locale}`;
if (entry.version === 'current') {
return `${localePrefix}/docs/${rules.defaultVersion}/${docPart}`;
}
@@ -291,7 +305,7 @@ function addCounterparts(entries) {
.map((candidate) => candidate.source_path);
const sourceCounterpart = groupEntries.find(
(candidate) => candidate.locale === 'en' && candidate.version ===
entry.version,
- ) || groupEntries.find((candidate) => candidate.locale === 'en' &&
candidate.version === 'current');
+ ) || (entry.locale === 'ja' ? null : groupEntries.find((candidate) =>
candidate.locale === 'en' && candidate.version === 'current'));
entry.source_counterpart = entry.locale === 'en' ? null :
sourceCounterpart?.source_path || null;
}
}
diff --git a/scripts/i18n/__tests__/t04.test.cjs
b/scripts/i18n/__tests__/t04.test.cjs
new file mode 100644
index 00000000000..7934418a964
--- /dev/null
+++ b/scripts/i18n/__tests__/t04.test.cjs
@@ -0,0 +1,128 @@
+const test = require('node:test');
+const assert = require('node:assert/strict');
+const fs = require('node:fs');
+const os = require('node:os');
+const path = require('node:path');
+const {buildScope} = require('../build-scope.cjs');
+const {japanesePages} = require('../search-plugin.cjs');
+const {sidebarAudit} = require('../audit-coverage.cjs');
+const {structure,alignHeadings} = require('../structure.cjs');
+const {protectionSignature} = require('../protection.cjs');
+const versions = ['4.x','3.x','2.1'];
+
+test('PR build scope separates content locales, shared inputs, and retired
versions', () => {
+ assert.deepEqual(buildScope(['docs/a.md'],versions).locales,['en']);
+
assert.deepEqual(buildScope(['i18n/zh-CN/docusaurus-plugin-content-docs/current/a.md'],versions).locales,['zh-CN']);
+ const japanese =
buildScope(['i18n/ja/docusaurus-plugin-content-docs/version-4.x/a.md'],versions);
+ assert.deepEqual(japanese.locales,['ja']);
+ assert.equal(japanese.docs_versions,'4.x');
+ assert.equal(japanese.enable_ja,true);
+
assert.deepEqual(buildScope(['sidebars.ts'],versions).locales,['en','zh-CN','ja']);
+ assert.equal(buildScope(['versions.json'],versions).need_full_build,true);
+
assert.deepEqual(buildScope(['versioned_docs/version-1.2/a.md'],versions).retired,['1.2']);
+ assert.equal(buildScope(['i18n/ja/code.json'],versions).skip_build,false);
+
assert.equal(buildScope(['i18n/zh-CN/code.json'],versions).skip_build,true);
+ assert.throws(() =>
buildScope(['i18n/fr/docusaurus-plugin-content-docs/current/a.md'],versions),/Unconfigured/);
+});
+
+test('Japanese index admits only current, reviewed and in-scope native
targets', () => {
+ const page = (pluginId,version,status='current',indexable=true) =>
({pluginId,version,status,indexable,useTarget:true,
+ permalink:`/ja/${pluginId}/${version}/${status}/${indexable}/`});
+ const pages =
[page('default','current'),page('default','4.x'),page('community','current'),
+
page('default','3.x'),page('default','current','stale_unknown'),page('default','current','current',false)];
+ const found =
japanesePages([{name:'docusaurus-plugin-content-docs',content:{translationManifest:{pages}}}]);
+ assert.deepEqual([...found.values()],['docs/dev','docs/4.x','community']);
+});
+
+test('sidebar audit discovers new native keys without a snapshot allowlist',
() => {
+ const root =
fs.mkdtempSync(path.join(os.tmpdir(),'japanese-sidebar-test-'));
+ try {
+ const dir = path.join(root,'i18n/ja/docusaurus-plugin-content-docs');
+ fs.mkdirSync(dir,{recursive:true});
+
fs.writeFileSync(path.join(dir,'current.json'),JSON.stringify({'sidebar.docs.category.Existing':{message:'既存'}}));
+ const native =
{labels:[{pluginId:'default',files:[{path:'current',content:{
+
'sidebar.docs.category.Existing':{message:'Existing'},'sidebar.docs.category.New':{message:'New'}}}]}]};
+ const entries =
sidebarAudit(native,root,{groups:[{id:'DEV',pluginId:'default',version:'current'}]});
+ assert.equal(entries.length,2);
+ assert.equal(entries.find(entry =>
entry.key.endsWith('.New')).state,'missing');
+ } finally {fs.rmSync(root,{recursive:true,force:true});}
+});
+
+test('protected Japanese changes reject SQL and route edits but allow
translated prose', async () => {
+ const en = '---\ntitle: English\nlanguage: en\n---\n\n# Overview\n\nHello
[guide](next.md).\n\n```sql\nSELECT 1;\n```\n';
+ const ja = '---\ntitle: 日本語\nlanguage: ja\n---\n\n# 概要
{#overview}\n\nこんにちは [ガイド](next.md)。\n\n```sql\nSELECT 1;\n```\n';
+ const source = await structure(en,'.md');
+ const candidate = await structure(await
alignHeadings(source,ja,'.md'),'.md');
+ assert.equal(protectionSignature(candidate),protectionSignature(source));
+ const broken = await structure(await
alignHeadings(source,ja.replace('SELECT 1','SELECT 2'),'.md'),'.md');
+ assert.notEqual(protectionSignature(broken),protectionSignature(source));
+ const wrongLink = await structure(await
alignHeadings(source,ja.replace('next.md','other.md'),'.md'),'.md');
+
assert.notEqual(protectionSignature(wrongLink),protectionSignature(source));
+});
+
+test('full audit finds documents and sidebar keys added after the T03
snapshot', async () => {
+ const {execFileSync} = require('node:child_process');
+ const {hash} = require('../state.cjs');
+ const {audit} = require('../audit-coverage.cjs');
+ const root = fs.mkdtempSync(path.join(os.tmpdir(),'japanese-audit-test-'));
+ const baseline = path.join(root,'baseline');
+ const write = (file,content) => {
+ const target =
path.join(root,file);fs.mkdirSync(path.dirname(target),{recursive:true});fs.writeFileSync(target,content);
+ };
+ try {
+ fs.mkdirSync(baseline);
+ for (const group of ['dev','4x','community'])
fs.writeFileSync(path.join(baseline,`${group}.json`),JSON.stringify({entryFiles:[]}));
+
fs.writeFileSync(path.join(baseline,'summary.json'),JSON.stringify({sourceCommit:'t03'}));
+ const groups = [
+
{id:'DEV',pluginId:'default',version:'current',sourceRoot:'docs',file:'docs/new.md'},
+
{id:'4X',pluginId:'default',version:'4.x',sourceRoot:'versioned_docs/version-4.x',file:'versioned_docs/version-4.x/new.md'},
+
{id:'COMMUNITY',pluginId:'community',version:'current',sourceRoot:'community',file:'community/new.md'},
+ ];
+
write('translations/scope/ja.json',JSON.stringify({schemaVersion:1,locale:'ja',maxBatchPages:20,groups,retainEnglish:[]}));
+ write('translations/config.json','{}');
+
write('translations/state/ja/docs.json',JSON.stringify({schemaVersion:1,pages:[]}));
+ for (const file of
['docusaurus.config.js','versions.json','sidebars.ts','sidebarsCommunity.json','versioned_sidebars/version-4.x-sidebars.json'])
+ write(file,file+'\n');
+ const pages = groups.map(group => {
+ const content = '# Newly registered\n';write(group.file,content);
+ return
{pluginId:group.pluginId,version:group.version,docId:'new',englishPath:group.file,sourcePath:group.file,
+
sourceHash:hash(content),targetPath:`i18n/ja/${group.id}/new.md`,targetHash:null,
+
status:'untranslated',reviewStatus:'unverified',useTarget:false,draft:false,permalink:`/ja/${group.id}/new/`};
+ });
+ execFileSync('git',['init','-q'],{cwd:root});
+ execFileSync('git',['add','.'],{cwd:root});
+ execFileSync('git',['-c','user.name=T04
Test','-c','[email protected]','commit','-qm','candidate
scope'],{cwd:root});
+ const native = {pages,reports:[],labels:[
+
{pluginId:'default',files:[{path:'current',content:{'sidebar.docs.category.New':{message:'New'}}},
+
{path:'version-4.x',content:{'sidebar.docs.category.New':{message:'New'}}}]},
+
{pluginId:'community',files:[{path:'current',content:{'sidebar.community.category.New':{message:'New'}}}]},
+ ]};
+ const report = await audit({root,sourceRef:'HEAD',baseline,native});
+ assert.deepEqual(report.groups.map(group => group.counts.new),[1,1,1]);
+ assert.deepEqual(report.groups.map(group =>
group.counts.pending),[1,1,1]);
+ assert.equal(report.labels.missing,3);
+ assert.equal(report.removed.length,0);
+ } finally {fs.rmSync(root,{recursive:true,force:true});}
+});
+
+test('source-only edits report stale Japanese content; edited stale targets
are blocked', async () => {
+ const {checkTranslations} = require('../check-translations.cjs');
+ const page =
{pluginId:'default',version:'current',docId:'test',locale:'ja',status:'stale_unknown',
+
sourcePath:'docs/test.md',targetPath:'i18n/ja/docusaurus-plugin-content-docs/current/getting-started/intro.mdx'};
+ const root = path.resolve(__dirname,'../../..');
+ const sourceEdit = await
checkTranslations({root,pages:[page],files:['docs/test.md'],records:new Map()});
+ assert.equal(sourceEdit.checked,0);
+ assert.deepEqual(sourceEdit.errors,[]);
+ const targetEdit = await
checkTranslations({root,pages:[page],files:[page.targetPath],records:new
Map()});
+ assert.equal(targetEdit.checked,1);
+ assert.match(targetEdit.errors[0].message,/current source\/state/);
+});
+
+test('required legacy link gate follows Japanese fallback in the same
version', () => {
+ const {existingLocalTarget} = require('../../check_move.js');
+ for (const version of ['current','version-4.x']) {
+ const file =
`i18n/ja/docusaurus-plugin-content-docs/${version}/getting-started/what-is-apache-doris.md`;
+ assert.equal(existingLocalTarget(file,'../ai/ai-overview.md'),true);
+
assert.equal(existingLocalTarget(file,'../ai/not-a-page-t04.md'),false);
+ }
+});
diff --git a/scripts/i18n/audit-coverage.cjs b/scripts/i18n/audit-coverage.cjs
new file mode 100644
index 00000000000..6ffef07052c
--- /dev/null
+++ b/scripts/i18n/audit-coverage.cjs
@@ -0,0 +1,122 @@
+// Recount the complete native scope on every run. The T03 snapshot is only a
+// comparison baseline, never an allowlist for new documents or sidebar keys.
+const assert = require('node:assert/strict');
+const fs = require('node:fs');
+const path = require('node:path');
+const {execFileSync} = require('node:child_process');
+const {hash, identity} = require('./state.cjs');
+const {collect, validateScope} = require('./inventory.cjs');
+
+const baselineDir =
path.resolve(__dirname,'../../translations/inventory/ja/2026-10-12');
+const key = page => identity({...page,locale:'ja'});
+const refContent = (root,ref,file) => {
+ try {return
execFileSync('git',['show',`${ref}:${file}`],{cwd:root,encoding:'utf8',stdio:['ignore','pipe','ignore'],maxBuffer:16*1024*1024});}
+ catch {return null;}
+};
+function baselineEntries(root = baselineDir) {
+ return ['dev','4x','community'].flatMap(group => {
+ const meta =
JSON.parse(fs.readFileSync(path.join(root,`${group}.json`),'utf8'));
+ return meta.entryFiles.flatMap(shard =>
JSON.parse(fs.readFileSync(path.join(root,shard),'utf8')).entries);
+ });
+}
+function sidebarAudit(native,root,scope) {
+ const result = [];
+ for (const group of scope.groups) {
+ const plugin = native.labels.find(item => item.pluginId ===
group.pluginId);
+ assert(plugin, `Missing native sidebar plugin ${group.pluginId}`);
+ const versionFile = plugin.files.find(file => file.path ===
(group.version === 'current' ? 'current' : `version-${group.version}`));
+ assert(versionFile, `Missing native sidebar keys ${group.id}`);
+ const suffix = group.pluginId === 'default' ? '' :
`-${group.pluginId}`;
+ const dir =
path.join(root,`i18n/ja/docusaurus-plugin-content-docs${suffix}`);
+ const target = path.join(dir,`${versionFile.path}.json`);
+ const translated = fs.existsSync(target) ?
JSON.parse(fs.readFileSync(target,'utf8')) : {};
+ const sourceKeys = Object.entries(versionFile.content).filter(([name])
=> name.startsWith('sidebar.'));
+ const retained = scope.retainSidebarLabels || [];
+ for (const [name, value] of sourceKeys) {
+ const exception = retained.find(entry => entry.pluginId ===
group.pluginId && entry.version === group.version && entry.key === name);
+ if (exception && (!exception.reason?.trim() || value.message !==
translated[name]?.message))
+ throw new Error(`Invalid retained sidebar label ${name}`);
+ const state = !translated[name]?.message ? 'missing' :
translated[name].message === value.message
+ ? exception ? 'retained_technical' : 'untranslated' :
'translated';
+ result.push({group:group.id,
pluginId:group.pluginId,version:group.version,key:name,
+ source:value.message,translation:translated[name]?.message ||
null,state});
+ }
+ for (const exception of retained.filter(entry => entry.pluginId ===
group.pluginId && entry.version === group.version))
+ assert(sourceKeys.some(([name]) => name === exception.key),`Unused
sidebar exception ${exception.key}`);
+ }
+ return result;
+}
+async function audit({root,sourceRef,baseline = baselineDir,native}) {
+ const commit =
execFileSync('git',['rev-parse',`${sourceRef}^{commit}`],{cwd:root,encoding:'utf8'}).trim();
+
execFileSync('git',['diff','--quiet',commit,'--','i18n/ja','translations/state/ja'],{cwd:root});
+ const untracked =
execFileSync('git',['ls-files','--others','--exclude-standard','--','i18n/ja','translations/state/ja'],
+ {cwd:root,encoding:'utf8'}).trim();
+ assert(!untracked,`Uncommitted Japanese targets/state cannot count at
${commit}: ${untracked}`);
+ const scope =
JSON.parse(fs.readFileSync(path.join(root,'translations/scope/ja.json'),'utf8'));
+ validateScope(scope);
+ native ||= await collect(root);
+ const old = new Map(baselineEntries(baseline).map(page =>
[key(page),page]));
+ const current = new Set(), groups = [];
+ const files = group =>
execFileSync('git',['ls-tree','-r','--name-only',commit,'--',group.sourceRoot],{cwd:root,encoding:'utf8'})
+ .split('\n').filter(file => /\.(md|mdx)$/.test(file));
+ for (const group of scope.groups) {
+ const pages = native.pages.filter(page => page.pluginId ===
group.pluginId && page.version === group.version);
+ assert(pages.length,`Missing native group ${group.id}`);
+ const sourceFiles = files(group), seen = new Set(), entries = [];
+ for (const page of pages) {
+ assert(page.englishPath.startsWith(group.sourceRoot+'/'),`Wrong
source root: ${page.englishPath}`);
+ assert(sourceFiles.includes(page.englishPath),`Native page missing
at source ref: ${page.englishPath}`);
+
assert.equal(hash(refContent(root,commit,page.englishPath)),hash(fs.readFileSync(path.join(root,page.englishPath),'utf8')),
+ `Source differs from ${commit}: ${page.englishPath}`);
+ assert(!seen.has(page.englishPath),`Duplicate native source
${page.englishPath}`);
+ seen.add(page.englishPath); current.add(key(page));
+ const previous = old.get(key(page));
+ const disposition = page.draft ? 'draft' :
scope.retainEnglish.some(entry => key(entry) === key(page)) ? 'retain_english'
: 'translate';
+ const state = disposition !== 'translate' ? disposition :
page.status === 'untranslated' ? 'untranslated'
+ : page.status === 'current' && page.reviewStatus !==
'unverified' && page.useTarget
+ && refContent(root,commit,page.targetPath) !== null
+ && hash(refContent(root,commit,page.targetPath)) ===
page.targetHash
+ &&
refContent(root,commit,'translations/state/ja/docs.json') !== null ? 'current'
: page.status;
+
entries.push({pluginId:page.pluginId,version:page.version,docId:page.docId,sourcePath:page.sourcePath,
+
targetPath:page.targetPath,sourceHash:page.sourceHash,targetHash:page.targetHash,
+
sourceCommit:commit,permalink:page.permalink,reviewStatus:page.reviewStatus,state,
+ change:!previous ? 'new' : previous.sourcePath !==
page.sourcePath ? 'moved'
+ : previous.sourceHash !== page.sourceHash ?
'source_changed' : 'unchanged'});
+ }
+ const unregistered = sourceFiles.filter(file => !seen.has(file));
+ const required = entries.filter(entry =>
!['draft','retain_english'].includes(entry.state));
+ groups.push({id:group.id,pluginId:group.pluginId,version:group.version,
+
counts:{registered:entries.length,required:required.length,current:required.filter(entry
=> entry.state === 'current').length,
+ pending:required.filter(entry => entry.state !==
'current').length,new:entries.filter(entry => entry.change === 'new').length,
+ changed:entries.filter(entry => entry.change ===
'source_changed').length,moved:entries.filter(entry => entry.change ===
'moved').length,
+ drafts:entries.filter(entry => entry.state ===
'draft').length,unregistered:unregistered.length},
+ unregistered,entries});
+ }
+ const removed = [...old.values()].filter(page => !current.has(key(page)))
+ .map(page =>
({pluginId:page.pluginId,version:page.version,docId:page.docId,sourcePath:page.sourcePath,
+ likelyMoveTo: groups.flatMap(group => group.entries).find(entry =>
entry.sourceHash === page.sourceHash
+ && entry.pluginId === page.pluginId && entry.version ===
page.version)?.sourcePath || null}));
+ const sidebar = sidebarAudit(native,root,scope);
+ const labels =
Object.fromEntries(['translated','untranslated','missing','retained_technical']
+ .map(state => [state,sidebar.filter(entry => entry.state ===
state).length]));
+ const routingInputs =
['docusaurus.config.js','versions.json','sidebars.ts','sidebarsCommunity.json','versioned_sidebars/version-4.x-sidebars.json',
+ 'translations/config.json','translations/scope/ja.json'];
+ routingInputs.push('translations/state/ja/docs.json');
+ for (const file of routingInputs)
assert.equal(refContent(root,commit,file),fs.readFileSync(path.join(root,file),'utf8'),
+ `Routing input differs from source ref: ${file}`);
+ return
{schemaVersion:1,sourceCommit:commit,baselineCommit:JSON.parse(fs.readFileSync(path.join(baseline,'summary.json'),'utf8')).sourceCommit,
+ routingInputHashes:Object.fromEntries(routingInputs.map(file =>
[file,hash(refContent(root,commit,file))])),
+ groups,removed,labels,sidebar,reports:native.reports};
+}
+if (require.main === module) {
+ const args = process.argv.slice(2), option = name =>
args[args.indexOf(name)+1];
+ const root = path.resolve(__dirname,'../..');
+ audit({root,sourceRef:args.includes('--source-ref') ?
option('--source-ref') : 'HEAD'})
+ .then(result => {const output = JSON.stringify(result,null,2)+'\n';
+ if(args.includes('--output'))
fs.writeFileSync(option('--output'),output); else process.stdout.write(output);
+
console.log(JSON.stringify({sourceCommit:result.sourceCommit,groups:result.groups.map(({id,counts})
=> ({id,counts})),
+ labels:result.labels,removed:result.removed.length}));
+ if(args.includes('--fail-on-pending') && (result.groups.some(group
=> group.counts.pending) || result.labels.missing ||
result.labels.untranslated)) process.exitCode=1;
+ }).catch(error => {console.error(error);process.exitCode=1;});
+}
+module.exports = {audit,sidebarAudit};
diff --git a/scripts/i18n/build-scope.cjs b/scripts/i18n/build-scope.cjs
new file mode 100644
index 00000000000..6d2e1c67604
--- /dev/null
+++ b/scripts/i18n/build-scope.cjs
@@ -0,0 +1,61 @@
+const fs = require('node:fs');
+const path = require('node:path');
+const {execFileSync} = require('node:child_process');
+const {locales} = require('../../config/i18n');
+
+function buildScope(files, versions) {
+ const active = ['current', ...versions];
+ const wanted = new Set(), languages = new Set(), json = new Set(), retired
= new Set();
+ let tier = 'skip';
+ const build = () => {if (tier === 'skip') tier = 'build';};
+ const shared = () => {locales.forEach(locale => languages.add(locale));
build();};
+ for (const file of files) {
+ let match;
+ if (['versions.json', 'config/versions-plugin.js'].includes(file))
{tier = 'full'; shared();}
+ else if (file.startsWith('docs/')) {languages.add('en');
wanted.add('current'); build();}
+ else if ((match = file.match(/^versioned_docs\/version-([^/]+)\//)))
{languages.add('en'); wanted.add(match[1]); build();}
+ else if ((match = file.match(/^i18n\/([^/]+)\/(.+)$/))) {
+ if (!locales.includes(match[1])) throw new Error(`Unconfigured
locale: ${match[1]}`);
+ const locale = match[1], content = match[2];
+ if (content.endsWith('.json')) {
+ json.add(file);
+ // Japanese dictionary changes need real rendering validation.
+ if (locale === 'ja') {languages.add(locale); build();}
+ } else {
+ languages.add(locale); build();
+ const version =
content.match(/^docusaurus-plugin-content-docs\/(current|version-([^/]+))\//);
+ if (version) wanted.add(version[2] || 'current');
+ }
+ } else if ((match =
file.match(/^versioned_sidebars\/version-(.+)-sidebars.json$/))) {shared();
wanted.add(match[1]);}
+ else if (file === 'sidebars.ts') {shared(); wanted.add('current');}
+ else if (/^(blog|community|releasenotes|course)\//.test(file))
{languages.add('en'); build();}
+ else if (/^sidebars(Community|Releases|Course)\./.test(file)) shared();
+ else if (/^(src|config)\//.test(file) ||
['docusaurus.config.js','package.json','yarn.lock','tailwind.config.js','tsconfig.json'].includes(file)
+ || /^scripts\/(key-features|last-update)\//.test(file) ||
/^scripts\/(patch-search-tokenize.js|build-locales.cjs|build-locale.cjs)$/.test(file)
+ ||
/^scripts\/i18n\/(docs-plugin|status-plugin|state|structure)\.cjs$/.test(file))
shared();
+ else if (['translations/config.json',
'translations/state/ja/docs.json'].includes(file)) {languages.add('ja');
build();}
+ else if (file === 'static/.htaccess') shared();
+ else if
(/^(static|\.github|\.claude|\.agents|\.vscode|\.idea|scripts|website-quality-governance|translations|plan-doc)\//.test(file)
+ || /\.(py|sh)$/.test(file)
+ ||
/^(README.md|AGENTS.md|TOC.md|LICENSE|NOTICE|store_format_v3.md|cspell.json|author.yml|buildVersions.json|\.gitignore|\.prettier.*|\.eslintrc.*)$/.test(file))
{}
+ else {languages.add('en'); build();}
+ }
+ for (const version of wanted) if (!active.includes(version))
{wanted.delete(version); retired.add(version);}
+ return {tier, skip_build: tier === 'skip', need_full_build: tier ===
'full',
+ docs_versions: tier === 'full' ? '' : active.filter(version =>
wanted.has(version)).join(',') || versions[0] || 'current',
+ locales: locales.filter(locale => languages.has(locale)), enable_ja:
languages.has('ja'),
+ json: [...json], retired: [...retired]};
+}
+if (require.main === module) {
+ const root = process.cwd();
+ const base = process.argv[2];
+ if (!base) throw new Error('Usage: build-scope.cjs <base SHA>');
+ const files = execFileSync('git', ['diff', '--name-only', base, 'HEAD'],
{encoding:'utf8'}).trim().split('\n').filter(Boolean);
+ const plan = buildScope(files,
JSON.parse(fs.readFileSync(path.join(root,'versions.json'))));
+ for (const file of plan.json) if (fs.existsSync(file))
JSON.parse(fs.readFileSync(file,'utf8'));
+ console.log(JSON.stringify({files, ...plan}, null, 2));
+ if (process.env.GITHUB_OUTPUT) fs.appendFileSync(process.env.GITHUB_OUTPUT,
+ Object.entries({...plan, locale_args: plan.locales.map(locale =>
`--locale ${locale}`).join(' '), locale_count: plan.locales.length})
+ .filter(([key]) =>
!['json','retired','locales'].includes(key)).map(([key,value]) =>
`${key}=${value}\n`).join(''));
+}
+module.exports = {buildScope};
diff --git a/scripts/i18n/check-translations.cjs
b/scripts/i18n/check-translations.cjs
new file mode 100644
index 00000000000..f1fdc72b6d9
--- /dev/null
+++ b/scripts/i18n/check-translations.cjs
@@ -0,0 +1,82 @@
+// Only edited translations/state are blocking. Source-only freshness is a
report.
+const fs = require('node:fs');
+const path = require('node:path');
+const {execFileSync} = require('node:child_process');
+const {identity, hash, loadRecords} = require('./state.cjs');
+const {structure, alignHeadings} = require('./structure.cjs');
+const {collect} = require('./inventory.cjs');
+const {protectionSignature} = require('./protection.cjs');
+
+async function checkTranslations({root, pages, reports = [], files, records,
previousRecords = new Map()}) {
+ const changed = new Set(files);
+ const selected = pages.filter(page => changed.has(page.targetPath)
+ || (changed.has('translations/state/ja/docs.json')
+ && JSON.stringify(records.get(identity(page))) !==
JSON.stringify(previousRecords.get(identity(page)))));
+ const errors = [];
+ const exceptionFile =
path.join(root,'translations/protection-exceptions/ja.json');
+ const exceptions = fs.existsSync(exceptionFile) ?
JSON.parse(fs.readFileSync(exceptionFile,'utf8')).pages : [];
+ const baselineState =
JSON.parse(execFileSync('git',['show','3379cb75733e1d8c453f6aae7b44b1fae5497ae2:translations/state/ja/docs.json'],
+ {cwd:root,encoding:'utf8'}));
+ const approved = new Map(baselineState.pages.map(record =>
[identity(record),record]));
+ for (const entry of exceptions) {
+ const record = approved.get(identity(entry));
+ const mergedTarget = record &&
execFileSync('git',['show',`3379cb75733e1d8c453f6aae7b44b1fae5497ae2:${record.targetPath}`],
+ {cwd:root,encoding:'utf8'});
+ const repaired = entry.repairedFromTargetHash === record?.targetHash
+ && mergedTarget?.includes('(ai/ai-overview.md)')
+ &&
hash(mergedTarget.replace('(ai/ai-overview.md)','(../ai/ai-overview.md)')) ===
entry.targetHash;
+ if (!record || record.sourceHash !== entry.sourceHash ||
+ !(record.targetHash === entry.targetHash || repaired) ||
!entry.reason?.trim())
+ errors.push({path:exceptionFile,message:'Protection exception is
not bound to the merged T01 PoC bytes'});
+ }
+ for (const page of selected) {
+ const record = records.get(identity(page));
+ if (!record && !fs.existsSync(path.join(root, page.targetPath)))
continue; // Intentional removal of both.
+ try {
+ if (page.status !== 'current' || !record) throw new
Error(`Translation must match its current source/state (${page.status})`);
+ const source =
fs.readFileSync(path.join(root,page.sourcePath),'utf8');
+ const target =
fs.readFileSync(path.join(root,page.targetPath),'utf8');
+ const ext = path.extname(page.sourcePath);
+ const parsed = await structure(source, ext);
+ const aligned = await alignHeadings(parsed, target, ext);
+ const exception = exceptions.find(entry => identity(entry) ===
identity(page)
+ && entry.sourceHash === page.sourceHash && entry.targetHash
=== page.targetHash && entry.reason?.trim());
+ if (!exception && protectionSignature(await structure(aligned,
ext)) !== protectionSignature(parsed)) {
+ throw new Error('Protected code, links, numbers, MDX or
metadata differ from the source');
+ }
+ if (record.sourceTechnicalHash !== parsed.technicalHash) throw new
Error('sourceTechnicalHash does not match the current source');
+ const committed = execFileSync('git', ['show',
`${record.sourceCommit}:${page.sourcePath}`],
+ {cwd:root, encoding:'utf8', stdio:['ignore','pipe','ignore'],
maxBuffer:16*1024*1024});
+ if (hash(committed) !== record.sourceHash) throw new
Error('sourceCommit/sourcePath does not contain the recorded sourceHash');
+ } catch (error) {errors.push({path:page.targetPath,
message:error.message});}
+ }
+ const known = new Set(pages.map(page => page.targetPath));
+ for (const file of files.filter(file =>
/^i18n\/ja\/docusaurus-plugin-content-docs[^/]*\/(current|version-[^/]+)\/.+\.(md|mdx)$/.test(file)))
{
+ if (fs.existsSync(path.join(root,file)) && !known.has(file))
errors.push({path:file, message:'Target has no registered source in the same
plugin/version'});
+ }
+ if (changed.has('translations/state/ja/docs.json')) for (const [key,
record] of records) {
+ if (JSON.stringify(record) !==
JSON.stringify(previousRecords.get(key)) && !pages.some(page => identity(page)
=== key)) {
+ errors.push({path:record.targetPath, message:'Edited state has no
native source identity'});
+ }
+ }
+ return {checked: selected.length, errors, freshness:
Object.fromEntries([...new Set(pages.map(page => page.status))]
+ .map(status => [status, pages.filter(page => page.status ===
status).length])), reports};
+}
+async function main() {
+ const root = path.resolve(__dirname,'../..');
+ const args = process.argv.slice(2), base = args[args.indexOf('--base')+1];
+ if (!args.includes('--base')) throw new Error('Usage:
check-translations.cjs --base <PR merge base>');
+ const files =
execFileSync('git',['diff','--name-only',base,'HEAD'],{cwd:root,encoding:'utf8'}).trim().split('\n').filter(Boolean);
+ const records = loadRecords(root,'ja');
+ let previousRecords = new Map();
+ try {
+ const previous =
JSON.parse(execFileSync('git',['show',`${base}:translations/state/ja/docs.json`],{cwd:root,encoding:'utf8'}));
+ previousRecords = new Map(previous.pages.map(record =>
[identity(record),record]));
+ } catch (error) {if (!String(error.message).includes('does not exist'))
throw error;}
+ const native = await collect(root);
+ const result = await
checkTranslations({root,...native,files,records,previousRecords});
+ console.log(JSON.stringify(result,null,2));
+ if (result.errors.length) process.exitCode = 1;
+}
+if (require.main === module) main().catch(error =>
{console.error(error);process.exitCode=1;});
+module.exports = {checkTranslations};
diff --git a/scripts/i18n/inventory.cjs b/scripts/i18n/inventory.cjs
index 9f277a80312..ffe4a74a011 100644
--- a/scripts/i18n/inventory.cjs
+++ b/scripts/i18n/inventory.cjs
@@ -270,4 +270,4 @@ async function main() {
}
if (require.main === module) main().catch(error => {console.error(error);
process.exitCode = 1;});
-module.exports = {validateScope, sidebarOrder, buildGroup, mergedCurrent,
renderBatches};
+module.exports = {validateScope, sidebarOrder, buildGroup, mergedCurrent,
renderBatches, collect};
diff --git a/scripts/i18n/paths.cjs b/scripts/i18n/paths.cjs
new file mode 100644
index 00000000000..576ab7413d3
--- /dev/null
+++ b/scripts/i18n/paths.cjs
@@ -0,0 +1,10 @@
+function sourceForLocalized(file) {
+ const match =
file.match(/^i18n\/[^/]+\/docusaurus-plugin-content-docs(-[^/]+)?\/(current|version-([^/]+))\/(.*)$/);
+ if (!match) return null;
+ const plugin = match[1] || '', version = match[3], relative = match[4];
+ const roots = {'':'docs', '-community':'community',
'-releases':'releasenotes', '-course':'course'};
+ if (!roots[plugin]) return null;
+ if (version) return plugin === '' ?
`versioned_docs/version-${version}/${relative}` : null;
+ return `${roots[plugin]}/${relative}`;
+}
+module.exports = {sourceForLocalized};
diff --git a/scripts/i18n/protection.cjs b/scripts/i18n/protection.cjs
new file mode 100644
index 00000000000..f4b4b42e254
--- /dev/null
+++ b/scripts/i18n/protection.cjs
@@ -0,0 +1,32 @@
+const cheerio = require('cheerio');
+const {hash} = require('./state.cjs');
+const translatable = new Set(['title', 'description', 'sidebar_label',
'keywords', 'language']);
+const visibleAttribute = name => ['alt', 'title', 'aria-label', 'description',
'label', 'caption'].includes(name);
+
+// Keep the T02 freshness hash unchanged. This separate signature permits prose
+// in HTML and visible string attributes while protecting executable MDX.
+function protectionSignature(parsed) {
+ const nodes = [], numbers = [];
+ for (const node of parsed.protectedNodes) {
+ if (node[0] === 'numbers') {numbers.push(...node[1]); continue;}
+ if (node[0] === 'html' && !node[3].trim().startsWith('<!--')) {
+ const $ = cheerio.load(node[3], {decodeEntities:false}, false);
+ $('*').contents().each((i, child) => {
+ if (child.type === 'text') {
+ numbers.push(...(child.data.match(/\d+(?:\.\d+)*/g) ||
[]));
+ child.data = '';
+ }
+ });
+ $('*').each((i, element) =>
Object.keys(element.attribs).forEach(name => {
+ if (visibleAttribute(name)) element.attribs[name] = '';
+ }));
+ nodes.push(['html', $.html()]);
+ } else if (node[0].startsWith('mdxJsx')) {
+ nodes.push([...node.slice(0,2), node[2].map(attribute =>
visibleAttribute(attribute.name)
+ && typeof attribute.value === 'string' ? {...attribute,
value:''} : attribute)]);
+ } else nodes.push(node);
+ }
+ return
hash(JSON.stringify([Object.fromEntries(Object.entries(parsed.technicalMetadata).filter(([key])
=> !translatable.has(key))),
+ nodes, numbers.sort(), parsed.headings.map(({depth,id}) =>
({depth,id}))]));
+}
+module.exports = {protectionSignature};
diff --git a/scripts/i18n/search-plugin.cjs b/scripts/i18n/search-plugin.cjs
new file mode 100644
index 00000000000..fbcdc843e7c
--- /dev/null
+++ b/scripts/i18n/search-plugin.cjs
@@ -0,0 +1,102 @@
+// Reuse the installed search plugin for EN/ZH. Japanese needs an index of
+// reviewed, current targets only; its native postBuild includes fallback HTML.
+const fs = require('node:fs');
+const path = require('node:path');
+const {createHash} = require('node:crypto');
+const nativeModule =
require('@yang1666204/docusaurus-search-local/dist/server/server');
+const native = nativeModule.default;
+const {processPluginOptions} =
require('@yang1666204/docusaurus-search-local/dist/server/server/utils/processPluginOptions');
+const {processDocInfos} =
require('@yang1666204/docusaurus-search-local/dist/server/server/utils/processDocInfos');
+const {scanDocuments} =
require('@yang1666204/docusaurus-search-local/dist/server/server/utils/scanDocuments');
+const {buildIndex} =
require('@yang1666204/docusaurus-search-local/dist/server/server/utils/buildIndex');
+
+const contexts = ['docs/dev', 'docs/4.x', 'community'];
+const normalize = url => url.replace(/\/+$/, '');
+function groupFor(page) {
+ if (page.pluginId === 'default' && page.version === 'current') return
'docs/dev';
+ if (page.pluginId === 'default' && page.version === '4.x') return
'docs/4.x';
+ if (page.pluginId === 'community' && page.version === 'current') return
'community';
+ return null;
+}
+function japaneseOptions(options) {
+ return {...options, language:['ja'],
docsRouteBasePath:['docs','community','ja/docs','ja/community'],
+ searchContextByPaths:contexts.map(value =>
({path:value,label:{ja:value === 'community' ? 'コミュニティ' : value === 'docs/dev'
? 'Dev ドキュメント' : '4.x ドキュメント'}})),
+ useAllContextsWithNoSearchContext:true, ignoreFiles:[]};
+}
+function targetFingerprint(root) {
+ const digester = createHash('sha256');
+ function scan(dir) {
+ if (!fs.existsSync(dir)) return;
+ for (const entry of
fs.readdirSync(dir,{withFileTypes:true}).sort((a,b) =>
a.name.localeCompare(b.name))) {
+ const file = path.join(dir,entry.name);
+ if (entry.isDirectory()) scan(file);
+ else if (/\.(md|mdx|json)$/.test(entry.name)) {
+ digester.update(path.relative(root,file));
+ digester.update(fs.readFileSync(file));
+ }
+ }
+ }
+ scan(path.join(root,'i18n/ja'));
+ scan(path.join(root,'translations/state/ja'));
+ for (const dir of ['docs','versioned_docs/version-4.x','community'])
scan(path.join(root,dir));
+ return digester.digest('hex').slice(0,16);
+}
+function indexFilename(context) {
+ const generated =
fs.readFileSync(path.join(context.generatedFilesDir,'@yang1666204/docusaurus-search-local/default/generated-constants.js'),'utf8');
+ const match = generated.match(/export const searchIndexUrl = ("[^"\n]+")/);
+ if (!match) throw new Error('Search index filename missing from generated
constants');
+ const value = JSON.parse(match[1]).split('?')[0];
+ if (!/^search-index\{dir\}(?:-[a-f0-9]+)?\.json$/.test(value)) throw new
Error('Unexpected search index filename');
+ return value;
+}
+function japanesePages(plugins) {
+ const selected = new Map();
+ for (const plugin of plugins.filter(item => item.name ===
'docusaurus-plugin-content-docs')) {
+ for (const page of plugin.content?.translationManifest?.pages || []) {
+ const group = groupFor(page);
+ if (group && page.status === 'current' && page.useTarget &&
page.indexable)
+ selected.set(normalize(page.permalink), group);
+ }
+ }
+ return selected;
+}
+async function buildJapaneseIndex(buildData, config, filename) {
+ const selected = japanesePages(buildData.plugins || []);
+ const nativePaths = processDocInfos(buildData, config).flatMap(item =>
item.paths);
+ const unique = new Map();
+ for (const item of nativePaths) if (selected.has(normalize(item.url)))
unique.set(normalize(item.url), item);
+ if (unique.size !== selected.size) throw new Error(`Japanese search routes
missing in static output: ${selected.size - unique.size}`);
+ const allDocuments = await scanDocuments([...unique.values()], config);
+ const write = (group, documents) => {
+ const name = filename.replace('{dir}', group ?
`-${group.replace(/\//g,'-')}` : '');
+ fs.writeFileSync(path.join(buildData.outDir,name),
JSON.stringify(buildIndex(documents,config)));
+ };
+ for (const group of contexts) write(group, allDocuments.map(documents =>
documents.filter(doc => selected.get(normalize(doc.u)) === group)));
+ write('',allDocuments);
+
fs.writeFileSync(path.join(buildData.outDir,'japanese-search-summary.json'),
JSON.stringify({schemaVersion:1,
+ indexedPages:selected.size,
contexts:Object.fromEntries(contexts.map(group =>
[group,[...selected.values()].filter(value => value ===
group).length]))})+'\n');
+}
+function searchPlugin(context, options) {
+ if (context.i18n.currentLocale !== 'ja') return native(context,options);
+ const jaOptions = japaneseOptions(options);
+ const plugin = native(context,jaOptions);
+ const config = processPluginOptions(jaOptions,context);
+ const constants =
path.join(context.generatedFilesDir,'@yang1666204/docusaurus-search-local/default/generated-constants.js');
+ const generated = fs.readFileSync(constants,'utf8');
+ const fingerprint = targetFingerprint(context.siteDir);
+ const updated = generated.replace(/(export const searchIndexUrl =
)("[^"\n]+")/,
+ (_,prefix,value) => {
+ const url = JSON.parse(value);
+ return prefix + JSON.stringify(`${url}${url.includes('?') ? '&' :
'?'}ja=${fingerprint}`);
+ });
+ if (updated === generated) throw new Error('Could not fingerprint Japanese
search index');
+ fs.writeFileSync(constants,updated);
+ const filename = indexFilename(context);
+ return {...plugin, postBuild: buildData =>
buildJapaneseIndex(buildData,config,filename)};
+}
+module.exports = searchPlugin;
+module.exports.validateOptions = nativeModule.validateOptions;
+module.exports.japaneseOptions = japaneseOptions;
+module.exports.groupFor = groupFor;
+module.exports.japanesePages = japanesePages;
+module.exports.buildJapaneseIndex = buildJapaneseIndex;
diff --git a/scripts/i18n/structure.cjs b/scripts/i18n/structure.cjs
index 5aa2c484527..25f0a549dc0 100644
--- a/scripts/i18n/structure.cjs
+++ b/scripts/i18n/structure.cjs
@@ -61,7 +61,7 @@ async function structure(raw, extension, anchorsMaintainCase
= false) {
.filter(([key]) => !['title', 'description',
'sidebar_label'].includes(key))
.sort(([a], [b]) => a.localeCompare(b)));
return {
- content, headings, format,
+ content, headings, format, protectedNodes, technicalMetadata,
technicalHash: hash(JSON.stringify([technicalMetadata, protectedNodes,
headings.map(({depth, id}) => ({depth, id}))])),
};
}
diff --git a/scripts/i18n/verify-search.cjs b/scripts/i18n/verify-search.cjs
new file mode 100644
index 00000000000..c9b0746eaea
--- /dev/null
+++ b/scripts/i18n/verify-search.cjs
@@ -0,0 +1,98 @@
+const assert = require('node:assert/strict');
+const fs = require('node:fs');
+const path = require('node:path');
+const http = require('node:http');
+const {chromium} = require('playwright');
+
+async function verify({site,devOnly = false}) {
+ const manifest =
JSON.parse(fs.readFileSync(path.join(site,'ja','japanese-search-summary.json'),'utf8'));
+
assert.deepEqual(Object.keys(manifest.contexts).sort(),['community','docs/4.x','docs/dev'].sort());
+
assert.equal(manifest.indexedPages,Object.values(manifest.contexts).reduce((a,b)=>a+b,0));
+ const indexFiles =
['search-index.json','search-index-docs-dev.json','search-index-docs-4.x.json','search-index-community.json'];
+ const indexes = Object.fromEntries(indexFiles.map(file =>
[file,JSON.parse(fs.readFileSync(path.join(site,'ja',file),'utf8'))]));
+ for (const [file,index] of Object.entries(indexes)) {
+ for (const section of index) for (const doc of section.documents) {
+ assert(doc.u.startsWith('/ja/'),`Cross-locale result in ${file}:
${doc.u}`);
+ if (file === 'search-index-community.json')
assert(doc.u.startsWith('/ja/community/'));
+ if (file === 'search-index-docs-dev.json')
assert(doc.u.startsWith('/ja/docs/dev/'));
+ if (file === 'search-index-docs-4.x.json')
assert(doc.u.startsWith('/ja/docs/4.x/'));
+ }
+ }
+ const server = http.createServer((req,res) => {
+ const pathname = decodeURIComponent(new
URL(req.url,'http://local').pathname);
+ const local = path.join(site,pathname.endsWith('/') ?
`${pathname}index.html` : pathname);
+ if (!local.startsWith(site+path.sep) || !fs.existsSync(local) ||
!fs.statSync(local).isFile()) {res.writeHead(404);res.end();return;}
+
res.setHeader('Content-Type',local.endsWith('.html')?'text/html':local.endsWith('.json')?'application/json':local.endsWith('.js')?'application/javascript':local.endsWith('.css')?'text/css':'application/octet-stream');
+ fs.createReadStream(local).pipe(res);
+ });
+ await new Promise(resolve => server.listen(0,'127.0.0.1',resolve));
+ const origin = `http://127.0.0.1:${server.address().port}`;
+ const browser = await chromium.launch({headless:true});
+ const cases = [
+ {locale:'ja',ctx:'docs/dev',query:'データベース',path:'/ja/docs/dev/'},
+ {locale:'ja',ctx:'docs/dev',query:'分析',path:'/ja/docs/dev/'},
+ {locale:'ja',ctx:'docs/dev',query:'SQL',path:'/ja/docs/dev/'},
+ {locale:'ja',ctx:'docs/dev',query:'match',path:'/ja/docs/dev/'},
+ {locale:'ja',ctx:'docs/dev',query:'query_vector',path:'/ja/docs/dev/'},
+ {locale:'ja',ctx:'docs/4.x',query:'SQL',path:'/ja/docs/4.x/'},
+ {locale:'ja',ctx:'community',query:'コミュニティ',path:'/ja/community/'},
+ ].filter(item => !devOnly || item.ctx !== 'docs/4.x');
+ const results = [];
+ try {
+ for (const item of cases) {
+ const page = await browser.newPage();
+ const requests = [], errors = [];
+ page.on('request',request =>
{if(request.url().includes('search-index'))requests.push(new
URL(request.url()).pathname);});
+ page.on('pageerror',error => errors.push(error.message));
+ page.on('response',response => {if(response.status() >= 400 &&
response.url().startsWith(origin)) errors.push(`${response.status()}
${response.url()}`);});
+ const url = `${origin}/ja/search/?${new
URLSearchParams({ctx:item.ctx,q:item.query})}`;
+ const response = await page.goto(url);
+ assert.equal(response.status(),200,url);
+ try {await page.waitForFunction(() =>
document.querySelectorAll('section article a').length >
0,null,{timeout:30000});}
+ catch (error) {throw new Error(`${item.ctx}: ${error.message};
requests=${requests}; errors=${errors}; body=${(await
page.locator('body').innerText()).slice(0,500)}`);}
+ const links = await page.locator('section article
a').evaluateAll(nodes => nodes.map(node => node.getAttribute('href')));
+ assert(links.length && links.every(link =>
link.startsWith(item.path)),`${item.ctx} leaked a result: ${links.join(',')}`);
+ assert(requests.some(request =>
request.includes(`search-index-${item.ctx.replace(/\//g,'-')}`)),`Wrong worker
index: ${requests}`);
+ assert.deepEqual(errors,[],`Search page errors: ${item.ctx}`);
+
results.push({...item,indexRequests:requests,results:links.length});
+ await page.close();
+ }
+ const bar = await browser.newPage();
+ const barRequests = [];
+ bar.on('request',request =>
{if(request.url().includes('search-index')) barRequests.push(new
URL(request.url()).pathname);});
+ await bar.goto(`${origin}/ja/docs/dev/getting-started/intro/`);
+ const input =
bar.locator('input[aria-label="Search"]:visible').first();
+ await input.focus();
+ await bar.waitForTimeout(1200);
+ await input.fill('query_vector');
+ const suggestion = bar.locator('[class*="suggestion_"]');
+ await suggestion.first().waitFor({timeout:30000});
+ assert(barRequests.some(request =>
request.includes('search-index-docs-dev')),`SearchBar used wrong index:
${barRequests}`);
+ results.push({searchBarQuery:'query_vector',suggestions:await
suggestion.count(),indexRequests:barRequests});
+ await bar.close();
+
+ const mobile = await
browser.newPage({viewport:{width:390,height:844}});
+ await mobile.goto(`${origin}/ja/search/?q=SQL&ctx=docs/dev`);
+ await mobile.locator('section article
a').first().waitFor({timeout:30000});
+ const box = await mobile.locator('input[type="search"]').boundingBox();
+ assert(box && box.x >= 0 && box.x + box.width <= 391,`Mobile search
input overflow: ${JSON.stringify(box)}`);
+ results.push({mobileSearch:true,inputWidth:Math.round(box.width)});
+ await mobile.close();
+
+ const page = await browser.newPage();
+ await
page.goto(`${origin}/ja/search/?q=zzzzzzzz_t04_nomatch_98765&ctx=docs/dev`);
+ try {await
page.getByRole('link',{name:'英語の原文を検索'}).waitFor({timeout:30000});}
+ catch (error) {throw new Error(`Missing original-language search link:
${(await page.locator('body').innerText()).slice(-600)}; ${error.message}`);}
+ const original = await
page.getByRole('link',{name:'英語の原文を検索'}).getAttribute('href');
+ assert(original.startsWith('/search/?q='),original);
+ assert(new URL(original,origin).searchParams.get('version') ===
'docs/dev/',original);
+ results.push({noResultsOriginalSearch:original});
+ await page.close();
+ } finally {await browser.close(); await new Promise(resolve =>
server.close(resolve));}
+ return {manifest,results};
+}
+if (require.main === module) {
+ const args = process.argv.slice(2),site =
path.resolve(args[args.indexOf('--site')+1]);
+ verify({site,devOnly:args.includes('--dev-only')}).then(result =>
console.log(JSON.stringify(result,null,2))).catch(error =>
{console.error(error);process.exitCode=1;});
+}
+module.exports = {verify};
diff --git a/src/theme/SearchBar/SearchBar.jsx
b/src/theme/SearchBar/SearchBar.jsx
index 4fc41010bbe..4f7627e40da 100644
--- a/src/theme/SearchBar/SearchBar.jsx
+++ b/src/theme/SearchBar/SearchBar.jsx
@@ -28,6 +28,7 @@ import { searchResultLimits } from
"../../utils/proxiedGeneratedConstants";
import useIsDocPage from '@site/src/hooks/use-is-doc';
import { debounce } from '@site/src/utils/debounce';
import { getLocalePrefix, normalizePathname } from '@site/src/utils/locale';
+import {searchWithLocaleFallback} from '@site/src/utils/searchInput';
async function fetchAutoCompleteJS() {
const autoCompleteModule = await import('@easyops-cn/autocomplete.js');
@@ -69,7 +70,8 @@ export default function SearchBar({ handleSearchBarToggle }) {
// It returns undefined for non-docs pages
const activePlugin = useActivePlugin();
const [isDocsPage] = useIsDocPage(false);
- let versionUrl = getVersionUrl(baseUrl, location.pathname, locales,
currentLocale, defaultLocale);
+ let versionUrl = currentLocale === 'ja' ? baseUrl
+ : getVersionUrl(baseUrl, location.pathname, locales, currentLocale,
defaultLocale);
// For non-docs pages while using plugin-content-docs with custom ids,
// this will throw an error of:
@@ -128,7 +130,7 @@ export default function SearchBar({ handleSearchBarToggle
}) {
// useAllContextsWithNoSearchContext is false).
const normalized = normalizePathname(location.pathname, locales);
const parts = normalized.replace(/^\//, '').split('/').filter(Boolean);
- if (parts[0] === 'docs' && parts.length > 1 &&
VERSIONS.includes(parts[1])) {
+ if (currentLocale !== 'ja' && parts[0] === 'docs' && parts.length > 1
&& VERSIONS.includes(parts[1])) {
parts.splice(1, 1);
}
const uri = parts.join('/');
@@ -149,7 +151,7 @@ export default function SearchBar({ handleSearchBarToggle
}) {
prevSearchContext.current = nextSearchContext;
}
setSearchContext(nextSearchContext);
- }, [location.pathname, locales]);
+ }, [location.pathname, locales, currentLocale]);
const hidden = !!hideSearchBarWithNoSearchContext &&
Array.isArray(searchContextByPaths) && searchContext === '';
const loadIndex = useCallback(
@@ -216,7 +218,7 @@ export default function SearchBar({ handleSearchBarToggle
}) {
) {
params.set('ctx', searchContext);
}
- if (versionUrl !== baseUrl) {
+ if (currentLocale !== 'ja' && versionUrl !== baseUrl) {
if (!versionUrl.startsWith(baseUrl)) {
throw new Error(
`Version url '${versionUrl}' does not start with
base url '${baseUrl}', this is a bug of
\`@yang1666204/docusaurus-search-local\`, please report it.`,
@@ -265,7 +267,7 @@ export default function SearchBar({ handleSearchBarToggle
}) {
[
{
source: debounce(async (input, callback) => {
- const result = await searchByWorker(versionUrl,
searchContext, input, searchResultLimits);
+ const result = await
searchWithLocaleFallback(searchByWorker, versionUrl, searchContext, input,
searchResultLimits, currentLocale);
callback(result);
}, 300),
templates: {
diff --git a/src/theme/SearchPage/SearchPage.jsx
b/src/theme/SearchPage/SearchPage.jsx
index 1aff5a001d9..410a14ba47d 100644
--- a/src/theme/SearchPage/SearchPage.jsx
+++ b/src/theme/SearchPage/SearchPage.jsx
@@ -18,6 +18,7 @@ import { Mark, searchContextByPaths,
useAllContextsWithNoSearchContext } from ".
import { searchResultLimits } from "../../utils/proxiedGeneratedConstants";
import styles from "./SearchPage.module.css";
import { normalizeContextByPath } from "../../utils/normalizeContextByPath";
+import {searchWithLocaleFallback} from "../../utils/searchInput";
export default function SearchPage() {
return (<Layout>
<SearchPageContent />
@@ -29,7 +30,7 @@ function SearchPageContent() {
const { searchValue, searchContext, searchVersion, updateSearchPath,
updateSearchContext, } = useSearchQuery();
const [searchQuery, setSearchQuery] = useState(searchValue);
const [searchResults, setSearchResults] = useState();
- const versionUrl = `${baseUrl}${searchVersion}`;
+ const versionUrl = currentLocale === 'ja' ? baseUrl :
`${baseUrl}${searchVersion}`;
const pageTitle = useMemo(() => searchQuery
? translate({
id: "theme.SearchPage.existingResultsTitle",
@@ -47,7 +48,7 @@ function SearchPageContent() {
updateSearchPath(searchQuery);
if (searchQuery) {
(async () => {
- const results = await searchByWorker(versionUrl,
searchContext, searchQuery, searchResultLimits);
+ const results = await searchWithLocaleFallback(searchByWorker,
versionUrl, searchContext, searchQuery, searchResultLimits, currentLocale);
setSearchResults(results);
})();
}
@@ -66,6 +67,10 @@ function SearchPageContent() {
}
}, [searchValue]);
const [searchWorkerReady, setSearchWorkerReady] = useState(false);
+ const sourceBaseUrl = currentLocale === 'ja' ? baseUrl.replace(/ja\/$/,
'') : baseUrl;
+ const englishSearch = `${sourceBaseUrl}search/?${new URLSearchParams({q:
searchQuery,
+ ...(searchContext?.startsWith('docs/') ? {ctx: 'docs'} : {}),
+ ...(searchContext === 'docs/dev' ? {version: 'docs/dev/'} :
{})}).toString()}`;
useEffect(() => {
async function doFetchIndexes() {
if (!Array.isArray(searchContextByPaths) ||
@@ -133,6 +138,7 @@ function SearchPageContent() {
message: "No documents were found",
description: "The paragraph for empty search result",
})}
+ {currentLocale === 'ja' && <> <a
href={englishSearch}>英語の原文を検索</a></>}
</p>) : (<p>
⚠️ The search index is only available when you run docusaurus
build!
diff --git a/src/utils/searchInput.js b/src/utils/searchInput.js
new file mode 100644
index 00000000000..d1d39947ff4
--- /dev/null
+++ b/src/utils/searchInput.js
@@ -0,0 +1,16 @@
+// TinySegmenter emits standalone '_' and '-' tokens for SQL identifiers, but
+// lunr removes those punctuation tokens from its index. Try the parts again
+// when the full identifier finds nothing; section-level indexing may also put
+// the useful term in a different section from a discarded stop word.
+export async function searchWithLocaleFallback(searchByWorker, versionUrl,
context, input, limit, locale) {
+ const normalized = locale === 'ja' ? input.replace(/[_-]/g, ' ') : input;
+ const results = await searchByWorker(versionUrl, context, normalized,
limit);
+ if (locale !== 'ja' || results.length || !/[_-]/.test(input)) return
results;
+ const parts = [...new Set(input.split(/[_-]+/).filter(part => part.length
> 1))]
+ .sort((a, b) => b.length - a.length);
+ for (const part of parts) {
+ const fallback = await searchByWorker(versionUrl, context, part,
limit);
+ if (fallback.length) return fallback;
+ }
+ return results;
+}
diff --git a/static/.htaccess b/static/.htaccess
index 342ab5ed593..01d7d727003 100644
--- a/static/.htaccess
+++ b/static/.htaccess
@@ -11,19 +11,28 @@
<IfModule mod_rewrite.c>
RewriteEngine On
+ RewriteCond %{REQUEST_URI} ^/docs/dev/
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^docs/dev(/.+)$ /docs/dev/getting-started/what-is-apache-doris
[R=301,L]
+ RewriteCond %{REQUEST_URI} ^/zh-CN/docs/dev/
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
- RewriteRule ^zh-CN/docs/dev(/.+)$
/zh-CN/docs/dev/getting-started/what-is-apache-doris [R=301,L]
+ RewriteRule ^(zh-CN/)?docs/dev(/.+)$
/zh-CN/docs/dev/getting-started/what-is-apache-doris [R=301,L]
- # Other zh-CN misses — render the zh-CN-locale 404 so the chrome/i18n
- # context (header/footer) is Chinese. URL is preserved.
+ # Preserve a real 404 status for missing Chinese routes.
RewriteCond %{REQUEST_URI} ^/zh-CN/
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
- RewriteRule ^ /zh-CN/404.html [L]
+ RewriteRule ^ - [R=404,L]
</IfModule>
+# The file is also copied inside each locale directory. Select the error page
+# by the original request URI while keeping the HTTP 404 status.
ErrorDocument 404 /404.html
+<If "%{REQUEST_URI} =~ m#^/zh-CN/#">
+ ErrorDocument 404 /zh-CN/404.html
+</If>
+<If "%{REQUEST_URI} =~ m#^/ja/# && -f '%{DOCUMENT_ROOT}/ja/404.html'">
+ ErrorDocument 404 /ja/404.html
+</If>
diff --git a/translations/protection-exceptions/ja.json
b/translations/protection-exceptions/ja.json
new file mode 100644
index 00000000000..49d573520ca
--- /dev/null
+++ b/translations/protection-exceptions/ja.json
@@ -0,0 +1,43 @@
+{
+ "schemaVersion": 1,
+ "pages": [
+ {
+ "pluginId": "default",
+ "version": "current",
+ "docId": "getting-started/what-is-apache-doris",
+ "locale": "ja",
+ "sourceHash":
"c1b950ba12598be8495190f93fd341dd646d98c837bf09c72ccc199c2ebb941e",
+ "targetHash":
"b22981d476dae36312c2d5631a1d6d10f6e7b28df8b66c96ecd48d8b27a7631f",
+ "reason": "T04 repairs the two merged T01 PoC AI links by adding the
missing ../. All other bytes are unchanged; the repair is checked against the
merged PoC target.",
+ "repairedFromTargetHash":
"565ac481aaafe738b0fd193d6f4617cff6698351cfc71c8d0b575de97634e4c4"
+ },
+ {
+ "pluginId": "default",
+ "version": "current",
+ "docId": "getting-started/intro",
+ "locale": "ja",
+ "sourceHash":
"5c870a2583db46c203dcd610b575d3888bf2ba442e41a58a8acb7177d9db4abe",
+ "targetHash":
"7b759880571be86dfa613ef71ea4b018e2c9b7f323eb9873c9596c1dafb0b33b",
+ "reason": "Merged T01 PoC: localized visible HTML/metadata, prose
quantities written as digits, and equivalent native route links. Exception is
bound to these exact source/target bytes; edited translations must pass
protection checks."
+ },
+ {
+ "pluginId": "default",
+ "version": "4.x",
+ "docId": "getting-started/what-is-apache-doris",
+ "locale": "ja",
+ "sourceHash":
"c1b950ba12598be8495190f93fd341dd646d98c837bf09c72ccc199c2ebb941e",
+ "targetHash":
"b22981d476dae36312c2d5631a1d6d10f6e7b28df8b66c96ecd48d8b27a7631f",
+ "reason": "T04 repairs the two merged T01 PoC AI links by adding the
missing ../. All other bytes are unchanged; the repair is checked against the
merged PoC target.",
+ "repairedFromTargetHash":
"565ac481aaafe738b0fd193d6f4617cff6698351cfc71c8d0b575de97634e4c4"
+ },
+ {
+ "pluginId": "community",
+ "version": "current",
+ "docId": "join-community",
+ "locale": "ja",
+ "sourceHash":
"c0fb07e339e759ca4bf071efabafb60ecb13e1978173dc88e3de79e7a32949b9",
+ "targetHash":
"69442d5fd55657623ba9b98cd6d2875fda3e031d30314877e7b6d2758ba45c8c",
+ "reason": "Merged T01 PoC: localized visible HTML/metadata, prose
quantities written as digits, and equivalent native route links. Exception is
bound to these exact source/target bytes; edited translations must pass
protection checks."
+ }
+ ]
+}
diff --git a/translations/scope/README.md b/translations/scope/README.md
index 67365ca072b..e78b6f7959c 100644
--- a/translations/scope/README.md
+++ b/translations/scope/README.md
@@ -52,3 +52,22 @@ These IDs belong to that snapshot. Record actual PRs, review
results and merge
SHAs as batches finish; generate future snapshots separately and reconcile new,
moved or changed pages explicitly. Do not renumber active assignments by
regenerating over this directory. T06 will supply incremental planning.
+
+## Continuous coverage audit (T04)
+
+Run `node scripts/i18n/audit-coverage.cjs --source-ref HEAD --output
/tmp/ja-coverage.json`
+from a clean checkout of the commit being checked. The audit loads the native
+plugins and sidebars again; the T03 snapshot is a comparison baseline only.
+It verifies source and routing inputs against that exact commit, then reports
+new, changed, moved and removed source nodes, current translations, pending
+translations, draft/unregistered exceptions, and every scoped sidebar message
+key. `retainSidebarLabels` in `ja.json` records technical names that remain
+English, with an explicit reason per plugin and version. A new sidebar key is
+never exempt just because it was absent from the T03 snapshot.
+
+The read-only `Japanese coverage audit` workflow runs weekly and supports
+manual dispatch; matching source PRs also get a report artifact. Pending
+Japanese work does not fail source PRs. Use `--fail-on-pending` only for the
+final T05 coverage gate. Source SHA and routing hashes in each report identify
+the exact checkout; after a failed or cancelled workflow, rerun it manually
+on current master. No model or bot credential is needed.
diff --git a/translations/scope/ja.json b/translations/scope/ja.json
index 7d60bba3c80..306374dc56c 100644
--- a/translations/scope/ja.json
+++ b/translations/scope/ja.json
@@ -3,9 +3,206 @@
"locale": "ja",
"maxBatchPages": 20,
"groups": [
- {"id": "DEV", "pluginId": "default", "version": "current", "sourceRoot":
"docs"},
- {"id": "4X", "pluginId": "default", "version": "4.x", "sourceRoot":
"versioned_docs/version-4.x"},
- {"id": "COMMUNITY", "pluginId": "community", "version": "current",
"sourceRoot": "community"}
+ {
+ "id": "DEV",
+ "pluginId": "default",
+ "version": "current",
+ "sourceRoot": "docs"
+ },
+ {
+ "id": "4X",
+ "pluginId": "default",
+ "version": "4.x",
+ "sourceRoot": "versioned_docs/version-4.x"
+ },
+ {
+ "id": "COMMUNITY",
+ "pluginId": "community",
+ "version": "current",
+ "sourceRoot": "community"
+ }
],
- "retainEnglish": []
+ "retainEnglish": [],
+ "retainSidebarLabels": [
+ {
+ "pluginId": "default",
+ "version": "current",
+ "key": "sidebar.docs.category.Flink Doris Connector",
+ "reason": "Connector product name"
+ },
+ {
+ "pluginId": "default",
+ "version": "current",
+ "key": "sidebar.docs.category.Iceberg Catalog",
+ "reason": "Catalog connector name"
+ },
+ {
+ "pluginId": "default",
+ "version": "current",
+ "key": "sidebar.docs.category.Paimon Catalog",
+ "reason": "Catalog connector name"
+ },
+ {
+ "pluginId": "default",
+ "version": "current",
+ "key": "sidebar.docs.category.Hudi Catalog",
+ "reason": "Catalog connector name"
+ },
+ {
+ "pluginId": "default",
+ "version": "current",
+ "key": "sidebar.docs.category.MaxCompute Catalog",
+ "reason": "Catalog connector name"
+ },
+ {
+ "pluginId": "default",
+ "version": "current",
+ "key": "sidebar.docs.category.DOG Stack",
+ "reason": "Named stack"
+ },
+ {
+ "pluginId": "default",
+ "version": "current",
+ "key": "sidebar.docs.category.information_schema",
+ "reason": "SQL system schema identifier"
+ },
+ {
+ "pluginId": "default",
+ "version": "current",
+ "key": "sidebar.docs.category.mysql",
+ "reason": "SQL system schema identifier"
+ },
+ {
+ "pluginId": "default",
+ "version": "current",
+ "key": "sidebar.docs.category.__internal_schema",
+ "reason": "SQL system schema identifier"
+ },
+ {
+ "pluginId": "default",
+ "version": "current",
+ "key": "sidebar.docs.category.HTTP API",
+ "reason": "Protocol/API name"
+ },
+ {
+ "pluginId": "default",
+ "version": "current",
+ "key": "sidebar.docs.category.FE HTTP API",
+ "reason": "Frontend HTTP API name"
+ },
+ {
+ "pluginId": "default",
+ "version": "current",
+ "key": "sidebar.docs.category.BE HTTP API",
+ "reason": "Backend HTTP API name"
+ },
+ {
+ "pluginId": "default",
+ "version": "current",
+ "key": "sidebar.docs.category.FAQ",
+ "reason": "Conventional documentation acronym"
+ },
+ {
+ "pluginId": "default",
+ "version": "current",
+ "key": "sidebar.docs.category.VARIANT",
+ "reason": "SQL data type"
+ },
+ {
+ "pluginId": "default",
+ "version": "current",
+ "key": "sidebar.docs.category.DML",
+ "reason": "SQL term"
+ },
+ {
+ "pluginId": "default",
+ "version": "current",
+ "key": "sidebar.docs.category.Table Stream",
+ "reason": "Product feature name"
+ },
+ {
+ "pluginId": "default",
+ "version": "4.x",
+ "key": "sidebar.docs.category.Iceberg Catalog",
+ "reason": "Catalog connector name"
+ },
+ {
+ "pluginId": "default",
+ "version": "4.x",
+ "key": "sidebar.docs.category.Paimon Catalog",
+ "reason": "Catalog connector name"
+ },
+ {
+ "pluginId": "default",
+ "version": "4.x",
+ "key": "sidebar.docs.category.Hudi Catalog",
+ "reason": "Catalog connector name"
+ },
+ {
+ "pluginId": "default",
+ "version": "4.x",
+ "key": "sidebar.docs.category.MaxCompute Catalog",
+ "reason": "Catalog connector name"
+ },
+ {
+ "pluginId": "default",
+ "version": "4.x",
+ "key": "sidebar.docs.category.DOG Stack",
+ "reason": "Named stack"
+ },
+ {
+ "pluginId": "default",
+ "version": "4.x",
+ "key": "sidebar.docs.category.information_schema",
+ "reason": "SQL system schema identifier"
+ },
+ {
+ "pluginId": "default",
+ "version": "4.x",
+ "key": "sidebar.docs.category.mysql",
+ "reason": "SQL system schema identifier"
+ },
+ {
+ "pluginId": "default",
+ "version": "4.x",
+ "key": "sidebar.docs.category.__internal_schema",
+ "reason": "SQL system schema identifier"
+ },
+ {
+ "pluginId": "default",
+ "version": "4.x",
+ "key": "sidebar.docs.category.HTTP API",
+ "reason": "Protocol/API name"
+ },
+ {
+ "pluginId": "default",
+ "version": "4.x",
+ "key": "sidebar.docs.category.FE HTTP API",
+ "reason": "Frontend HTTP API name"
+ },
+ {
+ "pluginId": "default",
+ "version": "4.x",
+ "key": "sidebar.docs.category.BE HTTP API",
+ "reason": "Backend HTTP API name"
+ },
+ {
+ "pluginId": "default",
+ "version": "4.x",
+ "key": "sidebar.docs.category.FAQ",
+ "reason": "Conventional documentation acronym"
+ },
+ {
+ "pluginId": "default",
+ "version": "4.x",
+ "key": "sidebar.docs.category.VARIANT",
+ "reason": "SQL data type"
+ },
+ {
+ "pluginId": "default",
+ "version": "4.x",
+ "key": "sidebar.docs.category.DML",
+ "reason": "SQL term"
+ }
+ ]
}
diff --git a/translations/state/ja/docs.json b/translations/state/ja/docs.json
index 97da48d32a4..ef8f44e2c62 100644
--- a/translations/state/ja/docs.json
+++ b/translations/state/ja/docs.json
@@ -12,7 +12,7 @@
"sourceCommit": "ec0bc51d17ff1c32df0241d61371f5f63db29b13",
"sourceHash":
"c1b950ba12598be8495190f93fd341dd646d98c837bf09c72ccc199c2ebb941e",
"sourceTechnicalHash":
"2c3ee90a60bae82e00653c8a088e174bf2ecd6eb8ce1bdda96d08cc50a160318",
- "targetHash":
"565ac481aaafe738b0fd193d6f4617cff6698351cfc71c8d0b575de97634e4c4",
+ "targetHash":
"b22981d476dae36312c2d5631a1d6d10f6e7b28df8b66c96ecd48d8b27a7631f",
"reviewStatus": "machine",
"ownership": "machine",
"translatedAt": "2026-10-11T19:57:10.000Z",
@@ -46,7 +46,7 @@
"sourceCommit": "ec0bc51d17ff1c32df0241d61371f5f63db29b13",
"sourceHash":
"c1b950ba12598be8495190f93fd341dd646d98c837bf09c72ccc199c2ebb941e",
"sourceTechnicalHash":
"2c3ee90a60bae82e00653c8a088e174bf2ecd6eb8ce1bdda96d08cc50a160318",
- "targetHash":
"565ac481aaafe738b0fd193d6f4617cff6698351cfc71c8d0b575de97634e4c4",
+ "targetHash":
"b22981d476dae36312c2d5631a1d6d10f6e7b28df8b66c96ecd48d8b27a7631f",
"reviewStatus": "machine",
"ownership": "machine",
"translatedAt": "2026-10-11T19:57:10.000Z",
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
