codeant-ai-for-open-source[bot] commented on code in PR #41137: URL: https://github.com/apache/superset/pull/41137#discussion_r3426138115
########## scripts/translations/latin_to_cyrillic_sr.py: ########## @@ -0,0 +1,214 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Normalize Latin-script entries in the Serbian (sr) catalog to Cyrillic. + +The AI backfill produced a mix of scripts: most entries are Serbian Cyrillic, +but a sizeable fraction came back in Serbian Latin (Gajica). The ``sr`` catalog +is meant to be Cyrillic (``sr_Latn`` is generated separately by transliterating +it), so this script rewrites any Latin-script translation into Cyrillic. + +Serbian Latin -> Cyrillic is a deterministic 1:1 mapping *except* that Latin +tokens which are not Serbian words — format placeholders (%(name)s, {x}, %s), +HTML tags, URLs, and brand/technical terms borrowed verbatim from the English +source (SQL, Superset, CSV, GeoJSON, epoch …) — must stay in Latin. The reliable +signal for "leave this alone" is: the token appears verbatim in the English +msgid. That mirrors the convention the AI already followed in the entries it +translated directly into Cyrillic (e.g. "да отворите SQL Lab", "Superset-овог"). + +Only entries whose translation contains no Cyrillic at all are converted, so the +already-correct Cyrillic majority is left untouched. + +Usage: + python scripts/translations/latin_to_cyrillic_sr.py # in place + python scripts/translations/latin_to_cyrillic_sr.py --dry-run # report only + python scripts/translations/latin_to_cyrillic_sr.py \ + --src superset/translations/sr/LC_MESSAGES/messages.po --dest /tmp/out.po +""" + +from __future__ import annotations + +import argparse +import re +import sys +from pathlib import Path + +try: + import polib # type: ignore[import-untyped] +except ImportError: + print("polib is required. Run: pip install polib", file=sys.stderr) + sys.exit(1) + +TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations" +DEFAULT_PO = TRANSLATIONS_DIR / "sr" / "LC_MESSAGES" / "messages.po" + +_CYRILLIC = re.compile(r"[Ѐ-ӿ]") +_LATIN = re.compile(r"[A-Za-zČĆŽŠĐčćžšđ]") + +# Spans that must never be transliterated: %-placeholders, brace placeholders, +# HTML tags, HTML entities, and URLs. Matched in priority order. +_PROTECT = re.compile( + r""" + %\([^)]*\)[#0\- +]?\d*(?:\.\d+)?[a-zA-Z] # %(name)s, %(line)d, %(v).2f + | %[#0\- +]?\d*(?:\.\d+)?[a-zA-Z%] # %s, %d, %.2f, %% + | \{[^{}]*\} # {name}, {0} + | </?[A-Za-z][^>]*> # HTML tags + | &[A-Za-z#0-9]+; # HTML entities + | https?://\S+ | www\.\S+ # URLs + """, + re.VERBOSE, +) + +# Serbian Latin -> Cyrillic. Digraphs first so dž/lj/nj are not split. +_DIGRAPHS: list[tuple[str, str]] = [ + ("dž", "џ"), ("Dž", "Џ"), ("DŽ", "Џ"), + ("lj", "љ"), ("Lj", "Љ"), ("LJ", "Љ"), + ("nj", "њ"), ("Nj", "Њ"), ("NJ", "Њ"), +] +_SINGLE: dict[str, str] = { + "a": "а", "b": "б", "c": "ц", "č": "ч", "ć": "ћ", "d": "д", "đ": "ђ", + "e": "е", "f": "ф", "g": "г", "h": "х", "i": "и", "j": "ј", "k": "к", + "l": "л", "m": "м", "n": "н", "o": "о", "p": "п", "r": "р", "s": "с", + "š": "ш", "t": "т", "u": "у", "v": "в", "z": "з", "ž": "ж", + "A": "А", "B": "Б", "C": "Ц", "Č": "Ч", "Ć": "Ћ", "D": "Д", "Đ": "Ђ", + "E": "Е", "F": "Ф", "G": "Г", "H": "Х", "I": "И", "J": "Ј", "K": "К", + "L": "Л", "M": "М", "N": "Н", "O": "О", "P": "П", "R": "Р", "S": "С", + "Š": "Ш", "T": "Т", "U": "У", "V": "В", "Z": "З", "Ž": "Ж", +} + + +def _transliterate_word(word: str) -> str: + """Transliterate a run of Latin letters into Cyrillic, honoring digraphs.""" + out: list[str] = [] + i = 0 + n = len(word) + while i < n: + two = word[i : i + 2] + # Match a digraph only when the second letter's case is consistent with + # a single letter (so "Ljiljana" -> "Љиљана", "LJUBA" -> "ЉУБА"). + matched = False + for src, dst in _DIGRAPHS: + if two == src: + out.append(dst) + i += 2 + matched = True + break + if matched: + continue + out.append(_SINGLE.get(word[i], word[i])) Review Comment: **Suggestion:** Unknown Latin letters are left unchanged character-by-character while surrounding letters are transliterated, which produces mixed-script corrupted tokens (for words containing letters like `q`, `w`, `x`, or `y`). Treat such tokens as non-Serbian terms and keep them verbatim (or handle them with an explicit whole-token rule) instead of partial transliteration. [logic error] <details> <summary><b>Severity Level:</b> Major ⚠️</summary> ```mdx - ❌ Serbian UI strings can show corrupted mixed scripts. - ⚠️ Technical terms like "query" may appear mangled. ``` </details> <details> <summary><b>Steps of Reproduction ✅ </b></summary> ```mdx 1. Run `python scripts/translations/latin_to_cyrillic_sr.py` so `main()` (scripts/translations/latin_to_cyrillic_sr.py:191-214) calls `normalize_catalog(DEFAULT_PO, DEFAULT_PO, dry_run=False)` with `DEFAULT_PO` pointing to `superset/translations/sr/LC_MESSAGES/messages.po` (line 56). 2. Ensure the catalog contains a Latin-only `msgstr` like `"SQL query"` where the token `"query"` does not appear verbatim in the English `msgid` (so `_keep_tokens()` at lines 149-152 does not add `"query"` to the `keep` set). 3. During `normalize_catalog()` (lines 162-188), `_convert_value()` (lines 155-159) is called for that `msgstr`, which delegates to `transliterate()` (lines 116-130) and then `_transliterate_outside()` (lines 133-146); for the Latin token `"query"` the `_LATIN` regex (line 59) matches, so `_transliterate_word()` (lines 93-113) is invoked. 4. In `_transliterate_word()`, characters `u`, `e`, and `r` are transliterated via `_SINGLE`, while unmapped letters `q` and `y` fall through to `out.append(_SINGLE.get(word[i], word[i]))` (line 111), producing a mixed-script token like `"qуерy"` in the saved catalog, which then surfaces as corrupted UI text when Superset renders that translation at runtime. ``` </details> [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=1f54d3d9d89b4cd2a2545dc76bf3c337&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=1f54d3d9d89b4cd2a2545dc76bf3c337&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) *(Use Cmd/Ctrl + Click for best experience)* <details> <summary><b>Prompt for AI Agent 🤖 </b></summary> ```mdx This is a comment left during a code review. **Path:** scripts/translations/latin_to_cyrillic_sr.py **Line:** 111:111 **Comment:** *Logic Error: Unknown Latin letters are left unchanged character-by-character while surrounding letters are transliterated, which produces mixed-script corrupted tokens (for words containing letters like `q`, `w`, `x`, or `y`). Treat such tokens as non-Serbian terms and keep them verbatim (or handle them with an explicit whole-token rule) instead of partial transliteration. Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise. Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix ``` </details> <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41137&comment_hash=5849669e420ef346e58e5b2e1dca7dee01b1727d201cc4b69d28f72e80a7de78&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41137&comment_hash=5849669e420ef346e58e5b2e1dca7dee01b1727d201cc4b69d28f72e80a7de78&reaction=dislike'>👎</a> ########## scripts/translations/triage_fuzzy.py: ########## @@ -0,0 +1,206 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Triage AI-backfilled ``#, fuzzy`` entries by confidence. + +``backfill_po.py`` marks every machine translation ``#, fuzzy`` so a human can +review it before it is compiled into a ``.mo``. Reviewing thousands of entries +by hand is impractical, so this script keeps only the *low-confidence* entries +fuzzy (the ones genuinely worth a human's attention) and clears the fuzzy flag ++ machine-translation attribution comment on the *high-confidence* ones. + +Confidence is derived from signals the backfill already recorded: + +* **Reference count** — the attribution comment records ``[refs: …]``, the other + languages that already had this string. More references = stronger + cross-language consensus on meaning. ``backfill_po.py``'s own ``--min-context`` + flag uses the same signal. Entries with very few references are ambiguous. +* **Short fragments** — single words / two-word labels ("Scale", "Order", "Key") + are the classic machine-translation failure: homonyms whose sense depends on + UI context. Kept fuzzy unless many languages agree. +* **Plurals** — Serbian (and other Slavic langs) need three plural forms with + correct case; these are error-prone, so weakly-supported plurals stay fuzzy. +* **Placeholder mismatch** — if the ``%(x)s`` / ``{x}`` / ``%s`` set differs + between source and translation the entry is suspect and stays fuzzy. + +IMPORTANT: clearing ``fuzzy`` here marks an entry as *accepted*, not +*human-reviewed*. It is a pragmatic triage for a single contributor's catalog, +not a substitute for native-speaker review. The entries left fuzzy are the ones +the maintainer still needs to read. + +Usage: + python scripts/translations/triage_fuzzy.py --lang sr + python scripts/translations/triage_fuzzy.py --lang sr --dry-run + python scripts/translations/triage_fuzzy.py --lang sr \ + --short-max-refs 9 --plural-max-refs 9 --min-refs 4 +""" + +from __future__ import annotations + +import argparse +import re +import sys +from pathlib import Path + +try: + import polib # type: ignore[import-untyped] +except ImportError: + print("polib is required. Run: pip install polib", file=sys.stderr) + sys.exit(1) + +TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations" + +_ATTRIBUTION = "Machine-translated via backfill_po.py" +_REFS_RE = re.compile(r"\[refs: ([^\]]+)\]") +_PLACEHOLDER_RE = re.compile( + r"%\([^)]*\)[#0\- +]?\d*(?:\.\d+)?[a-zA-Z]" # %(name)s, %(v).2f + r"|%[#0\- +]?\d*(?:\.\d+)?[a-zA-Z%]" # %s, %d, %% + r"|\{[^{}]*\}" # {name}, {0} +) + + +def _ref_count(entry: polib.POEntry) -> int: + """Number of reference languages recorded in the attribution comment.""" + m = _REFS_RE.search(entry.tcomment or "") + if m: + return len([x for x in m.group(1).split(",") if x.strip()]) + return 0 # "[no refs]" or no attribution + + +def _placeholder_mismatch(entry: polib.POEntry) -> bool: + """True if the singular translation's placeholder set differs from source.""" + if entry.msgid_plural: + return False # plural forms handled via the plural-confidence rule Review Comment: **Suggestion:** Placeholder validation is skipped for all pluralized entries, so a plural translation that drops or changes placeholders can still be auto-accepted when reference counts are high. This can later break string formatting at runtime. Run placeholder checks for each plural form instead of returning early for plural entries. [logic error] <details> <summary><b>Severity Level:</b> Critical 🚨</summary> ```mdx - ❌ Plural messages may format with missing placeholders and crash. - ❌ Localized API responses can fail in Serbian locales. ``` </details> <details> <summary><b>Steps of Reproduction ✅ </b></summary> ```mdx 1. Use `backfill_po.py` to machine-translate a catalog so `_apply_translation()` (scripts/translations/backfill_po.py:4-27, esp. lines 20-27) writes translations and sets the `fuzzy` flag on plural entries, adding attribution comments with `[refs: …]`. 2. Introduce a plural entry whose translation drops or alters a placeholder from the source, e.g. `msgid_plural` contains `"Deleted %(num)d datasets"` while one `msgstr_plural[...]` omits `%(num)d`, so that when used with `ngettext("Deleted %(num)d dataset", "Deleted %(num)d datasets", num=len(item_ids))` in `superset/datasets/api.py:898-23`, the translation would break if accepted. 3. Run `python scripts/translations/triage_fuzzy.py --lang sr` so `main()` (scripts/translations/triage_fuzzy.py:159-205) calls `triage()` (lines 127-156) on `superset/translations/sr/LC_MESSAGES/messages.po`. 4. In `triage()`, the plural entry is processed: `is_low_confidence()` (lines 97-114) calls `_placeholder_mismatch()` (lines 83-89), which returns `False` early for any plural (`if entry.msgid_plural: return False` at lines 85-86), so placeholder differences are ignored; if `_ref_count()` (lines 75-80) reports `refs > plural_max_refs` (line 110), the entry is treated as high-confidence, its `fuzzy` flag is cleared, and the broken plural translation is accepted, leading to a runtime formatting error when `ngettext` is evaluated in `superset/datasets/api.py:898-23`. ``` </details> [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=9226b7abf8ff47e0a2932cd7863329a1&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=9226b7abf8ff47e0a2932cd7863329a1&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) *(Use Cmd/Ctrl + Click for best experience)* <details> <summary><b>Prompt for AI Agent 🤖 </b></summary> ```mdx This is a comment left during a code review. **Path:** scripts/translations/triage_fuzzy.py **Line:** 85:86 **Comment:** *Logic Error: Placeholder validation is skipped for all pluralized entries, so a plural translation that drops or changes placeholders can still be auto-accepted when reference counts are high. This can later break string formatting at runtime. Run placeholder checks for each plural form instead of returning early for plural entries. Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise. Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix ``` </details> <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41137&comment_hash=744fbf28921cd599a6e86051e87398d7f24de7cfc3cd2cf6d4f9b47c08c30271&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41137&comment_hash=744fbf28921cd599a6e86051e87398d7f24de7cfc3cd2cf6d4f9b47c08c30271&reaction=dislike'>👎</a> ########## scripts/translations/triage_fuzzy.py: ########## @@ -0,0 +1,206 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Triage AI-backfilled ``#, fuzzy`` entries by confidence. + +``backfill_po.py`` marks every machine translation ``#, fuzzy`` so a human can +review it before it is compiled into a ``.mo``. Reviewing thousands of entries +by hand is impractical, so this script keeps only the *low-confidence* entries +fuzzy (the ones genuinely worth a human's attention) and clears the fuzzy flag ++ machine-translation attribution comment on the *high-confidence* ones. + +Confidence is derived from signals the backfill already recorded: + +* **Reference count** — the attribution comment records ``[refs: …]``, the other + languages that already had this string. More references = stronger + cross-language consensus on meaning. ``backfill_po.py``'s own ``--min-context`` + flag uses the same signal. Entries with very few references are ambiguous. +* **Short fragments** — single words / two-word labels ("Scale", "Order", "Key") + are the classic machine-translation failure: homonyms whose sense depends on + UI context. Kept fuzzy unless many languages agree. +* **Plurals** — Serbian (and other Slavic langs) need three plural forms with + correct case; these are error-prone, so weakly-supported plurals stay fuzzy. +* **Placeholder mismatch** — if the ``%(x)s`` / ``{x}`` / ``%s`` set differs + between source and translation the entry is suspect and stays fuzzy. + +IMPORTANT: clearing ``fuzzy`` here marks an entry as *accepted*, not +*human-reviewed*. It is a pragmatic triage for a single contributor's catalog, +not a substitute for native-speaker review. The entries left fuzzy are the ones +the maintainer still needs to read. + +Usage: + python scripts/translations/triage_fuzzy.py --lang sr + python scripts/translations/triage_fuzzy.py --lang sr --dry-run + python scripts/translations/triage_fuzzy.py --lang sr \ + --short-max-refs 9 --plural-max-refs 9 --min-refs 4 +""" + +from __future__ import annotations + +import argparse +import re +import sys +from pathlib import Path + +try: + import polib # type: ignore[import-untyped] +except ImportError: + print("polib is required. Run: pip install polib", file=sys.stderr) + sys.exit(1) + +TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations" + +_ATTRIBUTION = "Machine-translated via backfill_po.py" +_REFS_RE = re.compile(r"\[refs: ([^\]]+)\]") +_PLACEHOLDER_RE = re.compile( + r"%\([^)]*\)[#0\- +]?\d*(?:\.\d+)?[a-zA-Z]" # %(name)s, %(v).2f + r"|%[#0\- +]?\d*(?:\.\d+)?[a-zA-Z%]" # %s, %d, %% + r"|\{[^{}]*\}" # {name}, {0} +) + + +def _ref_count(entry: polib.POEntry) -> int: + """Number of reference languages recorded in the attribution comment.""" + m = _REFS_RE.search(entry.tcomment or "") + if m: + return len([x for x in m.group(1).split(",") if x.strip()]) Review Comment: **Suggestion:** Reference extraction uses the first `[refs: ...]` match in the comment, but comments can accumulate multiple backfill attribution lines over repeated runs; this can read stale reference counts and mis-triage entries. Parse the most recent attribution/reference line instead of the first match. [logic error] <details> <summary><b>Severity Level:</b> Major ⚠️</summary> ```mdx - ⚠️ Fuzzy triage may rely on outdated reference counts. - ⚠️ Some risky entries may be auto-accepted or over-reviewed. ``` </details> <details> <summary><b>Steps of Reproduction ✅ </b></summary> ```mdx 1. Run `backfill_po.py` to translate a catalog so `_apply_translation()` (scripts/translations/backfill_po.py:4-27) sets `entry.tcomment` to a line like `"Machine-translated via backfill_po.py (gpt-4) [refs: en, fr]"` for each fuzzy entry. 2. Re-run `backfill_po.py` later with a different model or a different `context_langs` set for the same entries; `_apply_translation()` appends another attribution line if it is not already present (lines 20-27), producing multi-line comments such as: first line with `[refs: en, fr]`, second line with `[refs: en, fr, de, es]`. 3. Run `python scripts/translations/triage_fuzzy.py --lang sr` so `main()` (scripts/translations/triage_fuzzy.py:159-205) invokes `triage()` (lines 127-156) on the updated catalog; `_ref_count()` (lines 75-80) uses `_REFS_RE.search(entry.tcomment or "")` (line 77), which returns the first `[refs: …]` occurrence in the multi-line comment, i.e. the older reference set. 4. Because `_ref_count()` is based on that stale first match, `is_low_confidence()` (lines 97-114) may treat an entry as having fewer or more references than the latest attribution indicates, misclassifying it as low-confidence (kept fuzzy unnecessarily) or high-confidence (unfuzzied when it should remain for review), thereby skewing the triage workflow even though newer `[refs: …]` data is present in the same comment. ``` </details> [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=bce7855014c64744b2455a7a9f838585&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=bce7855014c64744b2455a7a9f838585&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) *(Use Cmd/Ctrl + Click for best experience)* <details> <summary><b>Prompt for AI Agent 🤖 </b></summary> ```mdx This is a comment left during a code review. **Path:** scripts/translations/triage_fuzzy.py **Line:** 77:79 **Comment:** *Logic Error: Reference extraction uses the first `[refs: ...]` match in the comment, but comments can accumulate multiple backfill attribution lines over repeated runs; this can read stale reference counts and mis-triage entries. Parse the most recent attribution/reference line instead of the first match. Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise. Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix ``` </details> <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41137&comment_hash=844eb3552eeb6e06de181ed17e92e991585cfbe1ca1506c5d7713f9e3ed29cb4&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41137&comment_hash=844eb3552eeb6e06de181ed17e92e991585cfbe1ca1506c5d7713f9e3ed29cb4&reaction=dislike'>👎</a> ########## scripts/translations/triage_fuzzy.py: ########## @@ -0,0 +1,206 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Triage AI-backfilled ``#, fuzzy`` entries by confidence. + +``backfill_po.py`` marks every machine translation ``#, fuzzy`` so a human can +review it before it is compiled into a ``.mo``. Reviewing thousands of entries +by hand is impractical, so this script keeps only the *low-confidence* entries +fuzzy (the ones genuinely worth a human's attention) and clears the fuzzy flag ++ machine-translation attribution comment on the *high-confidence* ones. + +Confidence is derived from signals the backfill already recorded: + +* **Reference count** — the attribution comment records ``[refs: …]``, the other + languages that already had this string. More references = stronger + cross-language consensus on meaning. ``backfill_po.py``'s own ``--min-context`` + flag uses the same signal. Entries with very few references are ambiguous. +* **Short fragments** — single words / two-word labels ("Scale", "Order", "Key") + are the classic machine-translation failure: homonyms whose sense depends on + UI context. Kept fuzzy unless many languages agree. +* **Plurals** — Serbian (and other Slavic langs) need three plural forms with + correct case; these are error-prone, so weakly-supported plurals stay fuzzy. +* **Placeholder mismatch** — if the ``%(x)s`` / ``{x}`` / ``%s`` set differs + between source and translation the entry is suspect and stays fuzzy. + +IMPORTANT: clearing ``fuzzy`` here marks an entry as *accepted*, not +*human-reviewed*. It is a pragmatic triage for a single contributor's catalog, +not a substitute for native-speaker review. The entries left fuzzy are the ones +the maintainer still needs to read. + +Usage: + python scripts/translations/triage_fuzzy.py --lang sr + python scripts/translations/triage_fuzzy.py --lang sr --dry-run + python scripts/translations/triage_fuzzy.py --lang sr \ + --short-max-refs 9 --plural-max-refs 9 --min-refs 4 +""" + +from __future__ import annotations + +import argparse +import re +import sys +from pathlib import Path + +try: + import polib # type: ignore[import-untyped] +except ImportError: + print("polib is required. Run: pip install polib", file=sys.stderr) + sys.exit(1) + +TRANSLATIONS_DIR = Path(__file__).parent.parent.parent / "superset" / "translations" + +_ATTRIBUTION = "Machine-translated via backfill_po.py" +_REFS_RE = re.compile(r"\[refs: ([^\]]+)\]") +_PLACEHOLDER_RE = re.compile( + r"%\([^)]*\)[#0\- +]?\d*(?:\.\d+)?[a-zA-Z]" # %(name)s, %(v).2f + r"|%[#0\- +]?\d*(?:\.\d+)?[a-zA-Z%]" # %s, %d, %% + r"|\{[^{}]*\}" # {name}, {0} +) + + +def _ref_count(entry: polib.POEntry) -> int: + """Number of reference languages recorded in the attribution comment.""" + m = _REFS_RE.search(entry.tcomment or "") + if m: + return len([x for x in m.group(1).split(",") if x.strip()]) + return 0 # "[no refs]" or no attribution + + +def _placeholder_mismatch(entry: polib.POEntry) -> bool: + """True if the singular translation's placeholder set differs from source.""" + if entry.msgid_plural: + return False # plural forms handled via the plural-confidence rule + return set(_PLACEHOLDER_RE.findall(entry.msgid)) != set( + _PLACEHOLDER_RE.findall(entry.msgstr) + ) Review Comment: **Suggestion:** Comparing placeholder sets hides duplicate-count mismatches (for example, two `%s` in source versus one `%s` in translation), so broken translations may be treated as valid. Compare placeholder multisets/counts rather than sets to detect dropped repeated placeholders. [logic error] <details> <summary><b>Severity Level:</b> Critical 🚨</summary> ```mdx - ❌ Dropped duplicate placeholders can pass triage undetected. - ❌ Accepted strings may crash when formatted with arguments. ``` </details> <details> <summary><b>Steps of Reproduction ✅ </b></summary> ```mdx 1. Start with a machine-translated catalog where `backfill_po.py` has added fuzzy, attributed entries as in `_apply_translation()` (scripts/translations/backfill_po.py:4-27), including a singular message whose `msgid` contains the same placeholder multiple times, e.g. `"Copied %s rows and %s columns"` and whose `msgstr` mistakenly uses `"Copied %s rows and columns"` (only one `%s`). 2. Run `python scripts/translations/triage_fuzzy.py --lang sr` so `main()` (scripts/translations/triage_fuzzy.py:159-205) calls `triage()` (lines 127-156) over `superset/translations/sr/LC_MESSAGES/messages.po`. 3. For the affected entry, `is_low_confidence()` (lines 97-114) calls `_placeholder_mismatch()` (lines 83-89); `_PLACEHOLDER_RE.findall(entry.msgid)` returns `["%s", "%s"]` while `_PLACEHOLDER_RE.findall(entry.msgstr)` returns `["%s"]`, but converting both to sets at line 87 (`set([...])`) yields `{"%s"}` in both cases, so `_placeholder_mismatch()` returns `False` even though one `%s` was dropped. 4. If `_ref_count()` (lines 75-80) reports `refs >= min_refs` (line 106) and other low-confidence conditions do not apply, `is_low_confidence()` returns `False`, `triage()` (lines 139-153) clears the `fuzzy` flag and strips attribution, and the broken translation is accepted; when the application formats that string with two arguments in any view that uses it (similar to how `ngettext` is used in `superset/datasets/api.py:898-23`), Python string formatting raises due to the missing placeholder. ``` </details> [](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=5ce5322eca4f4b49ba976510374ec5c8&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=5ce5322eca4f4b49ba976510374ec5c8&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) *(Use Cmd/Ctrl + Click for best experience)* <details> <summary><b>Prompt for AI Agent 🤖 </b></summary> ```mdx This is a comment left during a code review. **Path:** scripts/translations/triage_fuzzy.py **Line:** 87:89 **Comment:** *Logic Error: Comparing placeholder sets hides duplicate-count mismatches (for example, two `%s` in source versus one `%s` in translation), so broken translations may be treated as valid. Compare placeholder multisets/counts rather than sets to detect dropped repeated placeholders. Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise. Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix ``` </details> <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41137&comment_hash=115b2d9d9bfe77e2099bd73b541f27236781715e66e7c78c99a759bea7501528&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F41137&comment_hash=115b2d9d9bfe77e2099bd73b541f27236781715e66e7c78c99a759bea7501528&reaction=dislike'>👎</a> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
