pomegranited opened a new issue, #32854: URL: https://github.com/apache/superset/issues/32854
## [SIP] Proposal for Translating Superset asset metadata ### Motivation Superset provides translation support for built-in components in the UI. However, the o[Open edX](https://openedx.org) project also needs the user-provided terms used in the assets themselves to be translatable, e.g. dashboard and chart title, axes labels, and metric labels. Open edX a Learning Management System that is run worldwide and often for multi-language deployments, e.g Arabic + Hebrew, French + English. We also need these asset translations to be easily maintained between upgrades of Superset, and to be re-deployable when translations are updated. An automatic way to export translated asset fields and import translations would also help integrate Superset into Open edX's ecosystem. Superset's requirements for this feature include (refs [1](https://github.com/apache/superset/issues/32139#issuecomment-2675387707), [2](https://github.com/apache/superset/issues/32139#issuecomment-2670053442)): * Zero performance hit for English `en` (or when running with a single system default language) The system default language strings should live in the field itself, so no extra lookups are required. * No data translations will be added to Superset's officially supported translations -- just like asset data, these translations must be managed by Superset users. * Translators can update asset translations in the Superset UI and see their updates in real time. Desirable features include: * Searchability of the appropriate translated text when user has selected a non-`en` language ([ref](https://github.com/apache/superset/issues/32139#issuecomment-2675846271)). Superset, by default, is configured to use English (`en`) as the single default language via the [`BABEL_DEFAULT_LOCALE`](https://github.com/apache/superset/blob/45ea11c1b65887755f30e5945ea280abc0847929/superset/config.py#L374) setting. Some translations for built-in UI components are provided for 16 additional languages which can be enabled via the [`LANGUAGES`](https://github.com/apache/superset/blob/45ea11c1b65887755f30e5945ea280abc0847929/superset/config.py#L378-L396) setting, but [these languages are disabled by default](https://github.com/apache/superset/blob/45ea11c1b65887755f30e5945ea280abc0847929/superset/config.py#L397-L399) for performance reasons, and due to incomplete available translations. Open edX aims to implement this feature to increase uptake of the use of Superset in our ecosystem. The more internationalised sites that run Superset, the more people will be motivated to fill in Superset's missing UI translations for more supported languages. ### Proposed Change We propose implementing this feature in two phases. This feature will be disabled by default, and will only be enabled if **both** are true: 1. The app-wide feature flag is enabled (e.g `ENABLE_I18N_ASSET_TRANSLATIONS`) 2. More than one non-default language is present in the [`LANGUAGES`](https://github.com/apache/superset/blob/45ea11c1b65887755f30e5945ea280abc0847929/superset/config.py#L378-L396) setting. #### Phase 1: Functionality 1. Add the base functionality for rendering configured translatable model fields using a new jinja filter called `i18n`. No changes to the UI or data will be visible to a user who edits a translatable field -- the user continues to provide the full `BABEL_DEFAULT_LOCALE` default text for the field. When this feature is enabled and the user views an asset, they will see their selected language's translation, if found, or the default text, if not. When rendering translatable fields: * If feature is disabled, or there's only one language configured, or the user is using the app's default language, simply return the field's default text. * Else, show `translated_text` found in the `i18n_translations` table for the given asset UUID + default text + user's current language. Fallback to the default text. 1. Export/import asset translations as part of the standard UX Export/Import asset bundle process. If this feature is disabled, translations will not be exported, and will be ignored on import. Else, translations will be exported to separate file(s) in the export zip file, and updated on import. #### Phase 2: UX Add buttons to the Superset UI to enable editors to provide translations of asset metadata in the Superset UI. Exact UX TBD, but see https://github.com/apache/superset/issues/13442 for one nice way to do this. When editing a templated field, the user should be able to see: * The full `default_text` value to be translated * All configured [`LANGUAGES`](https://github.com/apache/superset/blob/45ea11c1b65887755f30e5945ea280abc0847929/superset/config.py#L378-L396) to be translated This phase requires making a lot of UI changes, so will likely be broken into smaller contributions. It also would benefit from the input of a UI/UX designer, which we can provide. ### New or Changed Public Interfaces * Visualization types -- unchanged * Form data for saved dashboards and charts -- likely unchanged, but see "Open Question 4" below. * Export bundles will include new files for translations -- YML, format TBD. #### Phase 1: Functionality Configuration: 1. A new configuration feature flag would be added to enable/disable this feature, called `ENABLE_I18N_ASSET_TRANSLATIONS`. 2. A new configuration setting would be added to enable/configure caching of these translations, called `I18N_ASSET_TRANSLATIONS_CACHE_CONFIG`. Data passed between backend and frontend: 1. Add new jinja filter called `i18n` which uses the `i18n_translations` table described below to render translated text in a request. 1. Add caching support for these translations (per language + asset UUID) using the existing [Superset Caching](https://superset.apache.org/docs/configuration/cache/) conventions. Cache will be populated for a given language + asset UUID when the asset is viewed with a selected language. 2. Add a model mixin to add the concept of "templated fields" E.g the Dashboard model would have the following class variable: ```python templated_fields = [ 'dashboard_title', 'description', 'json_metadata', 'position_json', ] ``` 3. When rendering model fields to the "view" API: * If the app-wide setting is enabled: wrap templated field values in the `i18n` jinja filter, and render values. * Else: pass field values straight through. 1. "Export asset" backend APIs will be updated to export any stored translations for the exported assets. 1. "Import asset" backend APIs will be updated to update any translations provided in the imported bundle. Command line tools and arguments: 1. New command line tool will be added to cull any translations which are no longer referenced by their linked assets. #### Phase 2: UX 1. A new "translation UI component" would take these properties as input: * `asset_uuid` * `model_name` * `field_name` * `default_text` * `id` (optional, may be provided by user in jinja filter to disambiguate multiple translations in a single field.) 1. A "translation REST API" would be added to the backend to support these components for viewing and editing translated text for a given asset model field with the configured `BABEL_DEFAULT_LOCALE` and `LANGUAGES` settings. ### New dependencies A new Flask/FAB extension will hold the functionality for this feature, to keep the Superset code changes to a minimum. This extension will include: * a pypi package added as a Superset python requirement, with a mix-in class for the relevant asset Models. * a npm package to support UI changes, and may depend on other npm packages, e.g [AntD's Popover](https://ant.design/components/popover) ### Migration Plan and Compatibility A migration will be added to create a new table called `i18n_translations`. Fields will be: * `asset_uuid`: unique identifier for the asset being translated. * `default_text`: user-provided text for the configured `BABEL_DEFAULT_LOCALE` * `language_code`: language/locale code for the translated text * `translated_text`: `default_text` translation for the `language_code` * `model_name`: name of the table containing the translatable field. * `field_name`: name of the field being translated. * `id`: unique string identifier for the term being translated. Can be provided in the jinja template, or will be auto-generated if omitted. Indexes: * `asset_uuid` + `default_text` + `language_code` (unique): used to locate any `translated_text` fields when rendering a given asset. May also add `model_name` and/or `field_name` to this index, if needed. Notes: * Dashboards and charts that are saved or bookmarked will still work after the change, or when this change is enabled/disabled. ### Open Questions: #### Translation providers How best to support both translations provided by humans and translations provided by external automated services? Translation services would require extra context (e.g. service URLs, authentication credentials, extra parameters) that could be provided in configuration ([ref, 1st ¶](https://github.com/apache/superset/issues/32854#issuecomment-2790649473)). Humans benefit from performing translations close to the context where they're used - so we need a UI? #### Markup in fields Should translated fields require Authors to add jinja markup to their field values? **Pros:** 1. Generic `templated_fields` support a variety of use cases across the platform [ref](https://github.com/apache/superset/issues/32854#issuecomment-2791220088). 2. Authors can decide which assets and fields require translations (see Authority below). 3. Authors can provide extra context and optionally use a more complex template format. 4. Allows us to translate parts of complex fields like dashboard [`json_metadata`](https://github.com/apache/superset/blob/45ea11c1b65887755f30e5945ea280abc0847929/superset/models/dashboard.py#L141), [`position_json`](https://github.com/apache/superset/blob/45ea11c1b65887755f30e5945ea280abc0847929/superset/models/dashboard.py#L136) and slice [`params`](https://github.com/apache/superset/blob/45ea11c1b65887755f30e5945ea280abc0847929/superset/models/slice.py#L80), e.g. `metadata.native_filter_configuration.name`, `position.*.meta.text`, `params.x_axis_label`. **Cons:** Currently, we're proposing that Authors _not_ mark up translated fields, and instead, they will be automatically wrapped in the `i18n` filter, because: 1. Many languages would need to change the order of the translatable phrases in order to be sensible, e.g. "Panel de ingresos de la CompanyX", and so translating the full phrase is better. 2. Simplifies the editing user's experience by not requiring them to use jinja template syntax or guess which terms need translation and which don't. 3. Allows this feature to be enabled or disabled without having to change data. E.g. if the actual stored `dashboard_title = "CompanyX {{ "Revenue Dashboard" | i18n("rev_dash") }}`, this field becomes unintelligible if the feature is subsequently disabled. 4. We have enough information (asset UUID, model name, field name) to uniquely identify the translated field text if we treat it as an atomic unit, rather than allowing its value to be broken into some unknown number of parts. **Can we have it both ways?** #### Authority Who decides what assets and fields need translations? ([ref, 2nd ¶](https://github.com/apache/superset/issues/32854#issuecomment-2790649473)) Should it be the author, administrators, an external service, ...? #### Data Should the data itself be translatable ([ref, final •](https://github.com/apache/superset/issues/32854#issuecomment-2790649473))? This SIP likely won't decide/implement this point, but it needs to at least address the use case. #### Search/filtering Unsure how to provide asset search/filtering for translated text. Can we use [base_filters](https://flask-appbuilder.readthedocs.io/en/latest/advanced.html#base-filtering) somehow? ### Out of Scope 1. RTL support in the UI templates: https://github.com/apache/superset/issues/25258 2. Localized numbers and dates based on the user's selected language/locale. 3. Translating the actual chart data. ### Rejected Alternatives * [SIP-60](https://github.com/apache/superset/issues/13442) -- The frontend UI proposed in SIP-60 could be used here. But we propose an alternative backend implementation which uses a single `i18n_translation` table to store all translations instead of introducing individual `<field_name>_i18n` field for each translatable field. * [SIP-153](https://github.com/apache/superset/issues/32139) -- rejected because its approach was ["build time" focused and not as dynamic as desired](https://github.com/apache/superset/issues/32139#issuecomment-2670053442) and because it abuses the [frontend language packs shipped from the backend](https://github.com/apache/superset/issues/32139#issuecomment-2670069005), which is not ideal. SIP-153 also didn't support returning translated fields in the data APIs, while this solution does. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
