michael-s-molina opened a new issue, #41854: URL: https://github.com/apache/superset/issues/41854
# [SIP-220] Proposal for Extensions Storage ## Motivation This proposal gives extensions a [host-managed storage API](https://github.com/apache/superset/pull/39171) so persistence becomes a one-line call instead of a bespoke integration — isolation, TTL, and encryption handled once by the host, in one place, rather than reinvented (and re-debugged) per extension. Concretely, this makes it much easier to build things like: - **Third-party integrations that hold credentials** — e.g. a Slack or PagerDuty notification extension that needs to store a per-user API token, encrypted at rest, without asking the user to re-enter it every session. - **Background/long-running work with progress tracking** — a data-quality check or batch export extension that kicks off backend work and needs the frontend to poll for progress across page reloads, backed by a shared server-side cache slot. - **Cross-user shared configuration** — an extension-wide feature flag, announcement banner, or shared lookup table that an admin sets once and every user of the extension sees. - **User-specific saved artifacts that follow the user across devices** — e.g. a snippet-manager extension letting a user save and reuse their own SQL fragments, or an annotation extension letting a user attach personal notes to a chart, persisted server-side instead of stuck in one browser. Extensions can already persist data today — that's not the gap. An extension can write to `localStorage`, use Superset's own `key_value` table, or stand up and call out to a persistence layer entirely of its own outside Superset. All three are live options, not workarounds in the sense of being blocked; they're workarounds in the sense of being ad hoc, each extension author solving the same problem from scratch: - **No isolation guarantee.** Nothing stops two extensions from colliding on the same key, or one extension reading another's data — whatever isolation exists is a convention each extension author has to get right on their own, not something enforced. - **No consistent semantics.** TTL, encryption, and cross-user sharing get reinvented — and re-debugged — independently by every extension that needs them, because there is no shared primitive to build on. - **No shared, sanctioned default.** `localStorage` doesn't survive a browser clear and isn't reachable from backend code at all. `key_value` works but has no per-extension columns to scope, quota, or bulk-delete by. Standing up an entirely separate system outside Superset works too, but means every extension author that needs durable storage pays the cost of building and operating one, instead of a single primitive doing it for all of them. That's the gap this proposal closes: not "nothing durable exists today," but "nothing durable exists that's easy, consistent, and safe by default" — which is why the four scenarios above are harder to build well than they should be. ## Proposed Change This proposal introduces that shared primitive as a host-managed storage API, structured as three storage tiers, each mapping to a different backend and lifetime. The extension author picks the tier; the host handles namespacing, isolation, and (for Tier 3) encryption — once, instead of once per extension. | | Tier 1 — Local State | Tier 2 — `ephemeral` | Tier 3 — `persistent` | |---|---|---|---| | Backend | Browser storage | Server cache (Redis/Memcached/filesystem, via Flask-Caching) | `extension_storage` DB table | | Survives page reload | Yes | Yes | Yes | | Survives browser clear | No | Yes | Yes | | Survives server restart | Yes (client-side) | No | Yes | | TTL | N/A | Required on every write | N/A | | Encryption | No | No | Optional, opt-in per write | | Frontend-only | Yes | No | No | | API shape | `ctx.storage.local`, `ctx.storage.session` | `ctx.storage.ephemeral` | `ctx.storage.persistent` | Tier 1 has two variants, both browser-only and both sharing the same accessor shape: `local` (backed by `localStorage`, survives across browser sessions) and `session` (backed by `sessionStorage`, cleared when the tab closes). They are grouped under one tier rather than split into separate tiers because they differ only in *how long* browser-side data survives — not in backend, isolation model, or API shape — the same distinction Chrome Extensions draws between `storage.local` and `storage.session` under its single `storage` API, rather than treating them as separate extension platform tiers. All three tiers share the same accessor shape — `get(key)`, `set(key, value, options?)`, `remove(key)` — plus a `.shared` sibling accessor with identical methods that reads/writes data visible to every user of the extension instead of just the calling user. This mirrors the pattern used by Chrome Extensions (`storage.local` vs `storage.sync`) and Atlassian Forge (app-scoped KV store), adapted to Superset's per-user authentication model. **Isolation is structural, not conventional.** Every tier is automatically namespaced to the calling extension — an extension never constructs its own storage key or REST URL. This is enforced by `ctx.storage`, obtained via `extensions.getContext()` (frontend) or `get_context()` (backend), which returns an object pre-bound to the calling extension's identity. An extension cannot address another extension's storage through this API even if it wanted to, because the extension ID is never a parameter the caller supplies — it is baked into the object the host handed back. This only works because it's the host doing the namespacing: logic living in a library an extension author could skip or get wrong would not give the same guarantee. Centralizing storage in the host also means TTL, encryption, and cross-user sharing behave the same way across all three tiers instead of being reinvented per extension, and gives one place — rather than one per extension — to reason about per-extension quota, uninstall cleanup, and shared data (see *Resource-linked storage and cleanup ownership* below). **Frontend example:** ```ts import { extensions } from "@apache-superset/core"; const ctx = extensions.getContext(); await ctx.storage.local.set("theme", "dark"); await ctx.storage.ephemeral.set("job_progress", { pct: 42 }, { ttl: 3600 }); await ctx.storage.persistent.set("api_token", "sk-...", { encrypt: true }); // Shared (cross-user) data: await ctx.storage.persistent.shared.set("announcement", "New release!"); ``` **Backend example:** ```py from superset_core.extensions.context import get_context ctx = get_context() ctx.storage.ephemeral.set("job_progress", {"pct": 42}, ttl=3600) ctx.storage.persistent.set("config_key", {"version": 2}) ctx.storage.persistent.shared.get("announcement") ``` **How per-extension isolation is actually implemented (frontend):** each extension is loaded as its own Webpack Module Federation remote container. When the host loads a container, it calls the container's `init()` runtime entrypoint with a share-scope object — normally the literal, shared `__webpack_share_scopes__.default`, so that a `singleton: true` shared dependency resolves to the one host-provided instance every remote sees. The host instead builds a **per-container copy** of that share scope with only the `@apache-superset/core` entry replaced by a synthetic module whose `extensions.getContext` closure is hard-bound to that one extension's context object, and passes that copy to `container.init()`. Because Module Federation caches the resolved module per container, every import of `@apache-superset/core` inside that extension — however and whenever it is imported — resolves to this pre-bound copy. Because each container gets its own independent copy of the scope objec t, loading multiple extensions in parallel cannot leak one extension's context into another's. This is the same mechanism `ExtensionsLoader` already relies on for per-extension isolation generally (see `ExtensionContext`); this proposal extends what that context carries, from just extension metadata to a full `storage` namespace. Other approaches to ambient context were considered (see *Rejected Alternatives*), but the one proposed here is the better developer experience — extension authors call `getContext()` and get the right object back with no ceremony — and it comes for free from Module Federation's existing share-scope mechanism rather than requiring new infrastructure. **Backend isolation** uses the same shape without needing a Module Federation analog: a `contextvars.ContextVar` is set around each extension's execution and cleared afterward, giving `get_context()` correct behavior across async boundaries within a single request without cross-request leakage. **Tier 3 encryption:** opt-in per write (`encrypt: true` / `is_encrypted`). Reuses Superset's existing `EncryptedType` (`sqlalchemy-utils`) and engine resolution infrastructure — the same mechanism used for database connection credentials — rather than introducing a new crypto dependency. User-scoped encrypted values are keyed by an HMAC-SHA256 derivation of `SECRET_KEY` and the user's id, so one user's ciphertext cannot be decrypted as another's; shared/global encrypted values use `SECRET_KEY` directly. `extension_storage` rows are registered with the existing `SecretsMigrator`, so `SECRET_KEY` rotation re-encrypts extension secrets alongside database credentials with no extension-specific tooling. **Resource-linked storage and cleanup ownership.** The `extension_storage` schema and DAO include `resource_type`/`resource_uuid` columns for linking a stored row to a specific resource. That resource can be a built-in Superset asset (a dashboard, chart, or database) or a custom asset defined by another extension entirely — `resource_type` is a free-form string, not a fixed enum, since Superset has no way to know about every resource type an extension might invent. This proposal deliberately does **not** wire up automatic cleanup (e.g. an ORM `after_delete` hook cascading from `dashboards`/`slices`/`dbs` into `extension_storage`) when a linked resource is deleted. Two reasons: - **Superset can't know what "deleted" means for every `resource_type`.** A cascade hook can only exist for the finite set of resource types Superset's own models represent. For any extension-defined resource type, such a hook would silently do nothing — which is worse than having no cleanup at all, since it looks like cleanup exists but only covers a subset of cases, with no signal to the extension author that their resource type isn't covered. - **Only the extension that created the link knows when it's actually safe to delete.** Even for built-in resource types, an extension may want to retain resource-linked data past the resource's own deletion (e.g. for an audit trail), or clean it up on a different trigger than "the resource row disappeared." Ownership of storage cleanup semantics belongs to whichever extension created the data, not to a generic host-side hook guessing at intent. The right place for an extension to implement this cleanup is the Global Task Framework (GTF, `superset_core.tasks`) already available to extensions for async and background work — it already provides `TaskScope.SYSTEM` for admin-scoped background jobs, and cleanup-on-a-schedule is exactly that shape of work. To actually enumerate and remove the rows tied to a resource, an extension needs a backend API to call from within that cleanup task. This is backend-only, since it's bookkeeping the extension's own backend code performs against its own resources, not something a frontend UI needs. This proposal provides that API, giving an extension the standard set of operations — find, filter by resource, delete — scoped automatically to its own rows: ```py from superset_core.extensions.storage.dao import ExtensionStorageDAO # All operations are automatically scoped to the calling extension's rows. ExtensionStorageDAO.filter_by(resource_type="my-resource-type") ExtensionStorageDAO.delete(entries) ``` Once GTF supports recurring/scheduled execution (today's `@task` only supports one-shot dispatch via direct call or `.schedule()`), an extension's own cleanup task could look like: ```py # schedule= is illustrative — GTF only supports one-shot dispatch today. # Everything else here — ExtensionStorageDAO and its filter_by/delete # methods — is real and shipped as part of this proposal. from superset_core.extensions.storage.dao import ExtensionStorageDAO from superset_core.tasks.decorators import task from superset_core.tasks.types import TaskScope @task(scope=TaskScope.SYSTEM, schedule="@daily") def cleanup_orphaned_storage() -> None: live_resource_ids = my_extension_list_live_resource_ids() orphaned = [ entry for entry in ExtensionStorageDAO.filter_by(resource_type="my-resource-type") if entry.resource_uuid not in live_resource_ids ] ExtensionStorageDAO.delete(orphaned) ``` **GTF's recurring-schedule support is explicitly out of scope for this proposal.** Adding a `schedule` parameter to `@task` (and the dispatcher that scans for and fires due tasks) is a change to the task framework itself, used well beyond extension storage, and deserves its own design and review rather than being folded into this proposal — tracked as a follow-up once GTF's scheduling capability lands. ## New or Changed Public Interfaces **New public interface (frontend SDK, `@apache-superset/core`):** - `extensions.getContext()`, returning an `ExtensionContext` with `extension` (metadata) and `storage: ExtensionStorage` — `local`, `session`, `ephemeral`, and `persistent` tiers as described above. **New public interface (backend SDK, `superset_core`):** - `superset_core.extensions.context.get_context()`, the backend counterpart of the above — returning an `ExtensionContext` with `extension` and `storage`, the latter exposing `ephemeral` and `persistent` tiers (no `local`/`session` — those are frontend-only). - `superset_core.extensions.storage.dao.ExtensionStorageDAO`, a `BaseDAO` subclass over `ExtensionStorageEntry` (`superset_core.extensions.storage.models`), following the same dependency-injection pattern as `DashboardDAO`/`ChartDAO` in `superset_core.common.daos` and `TaskDAO` in `superset_core.tasks.daos`. Gives backend extension code `find_all`/`find_one_or_none`/`filter_by`/`query`/`create`/`update`/`delete`, automatically scoped to the calling extension's own rows. Intended for enumeration and bulk management (e.g. resource-linked cleanup jobs), not single-key access — see `persistent` above for that shape. **New REST endpoints** (`superset/extensions/storage/api.py`), used internally by the frontend SDK — extension authors do not call these directly: ``` GET/PUT/DELETE /api/v1/extensions/{publisher}/{name}/storage/ephemeral/{key} GET/PUT/DELETE /api/v1/extensions/{publisher}/{name}/storage/persistent/{key} ``` Both accept a `?shared=true` query parameter to target the `.shared` accessor instead of the per-user default. **New database table:** `extension_storage` (see *Migration Plan* below). **New configuration:** - `EXTENSIONS_EPHEMERAL_STORAGE` — Flask-Caching config for Tier 2 (cache backend, max TTL). Defaults to the existing `SupersetMetastoreCache`, so no new infrastructure is required out of the box; operators running Redis in production can point this at it exactly as they would `CACHE_CONFIG`. - `EXTENSIONS_PERSISTENT_STORAGE` — exposes `QUOTA_PER_EXTENSION` (default 100 MB). Enforced as a hard rejection at write time in `ExtensionStorageDAO.set()`: usage is computed as the sum of stored byte sizes across all of an extension's rows (global, every user's, and shared), and a write that would push the extension over quota raises `ExtensionStorageQuotaExceeded` (HTTP 413) rather than partially writing. Overwriting an existing key nets out that key's own current size against usage first, so replacing a value with one of the same or smaller size never spuriously fails even when the extension is already near quota. ## New dependencies None. Tier 3 encryption reuses `sqlalchemy-utils`' `EncryptedType`, already a dependency via Superset's existing encrypted-field infrastructure. Tier 2 reuses the existing Flask-Caching integration. No new NPM or PyPI packages are introduced. ## Migration Plan and Compatibility **Webpack configuration change.** Existing extensions that do not use storage are unaffected: nothing about loading, registration, or any other existing SDK surface changes for them. A webpack config change is only required to use `getContext().storage`. The host's own `webpack.config.js` declares `@apache-superset/core` as a Module Federation shared singleton (`shared: { '@apache-superset/core': { singleton: true, eager: true } }`). The per-extension isolation mechanism described above depends on being able to override what a given extension's container resolves that shared singleton to. For that override to take effect, **the extension's own webpack config must also declare `@apache-superset/core` as a Module Federation shared singleton** — i.e. add ```js "@apache-superset/core": { singleton: true, import: false }, ``` to the `shared` block of the extension's own `ModuleFederationPlugin` config, alongside the existing `react`/`react-dom`/`antd` entries. The `superset-extensions-cli` scaffolding template (`webpack.config.js.j2`) has been updated to include this by default, removing the prior `externalsType: "window"` external mapping it used instead (which does not participate in Module Federation's share-scope resolution and would otherwise conflict with the new shared entry). Every newly-scaffolded extension is therefore ready to use `getContext().storage` without any manual webpack edit; only extensions scaffolded before this change need the one-line addition above. Until the CLI template is updated, this is a manual step documented in `docs/developer_docs/extensions/storage.md`. No version gating or feature flag is needed on the host side — an extension either declares the shared singleton and gets isolated storage, or doesn't and its `getContext()` calls (if any) fail the same way they always have (host default throws, since there's no bound context to return). **Database migration:** one new table, `extension_storage` (`2026-06-23_12-00_e5f6a7b8c9d0_add_extension_storage_table.py`). Purely additive — no existing table is altered, no existing row is touched. The migration is reversible (`downgrade()` drops the two indexes then the table). ## Rejected Alternatives **Global "current execution" tracker for ambient `getContext()`** (the pattern Svelte and React use internally for `getContext`/`useContext`). Rejected because it relies on synchronous call-stack tracking: a global pointer is set immediately before running tracked code and cleared right after. This breaks the moment `getContext()` is called from inside an async callback — e.g. a storage helper that calls `getContext()` lazily inside an async function triggered from a UI event handler, a common pattern for any extension that lazily loads or persists state — because by the time the callback runs, the tracker has already been cleared or reassigned to a different extension. Frameworks that need ambient access to survive async boundaries (e.g. Nuxt's `unctx`) back the tracker with `AsyncLocalStorage`; no browser equivalent exists today (`AsyncContext` is still a TC39 proposal). The Module Federation share-scope override this proposal uses instead is immune to this problem because i solation is keyed by module identity, set once at load time, not by call-stack timing — `getContext()` returns the right thing regardless of when or how deep in an async chain it's called. **Explicit `activate(context)` entrypoint** (the VS Code model: the host calls a well-known exported function and passes a context object as a plain argument, rather than extensions retrieving it via an ambient call). Rejected in favor of the ambient model for developer experience: an `activate(context)` entrypoint requires an extension to receive the context once, at startup, and either thread it as a parameter through every function that eventually needs storage or hold onto it in some module-level variable for later use. The ambient model avoids both — any function, anywhere in the extension, at any point in its lifecycle, calls `getContext()` and gets the right object back, with nothing to plumb through call signatures and nothing to remember to stash. This also fits how extensions are already structured today: they register commands/views/menus as top-level module-evaluation side effects on import, and already call `getContext()` lazily from deep inside async UI code rather than threading a context object down from a single entrypoint, so adopting `activate(context)` would mean rewriting that structure as well. **Reusing the existing `key_value` table for Tier 3** instead of a dedicated `extension_storage` table. Rejected: `key_value` mixes extension data with Superset's own internal use (permalinks, distributed locks, OAuth2 PKCE verifiers, the metastore cache), with no per-extension columns to scope, quota, or bulk-delete by. A dedicated table gives `extension_id` as a first-class, indexed column, making per-extension quota enforcement and uninstall cleanup a single filtered query instead of requiring extension-aware parsing of a shared, general-purpose key space. **A dedicated encryption/key-management scheme for extension secrets** (e.g. a new Fernet-based mechanism with its own rotating key list). Rejected in favor of reusing `EncryptedType` and Superset's existing `SecretsMigrator`-based rotation path. A new scheme would mean extension secrets rotate on a different mechanism, with different operational runbooks, than every other encrypted secret Superset already manages — duplicated complexity for no isolation benefit, since per-user key derivation already prevents cross-user decryption within the existing mechanism. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
