michael-s-molina commented on code in PR #36298: URL: https://github.com/apache/superset/pull/36298#discussion_r2566475049
########## docs/developer_portal/extensions/development.md: ########## @@ -0,0 +1,245 @@ +--- +title: Development +sidebar_position: 5 +--- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Development + +This guide covers everything you need to know about developing extensions for Superset, from project structure to development workflow. + +## Project Structure + +The `apache-superset-extensions-cli` package provides a command-line interface (CLI) that streamlines the extension development workflow. It offers the following commands: + +``` +superset-extensions init: Generates the initial folder structure and scaffolds a new extension project. + +superset-extensions build: Builds extension assets. + +superset-extensions bundle: Packages the extension into a .supx file. + +superset-extensions dev: Automatically rebuilds the extension as files change. +``` + +When creating a new extension with `superset-extensions init <extension-name>`, the CLI generates a standardized folder structure: + +``` +dataset_references/ +├── extension.json +├── frontend/ +│ ├── src/ +│ ├── webpack.config.js +│ ├── tsconfig.json +│ └── package.json +├── backend/ +│ ├── src/ +│ └── dataset_references/ +│ ├── tests/ +│ ├── pyproject.toml +│ └── requirements.txt +├── dist/ +│ ├── manifest.json +│ ├── frontend +│ └── dist/ +│ ├── remoteEntry.d7a9225d042e4ccb6354.js +│ └── 900.038b20cdff6d49cfa8d9.js +│ └── backend +│ └── dataset_references/ +│ ├── __init__.py +│ ├── api.py +│ └── entrypoint.py +├── dataset_references-1.0.0.supx +└── README.md +``` + +The `extension.json` file serves as the declared metadata for the extension, containing the extension's name, version, author, description, and a list of capabilities. This file is essential for the host application to understand how to load and manage the extension. + +The `frontend` directory contains the source code for the frontend components of the extension, including React components, styles, and assets. The `webpack.config.js` file is used to configure Webpack for building the frontend code, while the `tsconfig.json` file defines the TypeScript configuration for the project. The `package.json` file specifies the dependencies and scripts for building and testing the frontend code. + +The `backend` directory contains the source code for the backend components of the extension, including Python modules, tests, and configuration files. The `pyproject.toml` file is used to define the Python package and its dependencies, while the r`equirements.txt` file lists the required Python packages for the extension. The `src` folder contains the functional backend source files, `tests` directory contains unit tests for the backend code, ensuring that the extension behaves as expected and meets the defined requirements. + +The `dist` directory is built when running the `build` or `dev` command, and contains the files that will be included in the bundle. The `manifest.json` file contains critical metadata about the extension, including the majority of the contents of the `extension.json` file, but also other build-time information, like the name of the built Webpack Module Federation remote entry file. The files in the `dist` directory will be zipped into the final `.supx` file. Although this file is technically a zip archive, the `.supx` extension makes it clear that it is a Superset extension package and follows a specific file layout. This packaged file can be distributed and installed in Superset instances. + +The `README.md` file provides documentation and instructions for using the extension, including how to install, configure, and use its functionality. + +## Extension Metadata + +The `extension.json` file contains all metadata necessary for the host application to understand and manage the extension: + +``` json +{ + "name": "dataset_references", + "version": "1.0.0", + "frontend": { + "contributions": { + "views": { + "sqllab.panels": [ + { + "id": "dataset_references.main", + "name": "Dataset references" + } + ] + } + }, + "moduleFederation": { + "exposes": ["./index"] + } + }, + "backend": { + "entryPoints": ["dataset_references.entrypoint"], + "files": ["backend/src/dataset_references/**/*.py"] + }, +} +``` + +The `contributions` section declares how the extension extends Superset's functionality through views, commands, menus, and other contribution types. The `backend` section specifies entry points and files to include in the bundle. + +## Interacting with the Host + +Extensions interact with Superset through well-defined, versioned APIs provided by the `@apache-superset/core` (frontend) and `apache-superset-core` (backend) packages. These APIs are designed to be stable, discoverable, and consistent for both built-in and external extensions. + +**Note**: The `superset_core.api` module provides abstract classes that are replaced with concrete implementations via dependency injection when Superset initializes. This allows extensions to use the same interfaces as the host application. + +### Frontend APIs + +The frontend extension APIs (via `@apache-superset/core`) are organized into logical namespaces such as `authentication`, `commands`, `extensions`, `sqlLab`, and others. Each namespace groups related functionality, making it easy for extension authors to discover and use the APIs relevant to their needs. For example, the `sqlLab` namespace provides events and methods specific to SQL Lab, allowing extensions to react to user actions and interact with the SQL Lab environment: + +``` typescript +export const getCurrentTab: () => Tab | undefined; + +export const getDatabases: () => Database[]; + +export const getTabs: () => Tab[]; + +export const onDidChangeEditorContent: Event<string>; + +export const onDidClosePanel: Event<Panel>; + +export const onDidChangeActivePanel: Event<Panel>; + +export const onDidChangeTabTitle: Event<string>; + +export const onDidQueryRun: Event<Editor>; + +export const onDidQueryStop: Event<Editor>; +``` + +The following code demonstrates more examples of the existing frontend APIs: + +``` typescript +import { core, commands, sqlLab, authentication, Button } from '@apache-superset/core'; +import MyPanel from './MyPanel'; + +export function activate(context) { + // Register a new panel (view) in SQL Lab and use shared UI components in your extension's React code + const panelDisposable = core.registerView('my_extension.panel', <MyPanel><Button/></MyPanel>); + + // Register a custom command + const commandDisposable = commands.registerCommand('my_extension.copy_query', { + title: 'Copy Query', + execute: () => { + // Command logic here + }, + }); + + // Listen for query run events in SQL Lab + const eventDisposable = sqlLab.onDidQueryRun(editor => { + // Handle query execution event + }); + + // Access a CSRF token for secure API requests + authentication.getCSRFToken().then(token => { + // Use token as needed + }); + + // Add all disposables for automatic cleanup on deactivation + context.subscriptions.push(panelDisposable, commandDisposable, eventDisposable); +} +``` + +### Backend APIs + +Backend APIs (via `apache-superset-core`) follow a similar pattern, providing access to Superset's models, sessions, and query capabilities. Extensions can register REST API endpoints, access the metadata database, and interact with Superset's core functionality. + +Extension endpoints are registered under a dedicated `/extensions` namespace to avoid conflicting with built-in endpoints and also because they don't share the same version constraints. By grouping all extension endpoints under `/extensions`, Superset establishes a clear boundary between core and extension functionality, making it easier to manage, document, and secure both types of APIs. + +``` python +from superset_core.api.models import Database, get_session +from superset_core.api.daos import DatabaseDAO +from superset_core.api.rest_api import add_extension_api +from .api import DatasetReferencesAPI + +# Register a new extension REST API +add_extension_api(DatasetReferencesAPI) + +# Fetch Superset entities via the DAO to apply base filters that filter out entities +# that the user doesn't have access to +databases = DatabaseDAO.find_all() + +# ..or apply simple filters on top of base filters +databases = DatabaseDAO.filter_by(uuid=database.uuid) +if not databases: + raise Exception("Database not found") + +return databases[0] + +# Perform complex queries using SQLAlchemy Query, also filtering out +# inaccessible entities +session = get_session() +databases_query = session.query(Database).filter( + Database.database_name.ilike("%abc%") +) +return DatabaseDAO.query(databases_query) + +# Bypass security model for highly custom use cases +session = get_session() +all_databases_containing_abc = session.query(Database).filter( + Database.database_name.ilike("%abc%") +).all() +``` + +In the future, we plan to expand the backend APIs to support configuring security models, database engines, SQL Alchemy dialects, etc. + +## Development Mode + +Development mode accelerates extension development by letting developers see changes in Superset quickly, without the need for repeated packaging and uploading. To enable development mode, set the `LOCAL_EXTENSIONS` configuration in your `superset_config.py`: Review Comment: `LOCAL_EXTENSIONS` is not a boolean value; it is an array containing the paths to the locations of extensions, which may reside in a different repository. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
