GitHub user mfournioux created a discussion: Is a custom SemanticLayer extension the right way to query an Apache Ossie model stored in Apache Polaris?
## What we are trying to do We want to describe our metrics once, in **Apache Ossie** format, store that definition in **Apache Polaris**, and then build charts and dashboards on it in Superset. The data itself lives in Iceberg tables, queried through a database connection Superset already has. Our stack: | Component | Version | |---|---| | Superset | `7.0.0rc1`, with `SEMANTIC_LAYERS` and `ENABLE_EXTENSIONS` on | | Apache Polaris | `1.8.0` | | Apache Ossie | `0.2.0.dev0` | The extension mechanism itself already works for us: we deployed [`betodealmeida/pandas-semantic-layer`](https://github.com/betodealmeida/pandas-semantic-layer) as a local extension, the connection form renders, and charts against it work. What we are unsure about is the design of an Ossie provider, which is why we are asking before writing code. ## What we found Our starting assumption was that something, somewhere, already knew how to turn an Ossie model into a SQL query. We looked, and could not find it. Here is what we checked: | Component | Turns Ossie into SQL? | What we observed | |---|---|---| | **Polaris 1.8.0** | No | Its semantic-models extension is a CRUD store: create, list, get, update, delete. On a live instance, `query`, `sql`, `compile`, `resolve`, `metrics` and `plan` all return `404`, while `semantic-models` returns `401`. | | **The Ossie spec** | No, deliberately | `expression_language.md` states: *"It does not define a query interface… Query interfaces will be defined in separate documents."* A reference Ossie → SQL compiler is listed as future work. | | **`ossie-converters/polaris`** | No | It creates empty Iceberg tables from a model and drops metrics, relationships and expressions. It never calls the `/semantic-models` API. | | **Superset 7.0.0rc1** | Provides the contract only | `SemanticLayer` and `SemanticView` ship as abstract base classes, and `get_table()` is abstract. | So our reading is that **generating the SQL is the consuming client's job**, which also matches the intent described in apache/polaris#4522: each client reads the same definition, picks its dialect from `expression.dialects[]`, and produces the query itself. **We would like that confirmed, because everything else we plan follows from it.** ### One distinction we want to state clearly It is easy to blur two different things, so to be explicit about what we are and are not reusing: - **Generating SQL.** Superset has `get_sqla_query` in `superset/models/helpers.py`, which builds SQL for Superset's own datasets. It is **not** used for semantic views. `SemanticView.get_query_str()` returns the literal string `"Not implemented for semantic layers"`, and `superset/semantic_layers/mapper.py` hands the request to `semantic_view.get_table()`. So Superset generates no SQL on this path, and we are not trying to reuse that code. - **Executing SQL.** Separately, `Database.execute(sql, options)` takes a SQL string that the caller already produced and runs it against an existing connection. That is the only piece we want to reuse, so that the extension does not need to ship its own driver or handle authentication. In short: **we write the SQL, an existing Superset connection runs it.** ## What we plan to build A backend-only Superset extension acting as that consuming client. Nothing in it is specific to a particular engine: the dialect comes from the model, and execution goes through whichever Superset connection is configured. ``` Ossie model (YAML) │ POST /api/catalog/polaris/v1/{cat}/namespaces/{ns}/semantic-models ▼ Apache Polaris ─────────── Iceberg catalog ───────────┐ │ GET /semantic-models │ ▼ ▼ Superset extension "ossie-semantic-layer" SQL engine OssieSemanticLayer : reads and parses the model ▲ OssieSemanticView : builds the SQL ───────────────┘ runs it via Database.execute() ▼ Explore / dashboards / MCP tools ``` Step by step: 1. `from_configuration()` fetches the model from the Polaris semantic-models API and parses it with the `ossie` Python library, caching it against the `entity-version` Polaris returns. 2. `get_dimensions()` and `get_metrics()` map Ossie fields and metrics onto Superset's types. 3. `get_table()` receives a `SemanticQuery` (metrics, dimensions, filters, order, limit) and builds a `SELECT` from it using **SQLGlot**, in the dialect the model declares through `expression.dialects[]`. Because Ossie expressions are already SQL fragments, this is mostly assembly: project the dimensions, add the aggregates, and derive the joins by walking `relationships[]`. 4. The resulting SQL is executed through `Database.execute()` on an existing Superset connection, and the DataFrame is converted to a `pa.Table`. 5. The generated SQL is returned in `SemanticResult.requests`, so it can be inspected when a chart looks wrong. ## Questions 1. **Is our reading correct** that no component compiles Ossie into SQL today, and that a provider is expected to do it itself? 2. **Is a custom extension the right home for this**, or is an official Ossie provider already planned, in which case we would rather help with that than duplicate it? 3. **Which API should an extension target for the 7.0 line?** On `master`, `from_configuration` takes `cache_token`, `context` and `credentials`, and there is a `SemanticExecutionContext`. None of these exist in `7.0.0rc1`, where the signature is simply `from_configuration(cls, configuration)`. The guide at `docs/developer_docs/extensions/extension-points/semantic-layers.md` documents the `master` signature and is not present in the `7.0.0rc1` tag, so following it on 7.0 produces an extension that fails to load. Is there guidance for extension authors on 7.0? 4. **Is `Database.execute()` the intended way for a provider to run the SQL it generated**, or is a provider expected to open its own connection to the engine? 5. **Is a 7.0-compatible reference provider planned?** `pandas-semantic-layer` is the only working example we found for rc1; `example_provider.py` and `tests/e2e_extensions/semantic_stub/` exist only on `master`. ## Contribution If the approach is confirmed, **we will open a PR.** One thing we noticed while scoping the work: the Ossie → SQL part is not specific to Polaris or to us. Any Superset deployment consuming an Ossie model would need the same logic. If that seems useful to others, we are happy to shape it as something reusable rather than keeping it inside our own extension, and to contribute it wherever maintainers think it belongs. If that is out of scope, we will simply keep it internal and ship the provider on its own. --- *Related: [SIP-182](https://github.com/apache/superset/issues/35003), apache/polaris#4522* GitHub link: https://github.com/apache/superset/discussions/45891 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
