villebro commented on issue #31408: URL: https://github.com/apache/superset/issues/31408#issuecomment-4587222797
A status update on the operator proposed in this SIP: The initial development work is complete, and we're approaching the first release (0.1.0), which will go through an ASF release vote. The operator lives at https://github.com/apache/superset-kubernetes-operator, with docs at https://apache.github.io/superset-kubernetes-operator/. Thanks to everyone who contributed to the discussion here, as it greatly helped shape the initial development work. I want to call out two major changes since the SIP and discussion here: 1. **No per-component child CRDs**: The SIP discussed separate CRDs for the individual components (web server, Celery workers, Beat, Flower, etc.). We tried this approach, but during implementation this turned out to add significant complexity with very limited value (see motivations below). There is a single public Superset CRD, which the parent controller resolves into all parent-owned resources (Deployments, Services, ConfigMaps, lifecycle Jobs, HPAs, PDBs, networking, monitoring). Per-component configuration lives under that one resource. 2. **More focus on lifecycle operations**: The operator runs database migrations on upgrade, database cloning for staging environments, application init, and secret-key rotation as sequenced, status-tracked tasks. This was an area where the Helm chart was most lacking, and where an operator can really bring added value. We plan to expand the lifecycle operations further. The two are connected, and both follow from how Superset actually works. Its components (web server, Celery workers, Beat, Flower, MCP) are parts of a single platform that share one secret key, metastore, and database schema; they aren't independently deployable services, so modeling each as its own CRD doesn't match how Superset is operated. Lifecycle tasks make this concrete: they need coordination across all of an instance's components, and independently-reconciled child CRs would fragment that. For example, the operator drains components by default during migrations, because old pods serving against a database whose schema is mid-migration can deadlock. This didn't come up in the SIP, but it's a problem we've seen over years of running Superset (see example [here](https://github.com/apache/superset/issues/33252)). During the drain the operator can serve a customizable maintenance page, and restores traffic once the new version is healthy. Rolling updates remain supported but a re opt-in, and generally not recommended. The resulting model is close to the existing Apache Superset Helm chart, so the component layout and configuration should be familiar to current chart users, and migration is largely a matter of translating values. The operator is built on current tooling (Go 1.26, built with latest Operator SDK and controller-runtime) and tracks new Kubernetes releases: CI covers the two most recent minors, plus best-effort coverage for the newest when Kind releases aren't in sync with k8s versions. Try it on a local kind cluster: 1. Install the operator (the chart bundles the CRD, with [version](https://apache.github.io/superset-kubernetes-operator/reference/downloads/?h=versions#chart-versions) `0.0.0-dev` tracking `main` branch on the repo): ```shell helm install superset-operator \ oci://ghcr.io/apache/superset-kubernetes-operator/charts/superset-operator \ --version 0.0.0-dev --namespace superset-operator-system --create-namespace ``` 2. From a checkout of the repo, bring up the dev dependencies (Postgres + Valkey) and a comprehensive Superset instance (web servers, Celery beat, Celery workers, Celery Flower, MCP): ```shell kubectl apply -f config/samples/dev-dependencies.yaml kubectl apply -f config/samples/superset_v1alpha1_superset_dev_full.yaml ``` 3. Once the migrate/init tasks finish, open the UI and log in with admin / admin: ```shell kubectl port-forward svc/superset-dev-web-server 8088:8088 ``` How you can help: - The API is still `v1alpha1`, so issues and design feedback now will help shape the 0.1.0 API. - If you currently run Superset via the official Helm chart, please review the [Helm migration guide](https://apache.github.io/superset-kubernetes-operator/user-guide/migration/). It's written against chart 0.15.5 / appVersion: 5.0.0 and documents the deliberate differences (bring-your-own database and cache, Secret-based secrets, managed lifecycle tasks, Gateway-API-first routing) and known parity gaps. Feedback before the vote is welcome. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
