younsl opened a new pull request, #392: URL: https://github.com/apache/superset-kubernetes-operator/pull/392
## Summary This PR adds an optional pre-upgrade metastore backup task, spec.lifecycle.backup. The operator never runs superset db downgrade and re-runs migrate on any lifecycle image change, so the only way back from a failed or unwanted upgrade is restoring the metastore as it was before the upgrade. The operator already describes the pre-upgrade backup as the safety net for this, but no such task exists yet. This task takes that snapshot automatically in front of migrate and rotate, verifies it, records it in status, and blocks the upgrade when it cannot, without taking the running instance down. [](https://raw.githubusercontent.com/younsl/superset-kubernetes-operator/pr-assets/lifecycle-backup/lifecycle-backup-architecture.svg) ## Background #135 by @villebro proposed a backup task and an approval-gated restore together with several breaking lifecycle changes, and was closed on 2026-07-02 to be broken up into smaller PRs. The breaking parts have since landed separately: the clone task was renamed to seed (#167), migrate now runs on any image change and the downgrade block was removed (#173), and the status version fields became tag (#174). Since #173, the documentation and code comments name the optional pre-upgrade backup as the safety net for upgrades, but the backup part of #135 was never re-landed. This PR carries only that part, on current main. It keeps the direction of #135: a lifecycle task that runs in front of migrate, deterministic Job naming, results recorded in status, and skipping the first install. Restore automation and object storage destinations are left for follow-ups. Compared to the backup part of #135: - The dump is verified by reading it back in full. In #135 the default script piped pg_dump into cat under set -e without pipefail, so a failed dump could still complete the Job and be recorded as a backup. - Backup is outside the task checksum cascade. In #135 its inputs, including destination and retention, fed the migrate checksum, so editing backup settings re-ran migrate, rotate, and init. - It runs before drain by default, so it adds no downtime and a failure leaves the current version serving. - Retention only prunes this Superset's backups, matched by UID through their manifests. #135 pruned every dump in the directory, which could delete another instance's backups on a shared volume. - The default MySQL image is mysql:8.4 (#389). #135 used mysql:8-alpine, which does not exist. Thanks to @villebro for the original design; review from you would be especially welcome. ## Details **When it runs** - At most once per lifecycle run, in front of the first pending migrate or rotate task, which covers image upgrades, migrate.trigger, and secret key rotation. Config-only changes that re-run only init never trigger it, and a retried migrate reuses the snapshot taken before the first attempt. - Before maintenance and drain by default (requiresDrain defaults to false), while the current version keeps serving. A slow backup adds no downtime, and a failed backup blocks the upgrade without taking the instance down. Setting requiresDrain to true drains first, so the snapshot also contains the writes made right before the upgrade. - In Supervised mode, only after the upgrade is approved. On a first install the database may not exist yet, so a confirmed-absent database is skipped and reported as Skipped. - It is outside the task checksum cascade: enabling or editing backup never re-runs another task. A golden test pins the task checksums of representative existing resources to the values computed on main. **Integrity** - The default command checks that the destination is writable, the database is reachable, and the pg_dump client is not older than the server, then dumps to a .partial file. - PostgreSQL dumps are read back in full with pg_restore to /dev/null, which decompresses and checks every data block. A TOC listing (pg_restore with the list option) passes truncated and corrupt archives, which we reproduced locally. MySQL dumps must end with the completion trailer. - The file is renamed into place only after verification, hashed again after the rename (post-commit check), and committed by writing a JSON manifest next to it (image tags, Alembic revision, database and client versions, size, SHA-256). Dumps are owner only (0600) and never overwritten. **Operability** - The last five verified backups are listed in status.lifecycle.backups with location, size, SHA-256, Alembic revision, and the image to restore with. The operator also emits BackupCompleted and BackupSkipped events. - Failures show the failing step and the tail of the tool error in status.lifecycle.backup.message instead of a bare Job failure, for example: pg_dump 16 cannot dump PostgreSQL 17; set spec.lifecycle.backup.image.tag to 17-alpine. - By default completed backups are never deleted. Optional retention.keepLast prunes only this Superset's backups, matched by the UID in their manifests, after a new backup is committed, so a volume shared with other instances is safe. - The user guide covers what is not backed up (SECRET_KEY, database roles, Valkey), a restore runbook that verifies the checksum first, and known limitations. **Security** - No new RBAC. Metastore credentials reach the Job only as secretKeyRef env vars, the Job never receives SECRET_KEY, and the manifest holds no secrets. - The result comes back through the container termination message, read only from Pods controlled by the backup Job. Because backup.command can be user supplied, the message is decoded strictly (unknown fields rejected, every field validated) before anything is written to status, and failure output goes through the existing credential redaction. **Other changes** - make test-e2e now passes an explicit go test timeout (E2E_TIMEOUT, default 45m), because the full suite now takes longer than the 10 minute go test default. - New fuzz target for the result parser, envtest CEL cases for retention, and updated docs (lifecycle, configuration, security, internals, migration, AGENTS.md) and release notes. <details> <summary>Kind e2e results: 31 of 31 specs passed</summary> Environment: kind v0.33.0 (podman provider), node image kindest/node:v1.37.0, Go 1.27.1, operator image built from this branch. Databases run in the restricted PSS namespace with [postgres:17-alpine](https://hub.docker.com/_/postgres) and [mysql:8.4](https://hub.docker.com/_/mysql). The failure scenario uses a postgres:16-alpine client against the 17 server. Result: Ran 31 of 31 specs in 677.7 seconds, 31 passed, 0 failed, 0 pending, 0 skipped. | Suite | Spec | Result | Time | |---|---|---|---| | Backup, PostgreSQL | takes a first-run backup before the initial migrate | Passed | 31.8s | | Backup, PostgreSQL | runs the backup Job with restricted PSS compatible defaults | Passed | 0.2s | | Backup, PostgreSQL | snapshots while the web server still serves, then drains and migrates | Passed | 66.1s | | Backup, PostgreSQL | does not back up on config-only changes | Passed | 8.0s | | Backup, PostgreSQL | does not redo a completed backup when only its spec changes | Passed | 20.1s | | Backup, PostgreSQL | blocks migrate on a failed backup, keeps serving, and reports why | Passed | 131.2s | | Backup, PostgreSQL | reuses the snapshot when a failed migrate is retried | Passed | 71.7s | | Backup, PostgreSQL | prunes this instance's older backups with retention.keepLast | Passed | 67.2s | | Backup, MySQL | dumps MySQL with the default image before each migrate | Passed | 89.1s | | Backup, first install | skips the dump when the metastore database does not exist yet | Passed | 43.0s | | Backup, MySQL first install | skips the dump, then createDatabase creates the database | Passed | 17.3s | | Backup, restore runbook | takes a pre-upgrade snapshot before an unwanted upgrade | Passed | 27.5s | | Backup, restore runbook | keeps dumps unreadable to other UIDs | Passed | 6.9s | | Backup, restore runbook | restores the snapshot with the runbook and resumes reconciliation | Passed | 19.6s | | Lifecycle, deployment, CRD validation, manager | 17 existing specs | Passed | 21.5s total | What the backup specs assert beyond pass or fail: - Every dump has a manifest whose SHA-256 matches the file, each PostgreSQL dump is read back in full, no .partial files remain, and dumps are mode 0600. - status.lifecycle.backups[0] points at the newest dump with its checksum, and the task message starts with Backup written. - The BackupCompleted event precedes the DrainingStarted event, so the backup finished before drain. - With an old pg_dump client the backup fails with the preflight message, migrate does not run, and the web server Deployment stays up throughout. - After retention.keepLast is set to 2, exactly the two newest backups and their manifests remain. - The restore runbook picks the dump from status, verifies it against the manifest checksum, and restores it. </details> <details> <summary>Script checks against real databases (local containers)</summary> The default scripts were run against postgres:17-alpine and mysql:8.4 containers, and they pass shellcheck. | Scenario | Result | |---|---| | PostgreSQL and MySQL dump | dump and manifest written, Alembic revision captured | | Wrong password | fails at connect with the server error | | Database missing on first run | skipped with a reason | | Database missing later | fails at connect | | pg_dump 16 client, PostgreSQL 17 server | fails at preflight with the image tag to use | | Destination not writable | fails before connecting, pointing at the PVC and fsGroup | | keepLast 2 on a volume shared with another instance | this instance keeps its two newest, the other instance's backup is untouched | | Truncated or byte-flipped archive | pg_restore with the list option passes both, the full read fails both | </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
