mistercrunch opened a new issue, #29546:
URL: https://github.com/apache/superset/issues/29546
[STILL A DRAFT - MORE THINKING REQUIRED]
## Motivation
Managing database migrations in Superset currently poses challenges such as
deadlocks and downtime during DDL changes. We aim to improve the process to
ensure smooth blue/green deployments, reduce downtime, and handle complex
migrations effectively.
## Proposed Change
1. **Impose Blue/Green-Safe Migrations as part of minor releases**:
- Implement migrations that support both pre- and post-PR versions of the
app.
- Use feature flags to control migration-dependent features. Operators
can run migrations while keeping flags off, then flip them post-migration.
Ensure migrations aren't needed when the feature flag is false.
- Each blue/green safe migration should add a release not in
`UPDATING.md` as to the migration and related feature flag. Release managers
would take note to flip a certain number of flags post-migration.
1. **Accumulated Cleanup Migrations**: Defer non-critical cleanup migrations
(e.g., column deletions) to major version upgrades, in a file
(`superset/migration/future.py`) imposing a certain structure that's compatible
with alembic-based approach.
1. **Complex Migrations During Release Windows**: Hold back complex,
non-blue/green-safe migrations for specific release windows to minimize risks
and ensure resource availability.
1. **Major release migration bundling**: On major version releases, the
"future" migrations are bundle into a chain of alembic migrations, and it's
well understood that these migration require downtime. PRs that were held up
for migrations that require downtime are also merged. Major version also flip
the default feature flags to true and/or removes related codepaths
1. **Enforce** through a new `CODEOWNERS` policy specific to migration. Have
a few committers familiar with this SIP enforce as code owners where migrations
are concerned.
## Examples
Say for a simple migration where I'd add a `is_published` flag to
dashboards, I'd need to:
- set up a feature flag referencing the migration, `IS_PUBLISHED_PR2387`
where 2387 represents the PR id that contains the related migration
-
## New or Changed Public Interfaces
- Introduce new feature flags for migration-dependent features.
- Update CLI tools to support deferred and bundled migrations.
## New Dependencies
- None.
## Migration Plan and Compatibility
- Detailed documentation on using feature flags for migrations.
- Establish a tracking system for deferred migrations to ensure they are
bundled and tested before major releases.
## Rejected Alternatives
- Immediate application of all migrations, which increases the risk of
downtime and deadlocks.
- Manual handling of complex migrations without specific release windows,
leading to unpredictable deployment issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org
