sushu0 opened a new pull request, #45790: URL: https://github.com/apache/superset/pull/45790
### SUMMARY Fixes #45460, following the agreed migration approach in https://github.com/apache/superset/issues/45460#issuecomment-6423140925. Older database forms stored metadata cache timeouts as strings. With cachelib 0.17, a stored value such as `"schema_cache_timeout": "3600"` reaches the cache backend and raises a TypeError. Add a data-only migration that converts non-negative integer strings in `catalog_cache_timeout`, `schema_cache_timeout` and `table_cache_timeout` to integers, including `"0"` to `0`. Only rows with a conversion are updated. Existing integers, missing keys, negative/invalid values, malformed JSON, non-object configurations and unknown settings are preserved. Invalid configurations are left for operator repair rather than assigned a guessed default. Decimal-aware JSON decoding preserves unrelated high-precision and very large numeric settings when an affected row is serialized again; the shared JSON wrapper does not expose this decoding option. Revision `9670ab7f6bc6` follows the current master head `d623a0cb6bb0`. This PR does not choose a 7.0 RC cherry-pick order; release-branch migration ordering remains a release-manager decision. ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF Not applicable; data migration only. A legacy string timeout causes the actual catalog/schema/table cached model methods to fail before migration. Reading the migrated row through the same methods writes successfully to a real cachelib SimpleCache. ### TESTING INSTRUCTIONS - Without the normalization rewrite, the migration regressions had **10 failures / 16 passes**. The cache cases first assert the real cachelib TypeError on stored legacy rows, then verify successful writes after upgrade. - `python -m pytest tests/unit_tests/migrations/test_normalize_metadata_cache_timeouts.py tests/unit_tests/migrations/test_single_migration_head.py tests/unit_tests/utils/cache_test.py -q` — **45 passed**. Covers all three cache paths, zero, mixed connections, invalid/unrelated data, Decimal preservation, idempotent no-write behavior, transaction rollback and downgrade. - `superset db heads` reports exactly `9670ab7f6bc6 (head)`; the project's single-head test also passes. - Changed-file Ruff and `python -m py_compile` passed. Staged-file pre-commit passed all applicable checks except the three native Windows `signal.SIGALRM`/`signal.alarm` errors in untouched `superset/utils/core.py`. The same mypy hook environment with `--platform linux --check-untyped-defs` passed both new Python files. Pylint excludes migration and test files under the repository configuration. For deployment-level verification, use a disposable metadata database with legacy string timeouts in `dbs.extra`, run `superset db upgrade`, inspect the integer values and load the catalog/schema/table lists with metadata caching configured. Local tests use actual SQLite rows, Alembic MigrationContext/Operations and project model/cache code; a full production metadata upgrade, live warehouse, browser flow and MySQL/PostgreSQL migration were not exercised locally. ### ADDITIONAL INFORMATION - [x] Has associated issue: #45460 - [ ] Required feature flags: - [ ] Changes UI - [x] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351)) - [ ] Migration is atomic, supports rollback & is backwards-compatible - [x] Confirm DB migration upgrade and downgrade tested - [x] Runtime estimates and downtime expectations provided - [ ] Introduces new feature or API - [ ] Removes existing feature or API **Transaction and downgrade:** updates use the Alembic connection without an independent commit; a real SQLite test verifies that the enclosing transaction can roll back the changes. Downgrade deliberately leaves integer values intact: older versions already accept integers, and the migration cannot distinguish an original integer from a converted value or a value saved after upgrade. The combined reversibility checkbox above is left unchecked because the original JSON types are not reconstructed. **Runtime/locks:** one SELECT over `dbs.id`/`dbs.extra`, followed by one UPDATE per affected database connection: O(number of configured connections), independent of warehouse row counts. There is no DDL or table rebuild. Changed metadata rows can remain locked until the normal Alembic transaction completes; use the usual upgrade window. Production wall-clock time and downtime were not measured. AI assistance disclosure: OpenAI Codex assisted with investigation, implementation, test preparation, automated review and local validation. This PR does not claim human-executed validation or prior human review of the new migration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
