sushu0 commented on code in PR #45790: URL: https://github.com/apache/superset/pull/45790#discussion_r4408089813
########## superset/migrations/versions/2026-10-27_00-00_9670ab7f6bc6_normalize_metadata_cache_timeouts.py: ########## @@ -0,0 +1,96 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Normalize legacy database metadata cache timeouts. + +Older database forms stored metadata cache timeouts as strings. Convert only +recognized non-negative integer strings, preserving malformed configurations +and unrelated fields instead of guessing at their intended values. + +Revision ID: 9670ab7f6bc6 +Revises: f72b6e9c104a +Create Date: 2026-10-27 00:00:00.000000 +""" + +from decimal import InvalidOperation + +# The shared JSON wrapper cannot decode decimals without losing stored precision. +import simplejson # noqa: TID251 +from alembic import op +from sqlalchemy import column, Integer, select, table, Text + +from superset.utils import json + +revision = "9670ab7f6bc6" +down_revision = "f72b6e9c104a" + +_DATABASES = table("dbs", column("id", Integer), column("extra", Text)) +_TIMEOUT_KEYS = ( + "catalog_cache_timeout", + "schema_cache_timeout", + "table_cache_timeout", +) + + +def upgrade() -> None: + """Convert legacy non-negative integer strings to integer timeouts.""" + connection = op.get_bind() + for database_id, raw_extra in connection.execute( + select(_DATABASES.c.id, _DATABASES.c.extra).with_for_update() + ): + try: + extra = simplejson.loads(raw_extra, use_decimal=True) + except (TypeError, ValueError, InvalidOperation): + continue + if not isinstance(extra, dict) or not isinstance( + extra.get("metadata_cache_timeout"), dict + ): + continue + timeouts = extra["metadata_cache_timeout"] + + changed = False + for key in _TIMEOUT_KEYS: + value = timeouts.get(key) + if not isinstance(value, str): + continue + try: + timeout = int(value) Review Comment: Addressed in e4bf004e212108b1858093a21b6378ca633ab367. Conversion is now gated by `value.isascii() and value.isdigit()` before `int()`. The redundant negative-result check is removed; the `ValueError` guard remains for digit strings that Python cannot parse. The regression cases preserve underscores, surrounding whitespace/newlines, a leading plus and non-ASCII digits, while accepting zero and leading zeros. These new checks failed against the previous migration runtime and pass after the change. The final migration/single-head/cache/JSON suite passed all 99 tests. ########## superset/migrations/versions/2026-10-27_00-00_9670ab7f6bc6_normalize_metadata_cache_timeouts.py: ########## @@ -0,0 +1,96 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Normalize legacy database metadata cache timeouts. + +Older database forms stored metadata cache timeouts as strings. Convert only +recognized non-negative integer strings, preserving malformed configurations +and unrelated fields instead of guessing at their intended values. + +Revision ID: 9670ab7f6bc6 +Revises: f72b6e9c104a +Create Date: 2026-10-27 00:00:00.000000 +""" + +from decimal import InvalidOperation + +# The shared JSON wrapper cannot decode decimals without losing stored precision. +import simplejson # noqa: TID251 +from alembic import op +from sqlalchemy import column, Integer, select, table, Text + +from superset.utils import json + +revision = "9670ab7f6bc6" +down_revision = "f72b6e9c104a" + +_DATABASES = table("dbs", column("id", Integer), column("extra", Text)) +_TIMEOUT_KEYS = ( + "catalog_cache_timeout", + "schema_cache_timeout", + "table_cache_timeout", +) + + +def upgrade() -> None: + """Convert legacy non-negative integer strings to integer timeouts.""" + connection = op.get_bind() + for database_id, raw_extra in connection.execute( + select(_DATABASES.c.id, _DATABASES.c.extra).with_for_update() + ): + try: + extra = simplejson.loads(raw_extra, use_decimal=True) + except (TypeError, ValueError, InvalidOperation): + continue + if not isinstance(extra, dict) or not isinstance( + extra.get("metadata_cache_timeout"), dict + ): + continue + timeouts = extra["metadata_cache_timeout"] + + changed = False + for key in _TIMEOUT_KEYS: + value = timeouts.get(key) + if not isinstance(value, str): + continue + try: + timeout = int(value) + except ValueError: + continue Review Comment: Addressed in e4bf004e212108b1858093a21b6378ca633ab367. Skipped JSON now emits a warning with the database ID and a static parse/range reason. Strings left unconverted emit a warning with that ID, the known metadata timeout key and a static reason. Neither path logs configuration values, raw JSON, exception messages or a traceback. The caplog regressions verify that invalid rows remain unchanged, the ID/key or JSON reason is present, and synthetic sensitive configuration text and values are absent. The final matching suite passed all 99 tests. ########## superset/migrations/versions/2026-10-27_00-00_9670ab7f6bc6_normalize_metadata_cache_timeouts.py: ########## @@ -0,0 +1,96 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Normalize legacy database metadata cache timeouts. + +Older database forms stored metadata cache timeouts as strings. Convert only +recognized non-negative integer strings, preserving malformed configurations +and unrelated fields instead of guessing at their intended values. + +Revision ID: 9670ab7f6bc6 +Revises: f72b6e9c104a +Create Date: 2026-10-27 00:00:00.000000 +""" + +from decimal import InvalidOperation + +# The shared JSON wrapper cannot decode decimals without losing stored precision. +import simplejson # noqa: TID251 +from alembic import op +from sqlalchemy import column, Integer, select, table, Text + +from superset.utils import json + +revision = "9670ab7f6bc6" +down_revision = "f72b6e9c104a" + +_DATABASES = table("dbs", column("id", Integer), column("extra", Text)) +_TIMEOUT_KEYS = ( + "catalog_cache_timeout", + "schema_cache_timeout", + "table_cache_timeout", +) + + +def upgrade() -> None: + """Convert legacy non-negative integer strings to integer timeouts.""" + connection = op.get_bind() + for database_id, raw_extra in connection.execute( Review Comment: Addressed in e4bf004e212108b1858093a21b6378ca633ab367. The SELECT result is materialized with `.all()` before the update loop starts. A regression wraps actual SQLite execution with a pending-read guard that refuses an UPDATE while the SELECT remains unconsumed; it failed against the prior loop and passes after materialization. The existing mixed-row, CAS and two-row transaction rollback tests also remain passing in the 99-test matching suite. This is a local cursor-ordering regression, not a live PostgreSQL/MySQL streaming-cursor test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
