villebro opened a new pull request, #40079:
URL: https://github.com/apache/superset/pull/40079
### SUMMARY
Fixes several issues in superset re-encrypt-secrets that surfaced with the
addition of the semantic_layers table (PK = uuid, with an encrypted
configuration column):
- Works on tables without an integer id PK. SecretsMigrator previously
hardcoded `SELECT id, ... FROM <table> and WHERE id = :id`. It now derives the
actual primary key column(s) from the Table metadata, so tables like
`semantic_layers` (whose PK is `uuid`) can be rotated without error.
- Idempotent re-runs. Re-encryption now checks whether the current key can
already decrypt each value and skips if so, regardless of what
previous_secret_key is supplied. Previously the check order was reversed, so
re-running with a previous key that still happened to decrypt (e.g. after a
successful rotation, or accidentally passing the current key as "previous")
would re-encrypt every value on every run.
- Per-column outcome summary. run() logs and returns a
ReEncryptStats(re_encrypted, skipped, null, failed) count at the end of the
flow. `NULL` values are counted separately from "already current" skips so
operators can tell them apart. On failure, the transaction rolls back and the
count of failed values is surfaced.
- Cleaner CLI exit codes.
- Missing `PREVIOUS_SECRET_KEY` → exit 0 with a yellow "nothing to
re-encrypt" notice (was exit 1). This lets scheduled rotation scripts run
unconditionally after a rotation is complete without starting to fail.
- Any failed field → exit 1 with a formatted "Re-encryption failed: ..."
message (was a partial catch on ValueError only, letting generic exceptions
leak as uncaught tracebacks).
- Successful run → prints the stats summary in green.
### BEFORE
```
2026-05-12 15:55:27,891:INFO:superset.utils.encrypt:Collecting info for re
encryption
2026-05-12 15:55:27,891:INFO:superset.utils.encrypt:Processing table: dbs
Traceback (most recent call last):
File
"/Users/ville/projects/superset/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py",
line 1910, in _execute_context
self.dialect.do_execute(
File
"/Users/ville/projects/superset/venv/lib/python3.11/site-packages/sqlalchemy/engine/default.py",
line 736, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: no such column: id
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/ville/projects/superset/venv/bin/superset", line 6, in
<module>
sys.exit(superset())
^^^^^^^^^^
File
"/Users/ville/projects/superset/venv/lib/python3.11/site-packages/click/core.py",
line 1442, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/Users/ville/projects/superset/venv/lib/python3.11/site-packages/click/core.py",
line 1363, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File
"/Users/ville/projects/superset/venv/lib/python3.11/site-packages/click/core.py",
line 1830, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/Users/ville/projects/superset/venv/lib/python3.11/site-packages/click/core.py",
line 1226, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/Users/ville/projects/superset/venv/lib/python3.11/site-packages/click/core.py",
line 794, in invoke
return callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/Users/ville/projects/superset/venv/lib/python3.11/site-packages/click/decorators.py",
line 34, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/Users/ville/projects/superset/venv/lib/python3.11/site-packages/flask/cli.py",
line 358, in decorator
return __ctx.invoke(f, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/Users/ville/projects/superset/venv/lib/python3.11/site-packages/click/core.py",
line 794, in invoke
return callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ville/projects/superset/superset/cli/update.py", line 122, in
re_encrypt_secrets
secrets_migrator.run()
File "/Users/ville/projects/superset/superset/utils/encrypt.py", line 218,
in run
rows = self._select_columns_from_table(conn, column_names, table_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ville/projects/superset/superset/utils/encrypt.py", line 156,
in _select_columns_from_table
return conn.execute(f"SELECT id, {','.join(column_names)} FROM
{table_name}") # noqa: S608
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/Users/ville/projects/superset/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py",
line 1370, in execute
return self._exec_driver_sql(
^^^^^^^^^^^^^^^^^^^^^^
File
"/Users/ville/projects/superset/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py",
line 1674, in _exec_driver_sql
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File
"/Users/ville/projects/superset/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py",
line 1953, in _execute_context
self._handle_dbapi_exception(
File
"/Users/ville/projects/superset/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py",
line 2134, in _handle_dbapi_exception
util.raise_(
File
"/Users/ville/projects/superset/venv/lib/python3.11/site-packages/sqlalchemy/util/compat.py",
line 211, in raise_
raise exception
File
"/Users/ville/projects/superset/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py",
line 1910, in _execute_context
self.dialect.do_execute(
File
"/Users/ville/projects/superset/venv/lib/python3.11/site-packages/sqlalchemy/engine/default.py",
line 736, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such column:
id
[SQL: SELECT id, configuration FROM semantic_layers]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
```
### AFTER
Successful rotation:
```
2026-05-12 16:26:17,793:INFO:superset.utils.encrypt:Collecting info for re
encryption
2026-05-12 16:26:17,794:INFO:superset.utils.encrypt:Re-encryption summary: 2
re-encrypted, 0 skipped, 4 null, 0 failed
2026-05-12 16:26:17,794:INFO:superset.utils.encrypt:All tables processed
Re-encryption complete: 2 re-encrypted, 0 skipped, 4 null, 0 failed.
```
No-op (SECRET_KEY already decrypts values successfully):
```
2026-05-12 16:27:59,631:INFO:superset.utils.encrypt:Collecting info for re
encryption
2026-05-12 16:27:59,634:INFO:superset.utils.encrypt:Re-encryption summary: 0
re-encrypted, 2 skipped, 4 null, 0 failed
2026-05-12 16:27:59,634:INFO:superset.utils.encrypt:All tables processed
Re-encryption complete: 0 re-encrypted, 2 skipped, 4 null, 0 failed.
```
Invalid secrets (neither `SECRET_KEY` nor `PREVIOUS_SECRET_KEY` decrypts
current values):
```
2026-05-12 16:29:35,246:INFO:superset.utils.encrypt:Collecting info for re
encryption
2026-05-12 16:29:35,247:ERROR:superset.utils.encrypt:Column [dbs.password]
cannot be decrypted under the previous or current secret key (ValueError:
Invalid decryption key)
2026-05-12 16:29:35,247:ERROR:superset.utils.encrypt:Column
[dbs.encrypted_extra] cannot be decrypted under the previous or current secret
key (ValueError: Invalid decryption key)
2026-05-12 16:29:35,248:INFO:superset.utils.encrypt:Re-encryption summary: 0
re-encrypted, 0 skipped, 4 null, 2 failed
Re-encryption failed: Re-encryption failed for 2 value(s); transaction
rolled back
```
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in
[SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
