CaseyLeask opened a new issue, #40489:
URL: https://github.com/apache/superset/issues/40489
### Bug description
`POST /api/v1/cachekey/invalidate` removes records from the `cache_keys`
metadata table but does NOT delete the corresponding entries in Redis when the
deployment configures `DATA_CACHE_CONFIG` with a different `CACHE_KEY_PREFIX`
than `CACHE_CONFIG`. Chart query results keep being served from the stale Redis
cache until natural TTL expiry. Subsequent chart views can even re-populate
`cache_keys` rows pointing at the same un-deleted Redis entries.
### Steps to reproduce
Deployment config (typical multi-cache setup):
```python
CACHE_CONFIG = {
"CACHE_TYPE": "RedisCache",
"CACHE_KEY_PREFIX": "superset_", # default cache, used for
metadata/screenshots
"CACHE_REDIS_URL": "redis://.../1",
}
DATA_CACHE_CONFIG = {
"CACHE_TYPE": "RedisCache",
"CACHE_KEY_PREFIX": "superset_data_", # ← different prefix
"CACHE_REDIS_URL": "redis://.../1",
}
STORE_CACHE_KEYS_IN_METADATA_DB = True
FEATURE_FLAGS = {"ENABLE_TEMPLATE_PROCESSING": True}
```
1. View a chart so it caches a result. Redis ends up with a key
`superset_data_<32-char-hash>` (chart query result).
2. Verify `cache_keys.cache_key` stores the bare `<32-char-hash>` (no
prefix) and `datasource_uid` is e.g. `293__table`.
3. Confirm Redis has the cached value: `redis-cli EXISTS
superset_data_<hash>` → `1`.
4. Call invalidate: `POST /api/v1/cachekey/invalidate` with body
`{\"datasource_uids\": [\"293__table\"]}`.
5. Response: `HTTP 201 {}` — reported success.
6. `SELECT COUNT(*) FROM cache_keys WHERE datasource_uid = '293__table'` →
`0` ✓ table side cleared.
7. `redis-cli EXISTS superset_data_<hash>` → still `1` ✗ Redis untouched.
8. Reload the chart in the browser → reads from Redis cache hit → returns
stale data. UI shows e.g. \"Cached 14 hours ago\" indefinitely.
9. Manually `redis-cli DEL superset_data_<hash>` → returns `1` → next chart
load re-queries cleanly.
### Expected results
`POST /api/v1/cachekey/invalidate` deletes both the `cache_keys` rows AND
the corresponding Redis entries, so subsequent chart loads run a fresh query.
### Actual results
Only `cache_keys` rows are deleted. Redis entries persist and continue
serving stale data until 24h TTL expiry. The endpoint logs `\"Some of the cache
keys were not deleted in the list\"` but returns 201 and doesn't surface the
failure to the caller.
### Root cause
`superset/cachekeys/api.py` (master + 4.0.1) at the invalidate handler:
```python
cache_keys = [c.cache_key for c in cache_key_objs] # bare hash strings, no
prefix
if cache_key_objs:
all_keys_deleted = cache_manager.cache.delete_many(*cache_keys)
# ↑ uses the DEFAULT cache (CACHE_CONFIG, prefix \"superset_\")
# but the actual Redis keys were written by data_cache
(DATA_CACHE_CONFIG, prefix \"superset_data_\")
```
flask-caching's `delete_many` prepends the cache backend's own
`CACHE_KEY_PREFIX` to each key passed in. Calling
`cache_manager.cache.delete_many(\"<hash>\")` issues `DEL superset_<hash>` to
Redis — but the actual key Redis holds is `superset_data_<hash>` because that
was written by `cache_manager.data_cache`. The DEL silently misses, returns 0
keys deleted, gets logged as \"Some of the cache keys were not deleted\" and
execution continues to delete the metadata rows.
In deployments where `CACHE_CONFIG.CACHE_KEY_PREFIX ==
DATA_CACHE_CONFIG.CACHE_KEY_PREFIX` (the default), this bug is invisible
because both prefixes match. The bug only manifests when an operator configures
distinct prefixes (the recommended setup for separating data cache from
metadata cache).
### Suggested fix
Use the cache backend that actually wrote the keys —
`cache_manager.data_cache.delete_many(...)` — for chart query result keys. The
chart data cache is the one storing the entries that `cache_keys` tracks (via
`QueryContextProcessor.cache_key` and friends in
`superset/common/query_context_processor.py`).
```diff
- all_keys_deleted = cache_manager.cache.delete_many(*cache_keys)
+ all_keys_deleted = cache_manager.data_cache.delete_many(*cache_keys)
```
(Or use whichever cache backend produced the entries — there may be multiple
if cache keys can originate from different cache backends; in that case the row
needs to capture which backend wrote it.)
### Environment
- Superset version: 4.0.1 (also confirmed present on `master` at HEAD as of
issue filing)
- Deployment: docker image
`apachesuperset.docker.scarf.sh/apache/superset:4.0.1`, helm-deployed on EKS
- Python: bundled 3.10
- Redis: bitnami-helm 6.x, DB 1 used for both default and data cache (with
distinct prefixes)
- Database: Postgres
### Additional context
Empirical confirmation from running deployment:
| Step | Result |
|---|---|
| `SELECT cache_key FROM cache_keys WHERE datasource_uid='293__table' ORDER
BY created_on DESC LIMIT 1;` | `e40c6033ba4eb5233ba3d31e29a6c38d` |
| `redis-cli EXISTS superset_data_e40c6033ba4eb5233ba3d31e29a6c38d` | `1` |
| `redis-cli EXISTS e40c6033ba4eb5233ba3d31e29a6c38d` (bare) | `0` |
| `redis-cli EXISTS superset_e40c6033ba4eb5233ba3d31e29a6c38d` (default
prefix) | `0` |
| `POST /api/v1/cachekey/invalidate {\"datasource_uids\":[\"293__table\"]}`
| `HTTP 201` |
| Re-check `SELECT COUNT(*) FROM cache_keys WHERE
datasource_uid='293__table';` | `0` (cleared) |
| Re-check `redis-cli EXISTS superset_data_e40c6033...` | **`1`** (not
deleted) |
| Chart UI \"Cached X ago\" timestamp on next reload | unchanged (cache hit
on stale Redis entry) |
| `redis-cli DEL superset_data_e40c6033...` | `1` |
| Chart UI \"Cached X ago\" on next reload | \"Cached just now\" (cache miss
→ fresh query) |
### Checklist
- [x] I have searched Superset docs and Slack and didn't find a solution
- [x] I have searched the GitHub issue tracker and didn't find a similar bug
report
- [x] I have checked the Superset upstream master branch for code that might
fix this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
