betodealmeida opened a new pull request, #40221: URL: https://github.com/apache/superset/pull/40221
<!--- Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/ Example: fix(dashboard): load charts correctly --> ### SUMMARY <!--- Describe the change below, including rationale and design decisions --> Adds a smart, containment-aware cache for the semantic-layer query path. The cache understands that one query's result can satisfy *another* query whose filters/limit/dimensions/metrics are narrower — and serves those subsequent requests from a locally post-processed `DataFrame` instead of hitting the warehouse. In dashboards and Explore the win is concrete: changing a filter range, shrinking a top-N, dropping a metric, or rolling up a dimension no longer round-trips to the database when a broader prior result is already cached. A small UI affordance in Explore signals that a chart was served from the semantic-layer cache (distinct from the existing per-query result cache). #### How it works ##### Cache layout * **Index**: one Redis key per `(view UUID, changed_on)` — `sv:idx:<uuid>:<changed_on_iso>`. Holds a list of `CachedEntry` descriptors. Bumping the view's `changed_on` (schema/definition change) invalidates the whole bucket atomically. * **Value**: one Redis key per canonicalized query — `sv:val:<uuid>:<changed_on_iso>:<query_digest>`. Holds the `SemanticResult` (Arrow table + provenance). * **TTL**: takes `SemanticView.cache_timeout` first, then the global `DATA_CACHE_CONFIG.CACHE_DEFAULT_TIMEOUT`. `CachedEntry` is a frozen dataclass that captures filters, dimension keys, metric IDs, limit/offset/order/group-limit, and the value key. It is the unit the lookup walks; the value is fetched lazily once a candidate is chosen. ##### Reuse modes When a query lands, `try_serve_from_cache` walks the bucket and asks `can_satisfy(entry, query)` for each entry. The check returns one of three `ReuseMode`s: | Mode | When | Post-processing | | --------- | --------------------------------------------------- | ------------------------------------------------- | | `EXACT` | Same dims, same metrics | Apply leftover filters / limit / order | | `PROJECT` | Same dims; cached metrics ⊃ new metrics | Drop the unused metric columns | | `ROLLUP` | Cached dims ⊃ new dims | `groupby` re-aggregate the cached `DataFrame` | In all three modes, cached metrics must be a non-strict superset of the new query's metrics. `ROLLUP` additionally requires the new metrics to use additive aggregations (`SUM`, `COUNT`, `MIN`, `MAX`). `PROJECT` has no additive constraint because row granularity is preserved — non-additive aggregates like `AVG` survive unchanged. When multiple entries satisfy the query, we rank them `EXACT < PROJECT < ROLLUP` and serve the cheapest. If the preferred entry's value has been evicted between index lookup and value fetch, we fall back to the next-best candidate. ##### Filter containment For `WHERE` filters, each cached predicate must be *implied by* some new predicate on the same column (cached cannot have hidden rows the new query needs). Conversely, new predicates not covered by the cache become "leftovers" and are reapplied to the cached DataFrame post-fetch, provided they reference columns that are still in the cached projection. The implication helpers cover ranges (`>`, `>=`, `<`, `<=`), `EQUALS`, `IN`/`NOT_IN`, `IS_NULL`/`IS_NOT_NULL`, and exact-match `LIKE`. `HAVING` and `ADHOC` filters require exact-set equality — without parsing them, we cannot prove containment. Filter values are coerced against the dimension's pyarrow type (`_coerce_filter_value`) before containment checks. This fixes a class of false-miss bugs where a UI sends `"1984"` as a string against an integer dimension and the cache fails to recognize that `a >= 1984` is a subset of `a >= 1982`. ##### Top-N safety When a cached entry has a `limit`, we only reuse it if the new query has the same order and a tighter limit (EXACT and PROJECT paths). For `ROLLUP`, we require the cached result to be *known complete* — if `payload.results.num_rows < entry.limit`, the source had fewer rows than the cap, so the rollup is over the whole population. Otherwise the cache is skipped (the index entry is preserved; only that reuse is blocked). ##### Capacity & eviction The per-view bucket caps at `MAX_ENTRIES_PER_VIEW = 128` (oldest entries dropped first). Each value lives under its own Redis key with its own TTL, so Redis-level LRU and TTLs do the actual eviction; the index is kept consistent by pruning entries whose values are gone on every lookup. ##### What is *not* cached * `offset != 0`: pagination beyond the first page is not reused (no efficient way to recover middle pages from a broader cached result). * `is_rowcount` and `force_query=True`: bypass the cache entirely. * Cross-view queries: cache keys are scoped per `(view, changed_on)`. ##### Provenance Every served result records the cache hit in its `requests` list as a `SemanticRequest(type="cache", definition=...)` with a mode-specific note — useful for debugging and for the Explore indicator to show "semantic smart cache" provenance. #### Frontend `CachedLabel` and its tooltip learn a new `cacheSource` prop: `"query"` (legacy) or `"semantic"`. `ChartPills` in Explore passes `"semantic"` when the chart result was served from the semantic-layer cache, distinguishing it from the older per-query result cache. #### Future work * Surface a "smart-cache disabled" toggle in Explore for debugging. * Investigate raising `MAX_ENTRIES_PER_VIEW` per-view via the `SemanticView` schema. * Add metrics/telemetry for hit rate, mode breakdown, and post-processing latency so we can tune containment heuristics with real data. ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF <!--- Skip this if not applicable --> ### TESTING INSTRUCTIONS <!--- Required! What steps can be taken to manually verify the changes? --> * **`tests/unit_tests/semantic_layers/cache_test.py`** — implication helpers, containment edge cases, mode selection, limit/order/offset behavior, and post-processing for each `ReuseMode`. * **`tests/unit_tests/semantic_layers/cache_integration_test.py`** — end-to-end coverage via `mapper.get_results` with an in-memory cache fixture: narrower-filter reuse, projection rollup, metric-subset (PROJECT) hits, the EXACT-vs-ROLLUP preference, and the eviction fallback path. * **`tests/unit_tests/semantic_layers/mapper_test.py`** — filter coercion regressions. ### ADDITIONAL INFORMATION <!--- Check any relevant boxes with "x" --> <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue --> - [ ] Has associated issue: - [ ] Required feature flags: - [ ] Changes UI - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351)) - [ ] Migration is atomic, supports rollback & is backwards-compatible - [ ] Confirm DB migration upgrade and downgrade tested - [ ] Runtime estimates and downtime expectations provided - [ ] Introduces new feature or API - [ ] Removes existing feature or API -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
