Mark-Onopchenko opened a new issue, #45798: URL: https://github.com/apache/superset/issues/45798
## [SIP] Proposal for opt-in dashboard filters for annotation layers ### Motivation Annotation layers (event markers, intervals or extra series on a time-series chart) ignore dashboard native filters and cross-filters. The host chart is filtered; its annotations are not. Once #45797 lands, only the dashboard time range gets through. Example, on a local dev instance with `cleaned_sales_data`: a dashboard with a Country filter, a "Sales Linear" line chart, and an Event layer sourced from a "Problem orders" table of 7 orders. With Country = USA, the line and the table narrow to USA, but all 7 markers stay. <img width="1223" height="823" alt="Image" src="https://github.com/user-attachments/assets/51c38f2a-0fd9-4842-8201-ae26721510f5"; /> Users have asked for this since 0.34 ([#8647](https://github.com/apache/superset/issues/8647), [#10817](https://github.com/apache/superset/issues/10817), [#13451](https://github.com/apache/superset/issues/13451), [Discussion #22836](https://github.com/apache/superset/discussions/22836), [#36830](https://github.com/apache/superset/issues/36830)). Forcing filters onto every annotation was tried in [PR #8057](https://github.com/apache/superset/pull/8057) and reverted in [PR #10115](https://github.com/apache/superset/pull/10115), because users compare filtered data against unfiltered annotations. So it has to be a per-layer choice. ### Proposed Change Add an **Apply dashboard filters** checkbox to each annotation layer, off by default. When ticked, the dashboard filters that reach the host chart also apply to the layer. It reuses two things Superset already has: the layer's `overrides` dict, gaining a `filters` key where `null` means "inherit from the host" (as for the time overrides), and the `isExtra` tag the frontend puts on every dashboard filter. #### Prerequisite: Phase 0 bug fix (#45796, PR #45797) [#45796](https://github.com/apache/superset/issues/45796): "Override time range" and "Override time grain" did nothing on charts with a generic x-axis, because `get_viz_annotation_data` never received the host query. [PR #45797](https://github.com/apache/superset/pull/45797) passes it in as `host_query` and resolves overrides in a new `_apply_annotation_overrides` helper. Phase 1 builds on both. #### Phase 1: chart-sourced layers (Table and Line sources) - With `"filters": null` in `overrides`, the host's `isExtra` filters, minus the time filter, are copied onto the source chart's query. Without the key, nothing changes. - Native-filter and cross-filter scoping already apply, since the layer inherits only what reached the host. - Filters on columns the source dataset lacks are skipped and listed in `rejected_filter_columns`. - No cache-key change: `overrides` is already part of the annotation cache key. #### Phase 2: Superset annotations - Annotations get an optional `dimensions` field, such as `{"country": ["USA"]}`, editable in the annotation editor. - When the layer opts in, an annotation is kept if, for every `IN`, `NOT IN`, `==` or `!=` filter, it is untagged on that column or one of its values passes. Other operators are skipped and reported. #### UI One checkbox, "Apply dashboard filters", next to "Override time range" in the layer popover. Superset annotation layers show it with their general options. <img width="730" height="496" alt="Image" src="https://github.com/user-attachments/assets/2af3e42d-22c7-4d14-8e08-53e524c483ab"; /> #### Request flow after the change ```mermaid sequenceDiagram participant B as Browser (dashboard) participant P as QueryContextProcessor participant A as get_viz_annotation_data participant S as Source chart B->>P: POST /api/v1/chart/data, dashboard filters tagged isExtra P->>P: run the host query, filtered rect rgba(80, 140, 255, 0.15) P->>P: Phase 2: keep Superset annotations whose dimensions match P->>A: layer, force and the host query (PR 45797) A->>A: apply overrides: time range and grain (PR 45797), filters (Phase 1) end A->>S: ChartDataCommand.run() with saved and dashboard filters S-->>A: 1 record, plus rejected_filter_columns A-->>P: annotation_data P-->>B: line shows USA only, 1 marker ``` Shaded steps are new. On the example dashboard, the layer returns 1 record instead of 7. ### New or Changed Public Interfaces All additive: | Interface | Change | Phase | | --- | --- | --- | | Annotation layer config (`annotation_layers` in form data and `POST /api/v1/chart/data`) | new `overrides.filters` key, value `null` | 1 | | Chart-data response | `rejected_filter_columns` per chart-sourced layer | 1 | | `Annotation` model | nullable `dimensions` column (JSON) | 2 | | Annotation REST API | optional `dimensions` on GET, POST and PUT | 2 | | Annotation export and import | optional `dimensions` key | 2 | | React | checkbox in `AnnotationLayer.tsx`; Dimensions field in `AnnotationModal.tsx` | 1, 2 | No changes to visualization plugins, dashboards, the CLI or deployment. ### New dependencies None. ### Migration Plan and Compatibility - **Phase 0** ([#45797](https://github.com/apache/superset/pull/45797)): ticked "Override time range" boxes start working, so existing chart-sourced layers follow the host's time range. Documented in `UPDATING.md`. - **Phase 1:** no change by default. A layer saved with the option on won't validate on older versions. - **Phase 2:** one nullable column, no backfill. Untagged annotations keep showing everywhere. - No stored URLs change. ### Rejected Alternatives - **Filters on every annotation by default:** tried and reverted ([PR #8057](https://github.com/apache/superset/pull/8057), [PR #10115](https://github.com/apache/superset/pull/10115)). - **Jinja `filter_values` in the source dataset** ([Discussion #22836](https://github.com/apache/superset/discussions/22836)): hand-written SQL per dataset, and no help for Superset annotations. - **Filtering in the browser:** needs every record plus each source's columns; filtering belongs where the query runs. - **A per-layer filter picker:** native-filter scoping already decides which filters apply; a picker can come later. - **Dimensions in `json_metadata`:** free-form user JSON that can't be validated. - **An `annotation_dimension` table:** heavier, and layers are small enough to filter in Python. - **An extension** ([SIP-151](https://github.com/apache/superset/issues/31932), [SIP-177](https://github.com/apache/superset/issues/34162)): extensions can't change how the chart-data request builds annotation data. ### Open Questions 1. Should untagged annotations match every filter (proposed) or none? 2. Should the dashboard show `rejected_filter_columns` to users? 3. SQL-expression filters (`extras.where`) carry no `isExtra` tag, so Phase 1 copies only simple filters. Is that enough? 4. Should Phase 2 also apply the host's time range? 5. Is exact column-name matching enough for dimensions? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
