DeepFriedYeti opened a new pull request, #40954: URL: https://github.com/apache/superset/pull/40954
<!--- Please write the PR title following the conventions at https://www.conventionalcommits.org/en/v1.0.0/ Example: fix(dashboard): load charts correctly --> ### SUMMARY Adds built-in anomaly detection as a post-processing operation for all ECharts Timeseries chart types (Line, Bar, Area, Scatter, Smooth Line, Step Line). Related: #40520 **SIP:** https://github.com/apache/superset/issues/40520 This feature highlights statistical anomalies directly on timeseries charts as red scatter points, enabling instant visual identification of outliers without exporting data. **Architecture** -> follows the same pattern as the existing Prophet forecast feature: 1. Control panel section with enable toggle and method-specific parameters 2. Frontend operator generates `anomaly_detection` entry in `post_processing` array 3. Backend post-processing function computes anomalies, appends `{column}__anomaly` columns 4. Frontend rendering recognizes `__anomaly` suffix and renders red scatter points **Detection methods:** - **Z-Score** -> Rolling mean/std, flags points where |z| > threshold. General purpose. - **MAD** -> Rolling Median Absolute Deviation. More robust to existing outliers. - **Prophet** -> Fits Prophet model, flags points outside confidence interval. Handles seasonality and trends. **Forecast integration:** When both Prophet forecast and anomaly detection are enabled, anomaly detection automatically runs on the forecast `__yhat` columns (which have complete data), skipping original columns that contain NaN for future periods. ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF <!--- Skip this if not applicable --> <img width="3210" height="1892" alt="image" src="https://github.com/user-attachments/assets/bf29370d-e7d9-4839-a6cc-cebefb2c96ab"; /> <img width="3218" height="1816" alt="image" src="https://github.com/user-attachments/assets/ee6f70dd-a689-407d-b5fb-2026baeef571"; /> ### TESTING INSTRUCTIONS **Backend:** ```bash docker exec superset-superset-1 python -m pytest tests/unit_tests/pandas_postprocessing/test_anomaly.py -v ``` Runs 13 tests covering: zscore, mad, prophet, invalid inputs, forecast column handling, null column skipping. **Frontend:** ```bash npx jest packages/superset-ui-chart-controls/test/operators/anomalyDetectionOperator.test.ts --no-coverage npx jest plugins/plugin-chart-echarts/test/utils/forecast.test.ts --no-coverage ``` Runs 7 operator tests + 17 forecast utility tests (including anomaly indicator). **Manual verification:** 1. Create a Time-series Line chart with any temporal dataset 2. In the Customize tab, expand "Anomaly Detection" section 3. Check "Enable Anomaly Detection" 4. Select method (Z-Score recommended for general use) 5. Run query -> red scatter points should appear at statistical outliers 6. (Optional) Enable Prophet Forecast alongside -> anomaly detection should work on the forecast line ### ADDITIONAL INFORMATION <!--- Check any relevant boxes with "x" --> <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue --> - [x] Has associated issue: https://github.com/apache/superset/issues/40520 - [ ] Required feature flags: - [x] Changes UI - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351)) - [ ] Migration is atomic, supports rollback & is backwards-compatible - [ ] Confirm DB migration upgrade and downgrade tested - [ ] Runtime estimates and downtime expectations provided - [x] Introduces new feature or API - [ ] Removes existing feature or API #### NOTE The program was tested solely for our own use cases, which might differ from yours. Signed-off-by: Ayush Raj [email protected] on behalf of Mercedes-Benz Research and Development India [https://github.com/mercedes-benz/mercedes-benz-foss/blob/master/PROVIDER_INFORMATION.md](https://github.com/Daimler/daimler-foss/blob/master/PROVIDER_INFORMATION.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
