This is an automated email from the ASF dual-hosted git repository. Gerrrr pushed a commit to branch custom-datasets in repository https://gitbox.apache.org/repos/asf/otava-playground.git
commit cea07c35f36b631705417e8250f93316f537a81a Author: Alex Sorokoumov <[email protected]> AuthorDate: Sat May 30 17:31:40 2026 -0700 Add Dataset mode for comparing algorithms on real series Adds a third top-level mode to the web visualizer ("Dataset", alongside "Single Pattern" and "Mix Patterns") for loading real performance data and seeing where each Otava algorithm variant places change points on the same series. The TigerBeetle dataset bundled in apache/otava's perf/perf_test.py ships as the default preset; a "Custom (paste below)" option accepts JSON-array or whitespace/comma-separated numbers. The Otava analysis panel now offers a checkbox per algorithm variant. In addition to split-edivisive and orig-edivisive, I have also added deterministic-edivisive to facilitate review of https://github.com/apache/otava/pull/154. Backend additions: - otava_test_data.datasets package + TIGERBEETLE preset. - GET /api/datasets — bundled-preset metadata. - GET /api/datasets/{name} — one preset's series + metadata. - GET /api/algorithms — which algorithm functions are available. - POST /api/compare — run multiple algorithms on a series, return all results. - /api/generate and /api/analyze accept otava_algorithm=split|orig|deterministic to pick which variant runs in single/mix mode. Algorithm names are validated as Literal["split","orig","deterministic"], so unknown values fail with 422 at the request boundary instead of returning 200 with an embedded error string. Dataset mode reuses the existing chart container with a single chart that overlays each enabled algorithm's change points as colour-coded vertical lines. The new "Show All Graphs" / "Analyse" buttons and the ground-truth sections (accuracy table, comparison tables, chart legend) are hidden in Dataset mode. Compare requests are guarded by an AbortController so a slow earlier request can't race a newer one. The results table is built with DOM nodes (not string concatenation) so server-supplied strings can't inject HTML. Custom-paste input filters out non-numeric tokens and tells the user how many were dropped. Also fixes an existing latent bug: Starlette 1.x changed TemplateResponse's positional argument order. The pre-existing `/` route was using the old signature, which crashed with "unhashable type: dict" once Jinja2's cache saw the context dict as the template name. Both routes updated. --- README.md | 10 + docs/index.md | 7 + docs/visualizer.md | 69 ++++ pyproject.toml | 1 + src/otava_test_data/datasets/__init__.py | 37 ++ src/otava_test_data/datasets/tigerbeetle.py | 75 ++++ src/otava_test_data/tests/test_compare_api.py | 161 +++++++++ src/otava_test_data/tests/test_datasets.py | 53 +++ src/otava_test_data/web/main.py | 201 +++++++++-- src/otava_test_data/web/static/css/style.css | 172 +++++++++ src/otava_test_data/web/static/js/app.js | 500 +++++++++++++++++++++++--- src/otava_test_data/web/templates/index.html | 90 ++++- 12 files changed, 1287 insertions(+), 89 deletions(-) diff --git a/README.md b/README.md index afea536..5ab2449 100644 --- a/README.md +++ b/README.md @@ -39,6 +39,16 @@ inv web-start Then open http://127.0.0.1:8100 in your browser. +### Compare algorithms on a real dataset + +A third mode, **Dataset**, lets you load a bundled or pasted time series and +see which change points each Otava algorithm variant +(`compute_change_points`, `compute_change_points_orig`, +`compute_change_points_deterministic`) detects on the same data. The TigerBeetle +benchmark dataset ships as the default preset. The Otava analysis panel now +exposes the same algorithm checkboxes in all three modes. See +`docs/visualizer.md` for details. + ## Installation ```bash diff --git a/docs/index.md b/docs/index.md index 55df55c..98f1799 100644 --- a/docs/index.md +++ b/docs/index.md @@ -105,6 +105,13 @@ otava-gen list otava-gen info step_function ``` +## Compare algorithms on a real dataset + +A third mode, **Dataset**, lets you load a bundled or pasted time series and +see which change points each Otava algorithm variant detects on the same data. +The TigerBeetle benchmark dataset ships as the default preset. See the +[Web Visualizer](visualizer.md) doc for details. + ## Contents ```{toctree} diff --git a/docs/visualizer.md b/docs/visualizer.md index 617f668..bae2143 100644 --- a/docs/visualizer.md +++ b/docs/visualizer.md @@ -355,3 +355,72 @@ The visualizer calculates these metrics when comparing Otava's results to ground ## Show All Graphs Click "Show All Graphs" to run analysis on all generators simultaneously. This produces a grid of charts showing how well the analysis methods perform across different types of change point patterns, allowing you to quickly compare detection accuracy across pattern types. + +## Dataset Mode — compare algorithms on a real series + +The mode toggle in the top section has three options: **Single Pattern**, **Mix +Patterns**, and **Dataset**. Dataset mode replaces the synthetic generator grid +with a dataset picker and a "Custom (paste below)" textarea so you can load a +real time series and see where each Otava algorithm variant places change +points on it. + +Ground-truth-only UI (accuracy metrics, comparison tables, chart legend) is +hidden in Dataset mode because real data doesn't come with known change points. + +### Bundled datasets + +| Name | Length | Description | +|------|--------|-------------| +| `tigerbeetle` | 365 | Same series used by `apache/otava`'s `perf/perf_test.py`. Has a couple of distinctive ups and downs, an anomalous drop, then an upward slope, then normal variance. | + +Add more presets by dropping a file in `src/otava_test_data/datasets/` and +registering it in `datasets/__init__.py::DATASETS`. + +### Algorithm checkboxes + +The Otava Analysis panel exposes a checkbox per algorithm variant in all three +modes: + +| Key | Otava function | CLI flag | +|-----|----------------|----------| +| `split` | `compute_change_points` | (default) | +| `orig` | `compute_change_points_orig` | `--orig-edivisive` | +| `deterministic` | `compute_change_points_deterministic` | `--deterministic-edivisive` ([PR](https://github.com/apache/otava/pull/154)) | + +If `compute_change_points_deterministic` isn't importable in the installed +otava version, the checkbox is disabled with a "(not in installed otava)" +indicator next to it. + +### Custom data + +Pick "Custom (paste below)" from the source dropdown and paste a series as +either a JSON array (`[1.2, 3.4, ...]`) or whitespace/comma-separated numbers. +Non-numeric tokens are filtered out and the UI tells you how many were dropped. + +### HTTP endpoints + +- `GET /api/datasets` — bundled-preset metadata. +- `GET /api/datasets/{name}` — one preset's series + metadata. +- `GET /api/algorithms` — change-point algorithms exposed by the installed + otava version. +- `POST /api/compare?window_len=...&max_pvalue=...&min_magnitude=...` — + run selected algorithms on a series. Body: + ```json + {"data": [1.2, 3.4, ...], "algorithms": ["split", "orig"]} + ``` + Response: + ```json + { + "results": { + "split": {"indices": [15, 71, ...], "count": 8}, + "orig": {"indices": [15, 71, ...], "count": 5} + }, + "parameters": {"window_len": 50, "max_pvalue": 0.001, "min_magnitude": 0.0} + } + ``` + If `algorithms` is omitted, every available algorithm runs. + +The synthetic-pattern endpoints `/api/generate/{name}` and +`/api/analyze/{name}` also accept an `otava_algorithm` query parameter +(`split` | `orig` | `deterministic`) so synthetic-pattern flows can pick an +algorithm variant too. diff --git a/pyproject.toml b/pyproject.toml index 537c8d7..80fe338 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -33,6 +33,7 @@ dev = [ "pytest-cov>=4.0.0", "invoke>=2.0.0", "ruff>=0.1.0", + "httpx>=0.27.0", # required by fastapi.testclient ] docs = [ "sphinx>=7.0.0", diff --git a/src/otava_test_data/datasets/__init__.py b/src/otava_test_data/datasets/__init__.py new file mode 100644 index 0000000..51c6009 --- /dev/null +++ b/src/otava_test_data/datasets/__init__.py @@ -0,0 +1,37 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Real-world performance datasets bundled for algorithm comparison.""" + +from otava_test_data.datasets.tigerbeetle import TIGERBEETLE + +DATASETS = { + TIGERBEETLE["name"]: TIGERBEETLE, +} + + +def list_datasets() -> list[dict]: + """Return metadata for every bundled dataset (without the series itself).""" + return [ + {"name": ds["name"], "title": ds["title"], "description": ds["description"]} + for ds in DATASETS.values() + ] + + +def get_dataset(name: str) -> dict | None: + """Return the full dataset entry (series + metadata) or None if unknown.""" + return DATASETS.get(name) diff --git a/src/otava_test_data/datasets/tigerbeetle.py b/src/otava_test_data/datasets/tigerbeetle.py new file mode 100644 index 0000000..e32970f --- /dev/null +++ b/src/otava_test_data/datasets/tigerbeetle.py @@ -0,0 +1,75 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +TigerBeetle benchmark series — same values used by apache/otava's +perf/perf_test.py for algorithm performance benchmarking. + +Source: https://github.com/apache/otava/blob/master/perf/perf_test.py +""" + +TIGERBEETLE_SERIES = [ + 26705, 26475, 26641, 26806, 26835, 26911, 26564, 26812, 26874, 26682, + 15672, 26745, 26460, 26977, 26851, 23412, 23547, 23674, 23519, 23670, + 23662, 23462, 23750, 23717, 23524, 23588, 23687, 23793, 23937, 23715, + 23570, 23730, 23690, 23699, 23670, 23860, 23988, 23652, 23681, 23798, + 23728, 23604, 23523, 23412, 23685, 23773, 23771, 23718, 23409, 23739, + 23674, 23597, 23682, 23680, 23711, 23660, 23990, 23938, 23742, 23703, + 23536, 24363, 24414, 24483, 24509, 24944, 24235, 24560, 24236, 24667, + 24730, 28346, 28437, 28436, 28057, 28217, 28456, 28427, 28398, 28250, + 28331, 28222, 28726, 28578, 28345, 28274, 28514, 28590, 28449, 28305, + 28411, 28788, 28404, 28821, 28580, 27483, 26805, 27487, 27124, 26898, + 27295, 26951, 27312, 27660, 27154, 27050, 26989, 27193, 27503, 27326, + 27375, 27513, 27057, 27421, 27574, 27609, 27123, 27824, 27644, 27394, + 27836, 27949, 27702, 27457, 27272, 28207, 27802, 27516, 27586, 28005, + 27768, 28543, 28237, 27915, 28437, 28342, 27733, 28296, 28524, 28687, + 28258, 28611, 29360, 28590, 29641, 28965, 29474, 29256, 28611, 28205, + 28539, 27962, 28398, 28509, 28240, 28592, 28102, 28461, 28578, 28669, + 28507, 28535, 28226, 28536, 28561, 28087, 27953, 28398, 28007, 28518, + 28337, 28242, 28607, 28545, 28514, 28377, 28010, 28412, 28633, 28576, + 28195, 28637, 28724, 28466, 28287, 28719, 28425, 28860, 28842, 28604, + 28327, 28216, 28946, 28918, 29287, 28725, 29148, 29541, 29137, 29628, + 29087, 28612, 29154, 29108, 28884, 29234, 28695, 28969, 28809, 28695, + 28634, 28916, 29852, 29389, 29757, 29531, 29363, 29251, 29552, 29561, + 29046, 29795, 29022, 29395, 28921, 29739, 29257, 29455, 29376, 29528, + 28909, 29492, 28984, 29621, 29026, 29457, 29102, 29114, 28924, 29162, + 29259, 29554, 29616, 29211, 29367, 29460, 28836, 29645, 29586, 28848, + 29324, 28969, 29150, 29243, 29081, 29312, 28923, 29272, 29117, 29072, + 29529, 29737, 29652, 29612, 29856, 29012, 30402, 29969, 29309, 29439, + 29285, 29421, 29023, 28772, 29692, 29416, 29267, 29542, 29904, 30045, + 29739, 29945, 29141, 29163, 29765, 29197, 29441, 28910, 29504, 29614, + 29643, 29506, 29420, 29672, 29432, 29784, 29888, 29309, 29247, 29816, + 29254, 29813, 29451, 29382, 29618, 28558, 29845, 29499, 29283, 29184, + 29246, 28790, 29952, 29145, 29415, 30437, 29227, 29605, 29859, 29156, + 29807, 29406, 29734, 29861, 29140, 29983, 29832, 29919, 29896, 29991, + 29266, 29001, 29459, 29548, 29310, 29042, 29303, 29894, 29091, 29018, + 29537, 29614, 29180, 29736, 29500, 29218, 29581, 28906, 28542, 29306, + 28987, 29878, 28865, 30272, 29707, 29662, 29815, 30492, 29347, 30096, + 29054, 30238, 28813, 31895, 28915, +] + +TIGERBEETLE = { + "name": "tigerbeetle", + "title": "TigerBeetle", + "description": ( + "TigerBeetle dataset used for demo purposes at Nyrkiö and as the " + "performance benchmark series in apache/otava's perf_test.py. " + "Contains a couple of distinctive ups and downs, an anomalous drop, " + "then an upward slope, with the rest as normal variance." + ), + "series": TIGERBEETLE_SERIES, +} diff --git a/src/otava_test_data/tests/test_compare_api.py b/src/otava_test_data/tests/test_compare_api.py new file mode 100644 index 0000000..f818118 --- /dev/null +++ b/src/otava_test_data/tests/test_compare_api.py @@ -0,0 +1,161 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Tests for the /api/datasets and /api/compare endpoints.""" + +from fastapi.testclient import TestClient + +from otava_test_data.web.main import app + +client = TestClient(app) + + +def test_list_datasets(): + r = client.get("/api/datasets") + assert r.status_code == 200 + body = r.json() + assert "datasets" in body + names = [d["name"] for d in body["datasets"]] + assert "tigerbeetle" in names + + +def test_get_tigerbeetle(): + r = client.get("/api/datasets/tigerbeetle") + assert r.status_code == 200 + body = r.json() + assert body["name"] == "tigerbeetle" + assert len(body["series"]) == 365 + + +def test_get_unknown_dataset_404(): + r = client.get("/api/datasets/nope") + assert r.status_code == 404 + + +def test_list_algorithms_includes_split(): + r = client.get("/api/algorithms") + assert r.status_code == 200 + body = r.json() + assert "split" in body["algorithms"] + # split must be available because apache-otava is a hard dependency. + assert body["algorithms"]["split"]["available"] is True + + +def test_compare_runs_split_on_tigerbeetle(): + series = client.get("/api/datasets/tigerbeetle").json()["series"] + r = client.post( + "/api/compare?max_pvalue=0.001", + json={"data": series, "algorithms": ["split"]}, + ) + assert r.status_code == 200 + body = r.json() + assert "split" in body["results"] + assert "indices" in body["results"]["split"] + # TigerBeetle has clear change points — split should detect at least one. + assert body["results"]["split"]["indices"] + + +def test_compare_rejects_empty_data(): + r = client.post("/api/compare", json={"data": [], "algorithms": ["split"]}) + assert r.status_code == 400 + + +def test_compare_rejects_unknown_algorithm(): + """Unknown algorithm names should fail validation, not return 200 with an inline error.""" + r = client.post( + "/api/compare", + json={"data": [1.0, 2.0, 1.0, 2.0, 1.0, 2.0], "algorithms": ["nonexistent"]}, + ) + assert r.status_code == 422 + + +def test_compare_default_runs_all_available_algorithms(): + """When `algorithms` is omitted, every available algorithm in ALGORITHMS runs.""" + available = { + name for name, info in client.get("/api/algorithms").json()["algorithms"].items() + if info["available"] + } + r = client.post( + "/api/compare", + json={"data": [1.0, 1.1, 1.0, 1.2, 5.0, 5.1, 5.0, 5.2, 5.3, 5.1]}, + ) + assert r.status_code == 200 + assert set(r.json()["results"].keys()) == available + + +def test_compare_accepts_floats_and_round_trips_through_json(): + """Float series shouldn't blow up at JSON serialization.""" + series = [1.5, 2.5, 1.7, 2.3, 1.4, 3.9, 4.1, 4.0, 4.2, 4.3, 4.1] + r = client.post("/api/compare", json={"data": series, "algorithms": ["split"]}) + assert r.status_code == 200 + body = r.json() + # Indices must be JSON-serializable plain ints, not numpy types. + for idx in body["results"]["split"]["indices"]: + assert isinstance(idx, int) + + +def test_index_exposes_dataset_mode(): + """The main page should expose the dataset-mode button and section.""" + r = client.get("/") + assert r.status_code == 200 + assert 'id="mode-dataset-btn"' in r.text + assert 'id="dataset-section"' in r.text + assert 'otava-algo-checkbox' in r.text + + +def test_generate_with_otava_algorithm_orig(): + """The /api/generate endpoint should accept otava_algorithm=orig.""" + r = client.get( + "/api/generate/step_function", + params={ + "length": 100, "seed": 42, + "run_otava": True, "otava_algorithm": "orig", + "max_pvalue": 0.001, + }, + ) + assert r.status_code == 200 + body = r.json() + assert "otava" in body + # No error path expected here. + assert "error" not in body["otava"] + assert "parameters" in body["otava"], body["otava"] + assert body["otava"]["parameters"]["algorithm"] == "orig" + + +def test_generate_rejects_unknown_otava_algorithm(): + r = client.get( + "/api/generate/step_function", + params={"length": 50, "seed": 42, "otava_algorithm": "nonexistent"}, + ) + assert r.status_code == 422 + + +def test_analyze_accepts_otava_algorithm(): + """`/api/analyze` documents the otava_algorithm param — verify it actually works.""" + r = client.get( + "/api/analyze/step_function", + params={ + "length": 100, "seed": 42, + "otava_algorithm": "orig", + "max_pvalue": 0.001, + }, + ) + assert r.status_code == 200 + body = r.json() + assert "otava" in body + assert "error" not in body["otava"] + assert body["otava"]["parameters"]["algorithm"] == "orig" diff --git a/src/otava_test_data/tests/test_datasets.py b/src/otava_test_data/tests/test_datasets.py new file mode 100644 index 0000000..956bbe0 --- /dev/null +++ b/src/otava_test_data/tests/test_datasets.py @@ -0,0 +1,53 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Tests for the bundled real-world datasets.""" + + +from otava_test_data.datasets import DATASETS, get_dataset, list_datasets + + +def test_list_includes_tigerbeetle(): + names = [d["name"] for d in list_datasets()] + assert "tigerbeetle" in names + + +def test_get_unknown_returns_none(): + assert get_dataset("not-a-dataset") is None + + +def test_tigerbeetle_metadata_and_shape(): + ds = get_dataset("tigerbeetle") + assert ds is not None + assert ds["name"] == "tigerbeetle" + assert ds["title"] + assert ds["description"] + assert isinstance(ds["series"], list) + # Same length as the upstream apache/otava perf_test.py series. + assert len(ds["series"]) == 365 + assert all(isinstance(v, (int, float)) for v in ds["series"]) + + +def test_list_omits_series_payload(): + # list_datasets() should not echo the full series — only metadata. + for d in list_datasets(): + assert "series" not in d + + +def test_registry_keyed_by_name(): + for name, ds in DATASETS.items(): + assert ds["name"] == name diff --git a/src/otava_test_data/web/main.py b/src/otava_test_data/web/main.py index 9771d2b..a223437 100644 --- a/src/otava_test_data/web/main.py +++ b/src/otava_test_data/web/main.py @@ -7,7 +7,7 @@ Or: otava-web import json from pathlib import Path -from typing import Any +from typing import Any, Literal import numpy as np from fastapi import FastAPI, Request, Query, Body @@ -16,6 +16,10 @@ from pydantic import BaseModel from fastapi.staticfiles import StaticFiles from fastapi.templating import Jinja2Templates + +# Algorithm name aliases — Literal type used both by query params and request bodies. +AlgorithmName = Literal["split", "orig", "deterministic"] + # Otava imports - optional dependency try: from otava.analysis import compute_change_points @@ -24,6 +28,21 @@ except ImportError: OTAVA_AVAILABLE = False compute_change_points = None +# Optional alternative algorithms — present on newer otava versions / +# https://github.com/apache/otava/pull/154. +# Feature-detected at import time so the /compare UI only offers what works. +try: + from otava.analysis import compute_change_points_orig +except ImportError: + compute_change_points_orig = None + +try: + from otava.analysis import compute_change_points_deterministic +except ImportError: + compute_change_points_deterministic = None + +from otava_test_data.datasets import DATASETS, get_dataset, list_datasets + from otava_test_data.generators.basic import ( constant, noise_normal, @@ -1190,15 +1209,17 @@ def run_otava_analysis( window_len: int = 30, max_pvalue: float = 0.05, min_magnitude: float = 0.0, + algorithm: str = "split", ) -> dict[str, Any]: """ - Run Otava change point detection on data. + Run an Otava change point detection algorithm on data. Args: - data: Time series data as numpy array. - window_len: Minimum window length for detection. + data: Time series data. + window_len: Window length (only meaningful for `split`). max_pvalue: Maximum p-value threshold for significance. min_magnitude: Minimum magnitude of change to report. + algorithm: Which algorithm to run — 'split' (default), 'orig', or 'deterministic'. Returns: Dictionary with detected change points and metrics. @@ -1211,36 +1232,60 @@ def run_otava_analysis( "count": 0, } - try: - result = compute_change_points( - data, - window_len=window_len, - max_pvalue=max_pvalue, - min_magnitude=min_magnitude, - ) + if algorithm == "split": + fn, kwargs = compute_change_points, { + "window_len": window_len, "max_pvalue": max_pvalue, + "min_magnitude": min_magnitude, + } + elif algorithm == "orig": + fn, kwargs = compute_change_points_orig, {"max_pvalue": max_pvalue} + elif algorithm == "deterministic": + fn, kwargs = compute_change_points_deterministic, { + "max_pvalue": max_pvalue, "min_magnitude": min_magnitude, + } + else: + return { + "error": f"unknown algorithm: {algorithm}", + "detected_change_points": [], "detected_indices": [], "count": 0, + } - # Result is a tuple, first element is list of ChangePoint objects + if fn is None: + return { + "error": f"{algorithm} not available in installed otava version", + "detected_change_points": [], "detected_indices": [], "count": 0, + } + + try: + # otava.analysis.compute_change_points crashes with an unmessaged ValueError + # on numpy arrays for some window/data combinations; a plain list of + # Python floats sidesteps it. Match the coercion done in /api/compare. + series = [float(v) for v in data] + result = fn(series, **kwargs) change_points_list = result[0] if isinstance(result, tuple) else result detected = [] for cp in change_points_list: - detected.append({ - "index": int(cp.index), # Convert numpy.int64 to int - "mean_before": float(cp.stats.mean_1), - "mean_after": float(cp.stats.mean_2), - "std_before": float(cp.stats.std_1), - "std_after": float(cp.stats.std_2), - "pvalue": float(cp.stats.pvalue), - }) + entry = {"index": int(cp.index)} + stats = getattr(cp, "stats", None) + if stats is not None: + for key in ("mean_1", "mean_2", "std_1", "std_2", "pvalue"): + val = getattr(stats, key, None) + if val is not None: + out_key = { + "mean_1": "mean_before", "mean_2": "mean_after", + "std_1": "std_before", "std_2": "std_after", + "pvalue": "pvalue", + }[key] + entry[out_key] = float(val) + detected.append(entry) return { "detected_change_points": detected, "detected_indices": [cp["index"] for cp in detected], "count": len(detected), "parameters": { - "window_len": window_len, - "max_pvalue": max_pvalue, - "min_magnitude": min_magnitude, + "window_len": window_len, "max_pvalue": max_pvalue, + "min_magnitude": min_magnitude, "algorithm": algorithm, }, } @@ -1392,6 +1437,7 @@ def timeseries_to_dict( window_len=params.get("window_len", 30), max_pvalue=params.get("max_pvalue", 0.05), min_magnitude=params.get("min_magnitude", 0.0), + algorithm=params.get("algorithm", "split"), ) result["otava"] = otava_result @@ -1413,9 +1459,9 @@ def timeseries_to_dict( async def index(request: Request): """Main page with generator visualization.""" return templates.TemplateResponse( + request, "index.html", { - "request": request, "generators": GENERATORS, "default_length": 200, "version": __version__, @@ -1457,6 +1503,7 @@ async def generate_data( length: int = Query(default=200, ge=10, le=2000), seed: int = Query(default=42), run_otava: bool = Query(default=False, description="Run Otava analysis"), + otava_algorithm: AlgorithmName = Query(default="split", description="Otava algorithm to run"), # noqa: B008 window_len: int = Query(default=99999, ge=5, le=100000, description="Otava window length"), max_pvalue: float = Query(default=0.01, ge=0.0, le=1.0, description="Otava max p-value"), tolerance: int = Query(default=5, ge=0, le=50, description="Accuracy tolerance"), @@ -1496,6 +1543,7 @@ async def generate_data( "window_len": window_len, "max_pvalue": max_pvalue, "tolerance": tolerance, + "algorithm": otava_algorithm, } return timeseries_to_dict(ts, include_otava=run_otava, otava_params=otava_params) except Exception as e: @@ -1510,6 +1558,7 @@ async def analyze_with_otava( generator_name: str, length: int = Query(default=200, ge=10, le=2000), seed: int = Query(default=42), + otava_algorithm: AlgorithmName = Query(default="split", description="Otava algorithm to run"), # noqa: B008 window_len: int = Query(default=30, ge=5, le=100, description="Otava window length"), max_pvalue: float = Query(default=0.00001, ge=0.0, le=1.0, description="Otava max p-value"), min_magnitude: float = Query(default=0.0, ge=0, description="Minimum change magnitude"), @@ -1550,6 +1599,7 @@ async def analyze_with_otava( "max_pvalue": max_pvalue, "min_magnitude": min_magnitude, "tolerance": tolerance, + "algorithm": otava_algorithm, } return timeseries_to_dict(ts, include_otava=True, otava_params=otava_params) except Exception as e: @@ -1643,6 +1693,109 @@ async def detect_change_points( ) +# ----- Dataset comparison: presets and multi-algorithm detection ----- + +ALGORITHMS = { + "split": { + "title": "split-edivisive (default)", + "description": ( + "Hunter-style split-merge e-divisive + Welch t-test significance. " + "Otava default (compute_change_points)." + ), + "available": compute_change_points is not None, + }, + "orig": { + "title": "orig-edivisive", + "description": ( + "Original e-divisive with permutation significance test " + "(compute_change_points_orig, --orig-edivisive)." + ), + "available": compute_change_points_orig is not None, + }, + "deterministic": { + "title": "deterministic-edivisive", + "description": ( + "Original e-divisive with deterministic Welch t-test significance " + "(compute_change_points_deterministic, --deterministic-edivisive, " + "https://github.com/apache/otava/pull/154)." + ), + "available": compute_change_points_deterministic is not None, + }, +} + + +def _run_algorithm(name: str, data, window_len: int, max_pvalue: float, + min_magnitude: float) -> dict[str, Any]: + """Adapter from /api/compare's response shape to run_otava_analysis.""" + res = run_otava_analysis( + data, + window_len=window_len, + max_pvalue=max_pvalue, + min_magnitude=min_magnitude, + algorithm=name, + ) + out = {"indices": res.get("detected_indices", []), "count": res.get("count", 0)} + if res.get("error"): + out["error"] = res["error"] + return out + + +class CompareRequest(BaseModel): + """Request body for /api/compare.""" + data: list[float] + algorithms: list[AlgorithmName] | None = None # default: all available + + [email protected]("/api/datasets") +async def get_datasets(): + """List bundled real-world datasets.""" + return {"datasets": list_datasets()} + + [email protected]("/api/datasets/{name}") +async def get_dataset_endpoint(name: str): + """Return one bundled dataset's series + metadata.""" + ds = get_dataset(name) + if ds is None: + return JSONResponse(status_code=404, content={"error": f"unknown dataset: {name}"}) + return ds + + [email protected]("/api/algorithms") +async def list_algorithms(): + """List change-point algorithms exposed by the installed otava package.""" + return {"algorithms": ALGORITHMS} + + [email protected]("/api/compare") +async def compare_algorithms( + request: CompareRequest, + window_len: int = Query(default=50, ge=5, le=100000), + max_pvalue: float = Query(default=0.001, ge=0.0, le=1.0), + min_magnitude: float = Query(default=0.0, ge=0), +): + """Run multiple change-point algorithms on the same series and return all results.""" + if not request.data: + return JSONResponse(status_code=400, content={"error": "No data provided"}) + if not OTAVA_AVAILABLE: + return JSONResponse(status_code=503, content={"error": "apache-otava not installed"}) + + algorithms = request.algorithms or [n for n, a in ALGORITHMS.items() if a["available"]] + # run_otava_analysis handles float coercion internally. + results = { + name: _run_algorithm(name, request.data, window_len, max_pvalue, min_magnitude) + for name in algorithms + } + return { + "results": results, + "parameters": { + "window_len": window_len, + "max_pvalue": max_pvalue, + "min_magnitude": min_magnitude, + }, + } + + def run(): """Run the web server.""" import uvicorn diff --git a/src/otava_test_data/web/static/css/style.css b/src/otava_test_data/web/static/css/style.css index 01cf3dd..dc9c65b 100644 --- a/src/otava_test_data/web/static/css/style.css +++ b/src/otava_test_data/web/static/css/style.css @@ -1843,3 +1843,175 @@ footer a:hover { grid-template-columns: 1fr; } } + +/* ============================================================ + Dataset mode — load a bundled or pasted series and compare + Otava algorithm variants on the same data. + ============================================================ */ + +.dataset-section { + background: #fafafa; + border: 1px solid #e2e8f0; + border-radius: 8px; + padding: 16px 20px; + margin: 0 0 20px; +} + +.dataset-section.hidden { + display: none; +} + +/* The mode toggle bar sits above the per-mode content. */ +.mode-toggle-section { + margin-bottom: 8px; +} + +.dataset-row { + display: flex; + flex-direction: column; + gap: 8px; + margin-bottom: 16px; +} + +.dataset-label { + display: flex; + flex-direction: column; + gap: 4px; + font-size: 14px; + font-weight: 600; + max-width: 480px; +} + +.dataset-label select { + padding: 6px 8px; + font: inherit; + border: 1px solid #cbd5e1; + border-radius: 4px; + background: #fff; +} + +.dataset-description { + font-size: 13px; + color: #64748b; + margin: 0; + min-height: 1.4em; +} + +.dataset-custom { + display: flex; + flex-direction: column; + gap: 4px; + font-size: 14px; + font-weight: 600; +} + +.dataset-custom.hidden { + display: none; +} + +.dataset-custom textarea { + font-family: ui-monospace, "SF Mono", Monaco, monospace; + font-size: 13px; + padding: 8px; + border: 1px solid #cbd5e1; + border-radius: 4px; + resize: vertical; +} + +.dataset-hint { + margin: 8px 0 0; + font-size: 13px; + color: #475569; +} + +.dataset-status { + margin: 8px 0 0; + padding: 6px 10px; + background: #eef2ff; + border: 1px solid #c7d2fe; + border-radius: 4px; + font-size: 13px; + color: #1e1b4b; +} + +.dataset-status--error { + background: #fef2f2; + border-color: #fecaca; + color: #7f1d1d; +} + +.otava-algo-row .algo-unavail { + margin-left: 4px; + color: #94a3b8; + font-style: italic; +} + +.dataset-results.hidden { + display: none; +} + +.dataset-results h3 { + margin: 16px 0 8px; +} + +.dataset-results-table { + width: 100%; + border-collapse: collapse; + font-size: 14px; +} + +.dataset-results-table th, +.dataset-results-table td { + border: 1px solid #e2e8f0; + padding: 6px 10px; + text-align: left; + vertical-align: top; +} + +.dataset-results-table th { + background: #f1f5f9; +} + +.dataset-results-table .swatch { + display: inline-block; + width: 12px; + height: 12px; + border-radius: 2px; + margin-right: 8px; + vertical-align: middle; +} + +/* In dataset mode, hide synthetic-data and ground-truth UI. */ +body[data-mode="dataset"] .hide-in-dataset { + display: none !important; +} + +/* Otava algorithm checkbox group (used in the Otava analysis panel). */ +.otava-algos { + display: flex; + flex-direction: column; + gap: 4px; + margin: 4px 0 12px; + font-size: 13px; +} + +.otava-algo-row { + display: flex; + align-items: center; + gap: 6px; + white-space: nowrap; + line-height: 1.4; +} + +.otava-algo-row input[type="checkbox"] { + flex-shrink: 0; +} + +.otava-algo-row input[disabled] { + cursor: not-allowed; +} + +.algo-hint { + color: #64748b; + font-size: 11px; +} diff --git a/src/otava_test_data/web/static/js/app.js b/src/otava_test_data/web/static/js/app.js index 8054e4f..134b241 100644 --- a/src/otava_test_data/web/static/js/app.js +++ b/src/otava_test_data/web/static/js/app.js @@ -12,14 +12,33 @@ let generatorTileCharts = {}; // Mini charts for generator tiles let analysisMethods = {}; // Tutorial content for analysis methods let tutorialVisible = false; // Track tutorial panel visibility -// Mix Mode State -let mixMode = false; // Single Pattern vs Mix Patterns mode +// Mode State - 'single' | 'mix' | 'dataset' +let currentMode = 'single'; let mixOperation = 'sum'; // 'sum' or 'append' let mixComponents = []; // [{name, data, changePoints, params, count}] let mixedData = null; // Combined data array let mixedChangePoints = []; // Merged ground truth change points let tileBadges = {}; // Track count badges on tiles +// Dataset Mode State +let bundledDatasets = []; // [{name, title, description}, ...] +let availableAlgorithms = {}; // {split: {title, available}, ...} +let datasetSeriesCache = {}; // name -> {series, timestamps?} + +// Helper that reads better at call sites than `currentMode === 'mix'`. +const isMixMode = () => currentMode === 'mix'; + +// Algorithm colors used when overlaying multiple algorithms on one chart. +const ALGO_COLORS = { + split: '#1f77b4', + orig: '#d62728', + deterministic: '#2ca02c', +}; + +// In-flight AbortController for /api/compare so out-of-order responses can't +// land after a newer one and desync the chart from the controls. +let datasetInflight = null; + // DOM Elements - Data Generation const generatorGrid = document.getElementById('generator-grid'); const lengthSlider = document.getElementById('length-slider'); @@ -29,15 +48,26 @@ const lengthMax = document.getElementById('length-max'); const seedInput = document.getElementById('seed-input'); const dynamicParams = document.getElementById('dynamic-params'); -// DOM Elements - Mix Mode +// DOM Elements - Mode toggle const modeSingleBtn = document.getElementById('mode-single-btn'); const modeMixBtn = document.getElementById('mode-mix-btn'); +const modeDatasetBtn = document.getElementById('mode-dataset-btn'); const mixInfo = document.getElementById('mix-info'); const mixRecipe = document.getElementById('mix-recipe'); const clearMixBtn = document.getElementById('clear-mix-btn'); +// DOM Elements - Dataset mode +const datasetSection = document.getElementById('dataset-section'); +const datasetSourceSelect = document.getElementById('dataset-source'); +const datasetDescription = document.getElementById('dataset-description'); +const customInputWrapper = document.getElementById('custom-input-wrapper'); +const customInput = document.getElementById('custom-input'); +const datasetStatusEl = document.getElementById('dataset-status'); +const datasetResultsSection = document.getElementById('dataset-results'); +const datasetResultsBody = document.getElementById('dataset-results-body'); + // DOM Elements - Otava Controls -const runOtavaCheckbox = document.getElementById('run-otava-checkbox'); +const otavaAlgoCheckboxes = document.querySelectorAll('.otava-algo-checkbox'); const windowLenInput = document.getElementById('window-len-input'); const maxPvalueInput = document.getElementById('max-pvalue-input'); const yMinInput = document.getElementById('y-min-input'); @@ -79,6 +109,25 @@ const stdDevNumInput = document.getElementById('stddev-num-input'); const DEFAULT_TOLERANCE = 0; // Exact match for True Positive const CLOSE_MATCH_TOLERANCE = 5; // Within 5 points for Close Match +/** Return the set of Otava algorithms the user has enabled. */ +function getEnabledOtavaAlgorithms() { + return Array.from(otavaAlgoCheckboxes) + .filter(el => el.checked && !el.disabled) + .map(el => el.dataset.algo); +} + +/** True if at least one Otava algorithm is enabled. */ +function isOtavaEnabled() { + return getEnabledOtavaAlgorithms().length > 0; +} + +/** Primary algorithm — the first checked one (drives the chart annotations + * + accuracy metrics in single/mix mode). Defaults to 'split'. */ +function primaryOtavaAlgorithm() { + const enabled = getEnabledOtavaAlgorithms(); + return enabled[0] || 'split'; +} + // DOM Elements - Actions const generateBtn = document.getElementById('generate-btn'); const showAllBtn = document.getElementById('show-all-btn'); @@ -128,8 +177,16 @@ const metricsTutorialPanel = document.getElementById('metrics-tutorial-panel'); // Initialize document.addEventListener('DOMContentLoaded', async () => { - await Promise.all([loadGenerators(), loadAnalysisMethods()]); + document.body.setAttribute('data-mode', 'single'); + await Promise.all([ + loadGenerators(), + loadAnalysisMethods(), + loadDatasets(), + loadAvailableAlgorithms(), + ]); await populateGeneratorGrid(); + populateDatasetUI(); + refreshDatasetSourceDescription(); setupEventListeners(); setupTutorialHandlers(); updateGeneratorInfo(); @@ -217,7 +274,7 @@ async function populateGeneratorGrid() { const generatorNames = Object.keys(generators); - if (mixMode) { + if (isMixMode()) { // Mix mode layout: Single row with clean patterns + noise + operation toggle const mixRow1Order = [ 'constant', @@ -344,15 +401,15 @@ async function populateGeneratorGrid() { } // Apply mix mode class to grid - generatorGrid.classList.toggle('mix-mode', mixMode); + generatorGrid.classList.toggle('mix-mode', isMixMode()); } /** * Create a generator tile element */ -function createGeneratorTile(name, info, preview, isMixMode) { +function createGeneratorTile(name, info, preview, forMixGrid) { const tile = document.createElement('div'); - tile.className = 'generator-tile' + (!isMixMode && name === selectedGenerator ? ' selected' : ''); + tile.className = 'generator-tile' + (!forMixGrid && name === selectedGenerator ? ' selected' : ''); tile.dataset.generator = name; // Preview container @@ -371,7 +428,7 @@ function createGeneratorTile(name, info, preview, isMixMode) { tile.appendChild(label); // Click handler - different for mix mode vs single mode - if (isMixMode) { + if (forMixGrid) { tile.addEventListener('click', () => addToMix(name)); } else { tile.addEventListener('click', () => selectGenerator(name)); @@ -379,7 +436,7 @@ function createGeneratorTile(name, info, preview, isMixMode) { // Create mini chart if (preview && preview.data) { - createTileChart(canvas, preview.data, !isMixMode && name === selectedGenerator); + createTileChart(canvas, preview.data, !forMixGrid && name === selectedGenerator); } return tile; @@ -695,13 +752,23 @@ function setupEventListeners() { generateBtn.addEventListener('click', generateData); showAllBtn.addEventListener('click', showAllPatterns); - // Mix mode controls - modeSingleBtn.addEventListener('click', () => toggleMixMode(false)); - modeMixBtn.addEventListener('click', () => toggleMixMode(true)); + // Mode toggle + modeSingleBtn.addEventListener('click', () => setMode('single')); + modeMixBtn.addEventListener('click', () => setMode('mix')); + modeDatasetBtn.addEventListener('click', () => setMode('dataset')); clearMixBtn.addEventListener('click', clearMix); + // Dataset mode controls + datasetSourceSelect.addEventListener('change', () => { + refreshDatasetSourceDescription(); + if (currentMode === 'dataset') runDatasetAnalysis(); + }); + customInput.addEventListener('change', () => { + if (currentMode === 'dataset') runDatasetAnalysis(); + }); + // Otava controls - runOtavaCheckbox.addEventListener('change', refreshDisplay); + otavaAlgoCheckboxes.forEach(cb => cb.addEventListener('change', refreshDisplay)); windowLenInput.addEventListener('change', refreshDisplay); maxPvalueInput.addEventListener('change', refreshDisplay); @@ -740,7 +807,7 @@ function setupEventListeners() { // Update generator info display function updateGeneratorInfo() { - if (mixMode && mixComponents.length > 0) { + if (isMixMode() && mixComponents.length > 0) { // Mix mode with components const totalCPs = mixedChangePoints ? mixedChangePoints.filter(cp => cp.type !== 'outlier').length : 0; generatorTitle.textContent = 'Mixed Pattern'; @@ -783,7 +850,7 @@ function updateGeneratorInfo() { // Update dynamic parameter inputs function updateDynamicParams() { - if(mixMode) updateMixParams(); + if(isMixMode()) updateMixParams(); else updateSingleParams(); } @@ -822,7 +889,7 @@ function renderParamWidgets(info, containerDiv, bindComponent){ function dynamicParamChanged(ev) { // console.log(ev); - if(mixMode && bindComponent){ + if(isMixMode() && bindComponent){ bindComponent.params[paramName] = ev.target.value; asyncRedraw(bindComponent); } @@ -1164,13 +1231,14 @@ async function generateData() { const name = selectedGenerator; const length = lengthInput.value; const seed = seedInput.value; - const runOtava = runOtavaCheckbox.checked; + const runOtava = isOtavaEnabled(); // Build query params const params = new URLSearchParams({ length, seed, run_otava: runOtava, + otava_algorithm: primaryOtavaAlgorithm(), window_len: windowLenInput.value, max_pvalue: maxPvalueInput.value, tolerance: DEFAULT_TOLERANCE, @@ -1409,7 +1477,7 @@ function updateChart(data) { // Track which is the last chart for showing X-axis const enabledMethods = []; - if (runOtavaCheckbox.checked) enabledMethods.push('otava'); + if (isOtavaEnabled()) enabledMethods.push('otava'); if (runMa) enabledMethods.push('ma'); if (runBoundary) enabledMethods.push('boundary'); if (runThreshold) enabledMethods.push('threshold'); @@ -1417,7 +1485,7 @@ function updateChart(data) { if (runStdDev) enabledMethods.push('stdDev'); // Create Otava chart if enabled - if (runOtavaCheckbox.checked) { + if (isOtavaEnabled()) { const { tp: otavaTp, cm: otavaCm, fp: otavaFp, exactMatches: otavaExact, closeMatches: otavaClose } = otavaClassification; const canvas = createChartContainer('otava', 'Otava Analysis', '#2563eb', otavaTp, otavaCm, otavaFp); const ctx = canvas.getContext('2d'); @@ -1812,7 +1880,7 @@ function updateChart(data) { // Store all results for accuracy metrics display data._methodResults = { - otava: runOtavaCheckbox.checked ? { + otava: isOtavaEnabled() ? { name: 'Otava', classification: otavaClassification, detectedIndices: detectedIndices @@ -2220,7 +2288,9 @@ async function showAllPatterns() { * Refresh the current display - calls the appropriate update function based on mode */ function refreshDisplay() { - if (mixMode && mixComponents.length > 0) { + if (currentMode === 'dataset') { + runDatasetAnalysis(); + } else if (currentMode === 'mix' && mixComponents.length > 0) { computeAndDisplayMixedData(); } else { generateData(); @@ -2228,36 +2298,372 @@ function refreshDisplay() { } /** - * Toggle between Single Pattern and Mix Patterns mode + * Switch between the three top-level modes: 'single' | 'mix' | 'dataset'. */ -function toggleMixMode(enable) { - mixMode = enable; +function setMode(mode) { + if (!['single', 'mix', 'dataset'].includes(mode)) return; + currentMode = mode; + + // Body data-attr drives CSS that hides ground-truth sections in dataset mode. + document.body.setAttribute('data-mode', mode); + + // Update button active state + modeSingleBtn.classList.toggle('active', mode === 'single'); + modeMixBtn.classList.toggle('active', mode === 'mix'); + modeDatasetBtn.classList.toggle('active', mode === 'dataset'); + + // Show/hide the right top section + const generatorSection = document.querySelector('.generator-grid-section'); + if (generatorSection) generatorSection.classList.toggle('hidden', mode === 'dataset'); + datasetSection.classList.toggle('hidden', mode !== 'dataset'); + + // Update the section heading next to the mode toggle. + const title = document.getElementById('mode-section-title'); + if (title) { + title.textContent = + mode === 'dataset' ? 'Dataset' : + mode === 'mix' ? 'Mix Patterns' : + 'Select Pattern'; + } - // Update button states - modeSingleBtn.classList.toggle('active', !enable); - modeMixBtn.classList.toggle('active', enable); + // Mix-info banner visible only in mix mode + mixInfo.classList.toggle('hidden', mode !== 'mix'); + generatorGrid.classList.toggle('mix-mode', mode === 'mix'); - // Toggle mix info visibility - mixInfo.classList.toggle('hidden', !enable); + if (mode === 'mix') { + updateMixDisplay(); + populateGeneratorGrid(); + } else if (mode === 'single') { + clearMix(); + populateGeneratorGrid(); + if (selectedGenerator) generateData(); + } else { + // Dataset mode: hide multi-chart and clear mix state silently. + multiChartContainer.classList.add('hidden'); + runDatasetAnalysis(); + } +} - // Toggle grid class - generatorGrid.classList.toggle('mix-mode', enable); +/* ==================================================================== + Dataset Mode + ==================================================================== */ - // Clear mix state when switching modes- or maybe not... - if (enable) { - //clearMix(); - updateMixDisplay(); +async function loadDatasets() { + try { + const r = await fetch('/api/datasets'); + const body = await r.json(); + bundledDatasets = body.datasets || []; + } catch (e) { + console.error('Failed to load datasets:', e); + bundledDatasets = []; + } +} + +async function loadAvailableAlgorithms() { + try { + const r = await fetch('/api/algorithms'); + const body = await r.json(); + availableAlgorithms = body.algorithms || {}; + } catch (e) { + console.error('Failed to load algorithms:', e); + availableAlgorithms = {}; + } +} + +function populateDatasetUI() { + // Dataset source dropdown. + datasetSourceSelect.innerHTML = ''; + bundledDatasets.forEach(ds => { + const opt = document.createElement('option'); + opt.value = ds.name; + opt.textContent = ds.title; + datasetSourceSelect.appendChild(opt); + }); + const customOpt = document.createElement('option'); + customOpt.value = '__custom__'; + customOpt.textContent = 'Custom (paste below)'; + datasetSourceSelect.appendChild(customOpt); + + // Disable Otava-panel checkboxes for algorithms the installed otava doesn't expose. + for (const [name, info] of Object.entries(availableAlgorithms)) { + const cb = document.getElementById(`otava-algo-${name}`); + if (!cb) continue; + if (!info.available) { + cb.checked = false; + cb.disabled = true; + // Append a sibling indicator instead of overwriting the existing + // .algo-hint, which may contain an inline link (e.g. PR #154). + if (!cb.parentElement.querySelector('.algo-unavail')) { + const span = document.createElement('span'); + span.className = 'algo-unavail'; + span.textContent = '(not in installed otava)'; + cb.parentElement.appendChild(span); + } + } + } +} + +async function loadCurrentDatasetSeries() { + const choice = datasetSourceSelect.value; + if (choice === '__custom__') { + const { series, dropped } = parseCustomSeries(customInput.value); + return { name: 'custom', title: 'Custom', series, dropped }; + } + if (!datasetSeriesCache[choice]) { + const r = await fetch(`/api/datasets/${encodeURIComponent(choice)}`); + if (!r.ok) throw new Error(`Failed to load ${choice}: ${r.status}`); + datasetSeriesCache[choice] = await r.json(); + } + return datasetSeriesCache[choice]; +} + +function parseCustomSeries(text) { + text = (text || '').trim(); + if (!text) return { series: [], dropped: 0 }; + let raw; + if (text.startsWith('[')) { + try { raw = JSON.parse(text).map(Number); } + catch (e) { raw = text.split(/[^0-9eE.\-+]+/).filter(Boolean).map(Number); } } else { - // Clear badges and restore normal tile behavior - clearMix(); - // Re-render the selected generator in single mode - if (selectedGenerator) { - generateData(); + raw = text.split(/[^0-9eE.\-+]+/).filter(Boolean).map(Number); + } + const series = raw.filter(v => Number.isFinite(v)); + return { series, dropped: raw.length - series.length }; +} + +function refreshDatasetSourceDescription() { + const choice = datasetSourceSelect.value; + if (choice === '__custom__') { + customInputWrapper.classList.remove('hidden'); + datasetDescription.textContent = 'Paste your own numeric series.'; + return; + } + customInputWrapper.classList.add('hidden'); + const ds = bundledDatasets.find(d => d.name === choice); + datasetDescription.textContent = ds ? ds.description : ''; +} + +function setDatasetStatus(msg, isError = false) { + if (!msg) { + datasetStatusEl.hidden = true; + datasetStatusEl.textContent = ''; + return; + } + datasetStatusEl.hidden = false; + datasetStatusEl.textContent = msg; + datasetStatusEl.classList.toggle('dataset-status--error', !!isError); +} + +function emptyResultsRow(message) { + datasetResultsBody.replaceChildren(); + const tr = document.createElement('tr'); + const td = document.createElement('td'); + td.colSpan = 3; + td.style.color = '#999'; + td.textContent = message; + tr.appendChild(td); + datasetResultsBody.appendChild(tr); + datasetResultsSection.classList.remove('hidden'); +} + +async function runDatasetAnalysis() { + // Claim the in-flight slot up front so every early-return path leaves + // datasetInflight in a clean state and the next call's abort() is meaningful. + if (datasetInflight) datasetInflight.abort(); + const ctrl = new AbortController(); + datasetInflight = ctrl; + const stillCurrent = () => datasetInflight === ctrl; + + let series; + let droppedTokens = 0; + try { + const ds = await loadCurrentDatasetSeries(); + series = ds.series; + droppedTokens = ds.dropped || 0; + } catch (e) { + if (stillCurrent()) { + datasetInflight = null; + setDatasetStatus(`Failed to load series: ${e.message || e}`, true); } + return; } + if (!stillCurrent()) return; + + const tooFewPoints = !series || series.length < 5; + const noAlgos = !tooFewPoints && getEnabledOtavaAlgorithms().length === 0; - // Rebuild grid for mix mode layout - populateGeneratorGrid(); + // Compose the status: token-warning + (optional) terminal hint, so an + // input problem isn't masked by the "need 5 points" message and vice versa. + const parts = []; + if (droppedTokens > 0) parts.push(`Ignored ${droppedTokens} non-numeric token(s).`); + if (tooFewPoints) parts.push('Need at least 5 numeric points.'); + setDatasetStatus(parts.join(' '), parts.length > 0); + + if (tooFewPoints) { + emptyResultsRow('Need at least 5 numeric points.'); + datasetInflight = null; + return; + } + if (noAlgos) { + renderDatasetChart(series, {}); + emptyResultsRow('Pick at least one algorithm in the Otava Analysis panel.'); + datasetInflight = null; + return; + } + + const algos = getEnabledOtavaAlgorithms(); + const params = new URLSearchParams({ + window_len: windowLenInput.value, + max_pvalue: maxPvalueInput.value, + min_magnitude: '0', + }); + + try { + document.body.classList.add('loading'); + const r = await fetch(`/api/compare?${params}`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ data: series, algorithms: algos }), + signal: ctrl.signal, + }); + if (!r.ok) { + const err = await r.json().catch(() => ({})); + throw new Error(err.error || `HTTP ${r.status}`); + } + const body = await r.json(); + if (!stillCurrent()) return; // superseded; let the newer call render + renderDatasetChart(series, body.results); + renderDatasetResultsTable(body.results); + updateDatasetStats(series, body.results); + datasetResultsSection.classList.remove('hidden'); + } catch (e) { + if (e.name === 'AbortError') return; // expected when superseded + if (stillCurrent()) setDatasetStatus(`Compare failed: ${e.message || e}`, true); + } finally { + if (stillCurrent()) { + datasetInflight = null; + document.body.classList.remove('loading'); + } + } +} + +function renderDatasetChart(series, resultsByAlgo) { + stackedCharts.forEach(c => c.destroy()); + stackedCharts = []; + stackedChartsContainer.innerHTML = ''; + + const container = document.createElement('div'); + container.className = 'stacked-chart'; + const canvas = document.createElement('canvas'); + canvas.id = 'canvas-dataset'; + container.appendChild(canvas); + stackedChartsContainer.appendChild(container); + + const labels = series.map((_, i) => i); + const annotations = {}; + for (const [algo, info] of Object.entries(resultsByAlgo || {})) { + const color = ALGO_COLORS[algo] || '#888'; + for (const idx of (info.indices || [])) { + annotations[`${algo}-${idx}`] = { + type: 'line', + xMin: idx, xMax: idx, + borderColor: color, + borderWidth: 2, + borderDash: [4, 4], + }; + } + } + + const chart = new Chart(canvas.getContext('2d'), { + type: 'line', + data: { + labels, + datasets: [{ + label: 'Series', + data: series, + borderColor: '#444', + backgroundColor: 'rgba(0,0,0,0)', + pointRadius: 1.5, + borderWidth: 1, + tension: 0, + }], + }, + options: { + responsive: true, + maintainAspectRatio: false, + scales: { + x: { title: { display: true, text: 'Index' } }, + y: { title: { display: true, text: 'Value' } }, + }, + plugins: { + legend: { position: 'top' }, + annotation: { annotations }, + }, + }, + }); + stackedCharts.push(chart); +} + +function renderDatasetResultsTable(resultsByAlgo) { + // Build rows as DOM nodes so server-supplied strings (algorithm names, + // error messages) can't break out and inject HTML. + datasetResultsBody.replaceChildren(); + const entries = Object.entries(resultsByAlgo); + if (entries.length === 0) { + const tr = document.createElement('tr'); + const td = document.createElement('td'); + td.colSpan = 3; + td.style.color = '#999'; + td.textContent = 'No algorithms selected.'; + tr.appendChild(td); + datasetResultsBody.appendChild(tr); + return; + } + for (const [algo, info] of entries) { + const color = ALGO_COLORS[algo] || '#888'; + const indices = info.indices || []; + const tr = document.createElement('tr'); + + const tdName = document.createElement('td'); + const swatch = document.createElement('span'); + swatch.className = 'swatch'; + swatch.style.background = color; + tdName.appendChild(swatch); + tdName.appendChild(document.createTextNode(algo)); + if (info.error) { + const span = document.createElement('span'); + span.style.color = '#900'; + span.textContent = ` (${info.error})`; + tdName.appendChild(span); + } + tr.appendChild(tdName); + + const tdCount = document.createElement('td'); + tdCount.textContent = String(indices.length); + tr.appendChild(tdCount); + + const tdIdx = document.createElement('td'); + const code = document.createElement('code'); + code.textContent = indices.length ? indices.join(', ') : '—'; + tdIdx.appendChild(code); + tr.appendChild(tdIdx); + + datasetResultsBody.appendChild(tr); + } +} + +function updateDatasetStats(series, resultsByAlgo) { + const n = series.length; + const mean = series.reduce((a, b) => a + b, 0) / n; + const variance = series.reduce((a, v) => a + (v - mean) ** 2, 0) / n; + const std = Math.sqrt(variance); + statLength.textContent = n.toString(); + statMean.textContent = mean.toFixed(2); + statStd.textContent = std.toFixed(2); + statCpTruth.textContent = '—'; + const totalDetected = Object.values(resultsByAlgo || {}) + .reduce((a, info) => a + (info.indices?.length || 0), 0); + statCpDetected.textContent = totalDetected.toString(); } /** @@ -2468,7 +2874,7 @@ function clearMix() { updateMixDisplay(); updateTileBadges(); - if (mixMode) { + if (isMixMode()) { // Clear charts stackedChartsContainer.innerHTML = ` <div class="stacked-chart" style="text-align: center; padding: 2rem;"> @@ -2658,7 +3064,7 @@ async function computeAndDisplayMixedData() { }; // Run Otava analysis on mixed data if enabled - if (runOtavaCheckbox.checked) { + if (isOtavaEnabled()) { try { const params = new URLSearchParams({ window_len: windowLenInput.value, @@ -2773,7 +3179,7 @@ function updateTileBadges() { tile.classList.remove('in-mix'); }); - if (!mixMode) return; + if (!isMixMode()) return; // Add badges for components in mix const sumClicks = {}; diff --git a/src/otava_test_data/web/templates/index.html b/src/otava_test_data/web/templates/index.html index 7649ad0..e0c9057 100644 --- a/src/otava_test_data/web/templates/index.html +++ b/src/otava_test_data/web/templates/index.html @@ -6,7 +6,7 @@ <title>Otava Test Data Visualizer</title> <script src="https://cdn.jsdelivr.net/npm/chart.js";></script> <script src="https://cdn.jsdelivr.net/npm/chartjs-plugin-annotation";></script> - <link rel="stylesheet" href="/static/css/style.css"> + <link rel="stylesheet" href="/static/css/style.css?v={{ version }}"> </head> <body> <header> @@ -53,15 +53,20 @@ </div> <main> - <!-- Generator Selection Grid --> - <section class="generator-grid-section"> + <!-- Mode toggle stays visible in every mode --> + <section class="mode-toggle-section"> <div class="section-header"> - <h3 class="section-title">Select Pattern</h3> + <h3 class="section-title" id="mode-section-title">Select Pattern</h3> <div class="mode-toggle"> <button class="mode-btn active" data-mode="single" id="mode-single-btn">Single Pattern</button> <button class="mode-btn" data-mode="mix" id="mode-mix-btn">Mix Patterns</button> + <button class="mode-btn" data-mode="dataset" id="mode-dataset-btn">Dataset</button> </div> </div> + </section> + + <!-- Generator Selection Grid --> + <section class="generator-grid-section"> <div class="mix-info hidden" id="mix-info"> <span class="mix-recipe" id="mix-recipe">Click patterns to add...</span> <button class="clear-mix-btn" id="clear-mix-btn">Clear</button> @@ -71,8 +76,28 @@ </div> </section> + <!-- Dataset Mode: real-world preset or pasted series --> + <section class="dataset-section hidden" id="dataset-section"> + <div class="dataset-row"> + <label class="dataset-label"> + Source: + <select id="dataset-source"></select> + </label> + <p id="dataset-description" class="dataset-description"></p> + <label id="custom-input-wrapper" class="dataset-custom hidden"> + Custom series (numbers, comma- or whitespace-separated, or a JSON array): + <textarea id="custom-input" rows="4" placeholder="e.g. 100, 102, 99, 110, ..."></textarea> + </label> + </div> + <p class="dataset-hint"> + Pick which algorithms to overlay in the + <strong>Otava Analysis</strong> panel below. + </p> + <p class="dataset-status" id="dataset-status" hidden></p> + </section> + <!-- Dynamic Parameters --> - <section class="controls top-controls" id="params-section"> + <section class="controls top-controls hide-in-dataset" id="params-section"> <div id="dynamic-params" class="dynamic-params"> <!-- Dynamic parameters will be inserted here --> </div> @@ -88,7 +113,7 @@ <h3 class="section-title">Analysis Controls</h3> <div class="analysis-toolbar"> - <div class="control-actions"> + <div class="control-actions hide-in-dataset"> <button id="generate-btn" class="btn-primary">Analyse</button> <button id="show-all-btn" class="btn-secondary">Show All Graphs</button> </div> @@ -118,11 +143,29 @@ <!-- Otava Analysis Controls --> <div class="controls otava-controls"> <div class="panel-header"> - <label for="run-otava-checkbox" class="panel-title">Otava Analysis</label> - <input type="checkbox" id="run-otava-checkbox" checked> + <span class="panel-title">Otava Analysis</span> <button class="method-help-btn" data-method="otava" title="Learn about Otava">?</button> </div> + <div class="otava-algos" id="otava-algos"> + <label class="otava-algo-row"> + <input type="checkbox" class="otava-algo-checkbox" + data-algo="split" id="otava-algo-split" checked> + split-edivisive <span class="algo-hint">(default)</span> + </label> + <label class="otava-algo-row"> + <input type="checkbox" class="otava-algo-checkbox" + data-algo="orig" id="otava-algo-orig"> + orig-edivisive <span class="algo-hint">(permutation test)</span> + </label> + <label class="otava-algo-row"> + <input type="checkbox" class="otava-algo-checkbox" + data-algo="deterministic" id="otava-algo-deterministic"> + deterministic-edivisive + <span class="algo-hint">(<a href="https://github.com/apache/otava/pull/154"; target="_blank">PR #154</a>)</span> + </label> + </div> + <div class="panel-fields"> <div class="control-group"> <label for="window-len-input" class="param-label">Window Length</label> @@ -152,7 +195,7 @@ </div> <!-- Moving Average Analysis Controls --> - <div class="controls ma-controls"> + <div class="controls ma-controls hide-in-dataset"> <div class="panel-header"> <label for="run-ma-checkbox" class="panel-title">Moving Average Analysis</label> <input type="checkbox" id="run-ma-checkbox"> @@ -187,7 +230,7 @@ </div> <!-- Boundary Analysis Controls --> - <div class="controls boundary-controls"> + <div class="controls boundary-controls hide-in-dataset"> <div class="panel-header"> <label for="run-boundary-checkbox" class="panel-title">Boundary Analysis</label> <input type="checkbox" id="run-boundary-checkbox"> @@ -219,7 +262,7 @@ </div> <!-- Threshold Based Alerts Controls --> - <div class="controls threshold-controls"> + <div class="controls threshold-controls hide-in-dataset"> <div class="panel-header"> <label for="run-threshold-checkbox" class="panel-title">Threshold Alerts</label> <input type="checkbox" id="run-threshold-checkbox"> @@ -237,7 +280,7 @@ </div> <!-- Sliding Window Controls --> - <div class="controls sliding-window-controls"> + <div class="controls sliding-window-controls hide-in-dataset"> <div class="panel-header"> <label for="run-sliding-window-checkbox" class="panel-title">Sliding Window</label> <input type="checkbox" id="run-sliding-window-checkbox"> @@ -259,7 +302,7 @@ </div> <!-- Std Dev Controls --> - <div class="controls stddev-controls"> + <div class="controls stddev-controls hide-in-dataset"> <div class="panel-header"> <label for="run-stddev-checkbox" class="panel-title">Std Dev</label> <input type="checkbox" id="run-stddev-checkbox"> @@ -279,7 +322,7 @@ </section> <!-- Generator Info --> - <section class="generator-info" id="generator-info"> + <section class="generator-info hide-in-dataset" id="generator-info"> <div class="generator-info-header"> <div class="generator-info-main"> <h3 id="generator-title">Constant</h3> @@ -309,7 +352,7 @@ </section> <!-- Chart Legend --> - <section class="chart-legend"> + <section class="chart-legend hide-in-dataset"> <div class="legend-item"> <span class="legend-line ground-truth"></span> <span>Ground Truth</span> @@ -329,7 +372,7 @@ </section> <!-- Accuracy Metrics --> - <section class="accuracy-metrics" id="accuracy-metrics"> + <section class="accuracy-metrics hide-in-dataset" id="accuracy-metrics"> <h3>Detection Accuracy</h3> <table class="accuracy-table"> <thead> @@ -375,7 +418,7 @@ </section> <!-- Change Points Comparison Table --> - <section class="change-points-detail" id="change-points-detail"> + <section class="change-points-detail hide-in-dataset" id="change-points-detail"> <h3>Change Points Comparison</h3> <div class="comparison-tables"> <div class="table-wrapper"> @@ -412,6 +455,17 @@ </div> </section> + <!-- Dataset Mode: per-algorithm detected change-point table --> + <section class="dataset-results hidden" id="dataset-results"> + <h3>Detected change-point indices per algorithm</h3> + <table class="dataset-results-table"> + <thead> + <tr><th style="width:30%;">Algorithm</th><th style="width:8%;">Count</th><th>Indices</th></tr> + </thead> + <tbody id="dataset-results-body"></tbody> + </table> + </section> + <!-- Multi-chart view for "Show All" --> <section class="multi-chart-container hidden" id="multi-chart-container"> <h2>All Test Patterns - Otava Comparison</h2> @@ -431,6 +485,6 @@ </p> </footer> - <script src="/static/js/app.js"></script> + <script src="/static/js/app.js?v={{ version }}"></script> </body> </html>
