jbsmith7741 opened a new issue, #8941: URL: https://github.com/apache/devlake/issues/8941
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar feature requirement. ### Use case As an engineering manager, I want to view **historical code coverage and quality metric trends** for my SonarQube projects so that I can: - Track whether test coverage is improving or regressing over time. - Set OKR baselines and measure engineering quality improvements quarter over quarter. - Compare quality trajectories across teams and projects. - Correlate coverage changes with releases and refactors. Today, DevLake's SonarQube plugin only stores a **point-in-time snapshot** of metrics. Each sync overwrites the previous values, so there is no way to answer "How has our coverage trended over the past year?" without an external data pipeline. ### Description #### Problem The SonarQube plugin collects per-file metrics via `measures/component_tree` and stores them in `_tool_sonarqube_file_metrics` / `cq_file_metrics`. These tables are keyed by `(connection_id, file_metrics_key)` with no temporal dimension — every sync replaces the prior row. The only timestamp in the model is `last_analysis_date` on `_tool_sonarqube_projects` / `cq_projects`, which reflects only the most recent analysis. #### SonarQube APIs available but not used | Endpoint | Purpose | |----------|---------| | `api/measures/search_history` | Time-series metric values at project level | | `api/project_analyses/search` | Analysis metadata (dates, versions, quality gate events) | `measures/search_history` accepts `from` / `to` date parameters and returns chronological metric values across all retained analyses for a given project component. It supports pagination (`ps` max 1000) and the full set of standard metric keys (`coverage`, `ncloc`, `bugs`, `vulnerabilities`, `code_smells`, `security_hotspots`, ratings, duplication, complexity, etc.). #### Why project-level, not file-level SonarQube's housekeeping [deletes file-level and directory-level measures after each new analysis](https://community.sonarsource.com/t/api-measures-search-history-returns-only-date-history/730). Querying `measures/search_history` for a file key returns analysis dates but **no values**. Project-level history, by contrast, is retained for up to 5 years (thinned over time by [housekeeping rules](https://docs.sonarsource.com/sonarqube-server/latest/instance-administration/system-functions/housekeeping/)): | Age | Retention | |-----|-----------| | < 1 day | All snapshots | | > 1 day | 1 snapshot/day | | > 1 month | 1 snapshot/week | | > 1 year | 1 snapshot/month | | > 2 years | Only VERSION-tagged snapshots | | > 5 years | All deleted | File-level historical metrics are out of scope for this feature because the upstream data simply does not exist. The current per-file `cq_file_metrics` table continues to serve the "latest state" use case. #### Sync Policy integration The new collectors should respect the blueprint **Sync Policy `TimeAfter`** field, following the same `NewStatefulApiCollector` / `CollectorStateManager` pattern used by the Jira and GitHub plugins: - **Initial / full sync:** collect from `syncPolicy.TimeAfter` to now. - **Incremental syncs:** collect from `state.LatestSuccessStart` to now. - **`fullSync`:** re-fetch the entire history window. `GetSince()` maps directly to the SonarQube `from` query parameter. #### Proposed metrics All returned in a single paginated API call, so there is no added cost for breadth: | Category | Metric keys | |----------|-------------| | Coverage | `coverage`, `ncloc` | | Reliability | `bugs`, `reliability_rating` | | Maintainability | `code_smells`, `sqale_rating`, `complexity`, `cognitive_complexity` | | Security | `vulnerabilities`, `security_rating`, `security_hotspots` | | Duplication | `duplicated_lines_density` | `new_*` variants are intentionally excluded — SonarQube does not retain reliable history for them. ### Related issues _No response_ ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
