dmorfav opened a new issue, #9419: URL: https://github.com/apache/devlake/issues/9419
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/devlake/issues?q=is%3Aissue) and found no similar issues. ### What happened Since #8931, Jira issues are collected through `/rest/api/3/search/jql?expand=changelog`, which embeds only the newest 40 changelog histories per issue. Until #9377, `collectIssueChangelogs` fetched the full changelog only for issues with `changelog_total > 100`, so issues with 41–100 histories kept only their newest 40 entries and lost the oldest ones, typically their first status transitions (for example the first move to "In Progress"). #9377 fixed the threshold on `main`, but two things are still open: 1. **The latest published image, v1.0.3-beta19, has the bug.** It already contains #8931 (it collects through `search/jql`) but not #9377. `v1.0.3` and `release-v1.0` still use the agile endpoint and are not affected. 2. **Upgrading does not repair data already collected.** In incremental mode `collectIssueChangelogs` only selects issues with `i.updated > since`. Issues that were truncated by an affected build and have not been updated since are never re-collected, so their history stays incomplete after the upgrade unless the user runs a full sync. On one of our boards, 213 of 329 issues finished in the last six months had an incomplete changelog (fewer rows in `_tool_jira_issue_changelogs` than `changelog_total`). For 89 of them the entry into the first in-progress status was lost, so any metric that starts at that transition (cycle time, flow efficiency, issues "started and finished") silently left them out. ### What do you expect to happen 1. A beta (or the next release) that includes #9377. 2. After upgrading, incremental syncs also re-collect the changelog of issues whose stored history is incomplete, or the upgrade notes say that a full sync is needed. ### How to reproduce 1. On v1.0.3-beta19, collect a Jira board that has issues with between 41 and 100 changelog entries. 2. Compare, per issue, `changelog_total` in `_tool_jira_issues` with the number of rows in `_tool_jira_issue_changelogs` (query below): the affected issues keep only their most recent histories. In our instance 624 issues are affected and 600 of them have exactly 40 rows. 3. Upgrade to a build that contains #9377 and run an incremental sync: issues that have not been updated since the previous sync still have 40 rows. ### Anything else Possible fix for the second point: in `collectIssueChangelogs`, also select issues whose stored changelog is incomplete, even if they were not updated: ```go if apiCollector.IsIncremental() && apiCollector.GetSince() != nil { clauses = append(clauses, dal.Where( "(i.updated > ? OR (SELECT COUNT(*) FROM _tool_jira_issue_changelogs c "+ "WHERE c.connection_id = i.connection_id AND c.issue_id = i.issue_id) < i.changelog_total)", apiCollector.GetSince())) } ``` Workaround: after upgrading, trigger the blueprint once with "Full Sync". To find affected issues: ```sql SELECT t.connection_id, t.issue_key, t.changelog_total, COUNT(c.changelog_id) AS stored_changelogs FROM _tool_jira_issues t LEFT JOIN _tool_jira_issue_changelogs c ON c.connection_id = t.connection_id AND c.issue_id = t.issue_id GROUP BY t.connection_id, t.issue_id, t.issue_key, t.changelog_total HAVING COUNT(c.changelog_id) < t.changelog_total; ``` ### Version v1.0.3-beta19 (#9377 is on `main` only) ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
