dmorfav opened a new issue, #9419:
URL: https://github.com/apache/devlake/issues/9419

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/devlake/issues?q=is%3Aissue) and found no 
similar issues.
   
   ### What happened
   
   Since #8931, Jira issues are collected through 
`/rest/api/3/search/jql?expand=changelog`, which embeds only the newest 40 
changelog histories per issue. Until #9377, `collectIssueChangelogs` fetched 
the full changelog only for issues with `changelog_total > 100`, so issues with 
41–100 histories kept only their newest 40 entries and lost the oldest ones, 
typically their first status transitions (for example the first move to "In 
Progress").
   
   #9377 fixed the threshold on `main`, but two things are still open:
   
   1. **The latest published image, v1.0.3-beta19, has the bug.** It already 
contains #8931 (it collects through `search/jql`) but not #9377. `v1.0.3` and 
`release-v1.0` still use the agile endpoint and are not affected.
   2. **Upgrading does not repair data already collected.** In incremental mode 
`collectIssueChangelogs` only selects issues with `i.updated > since`. Issues 
that were truncated by an affected build and have not been updated since are 
never re-collected, so their history stays incomplete after the upgrade unless 
the user runs a full sync.
   
   On one of our boards, 213 of 329 issues finished in the last six months had 
an incomplete changelog (fewer rows in `_tool_jira_issue_changelogs` than 
`changelog_total`). For 89 of them the entry into the first in-progress status 
was lost, so any metric that starts at that transition (cycle time, flow 
efficiency, issues "started and finished") silently left them out.
   
   ### What do you expect to happen
   
   1. A beta (or the next release) that includes #9377.
   2. After upgrading, incremental syncs also re-collect the changelog of 
issues whose stored history is incomplete, or the upgrade notes say that a full 
sync is needed.
   
   ### How to reproduce
   
   1. On v1.0.3-beta19, collect a Jira board that has issues with between 41 
and 100 changelog entries.
   2. Compare, per issue, `changelog_total` in `_tool_jira_issues` with the 
number of rows in `_tool_jira_issue_changelogs` (query below): the affected 
issues keep only their most recent histories. In our instance 624 issues are 
affected and 600 of them have exactly 40 rows.
   3. Upgrade to a build that contains #9377 and run an incremental sync: 
issues that have not been updated since the previous sync still have 40 rows.
   
   ### Anything else
   
   Possible fix for the second point: in `collectIssueChangelogs`, also select 
issues whose stored changelog is incomplete, even if they were not updated:
   
   ```go
   if apiCollector.IsIncremental() && apiCollector.GetSince() != nil {
        clauses = append(clauses, dal.Where(
                "(i.updated > ? OR (SELECT COUNT(*) FROM 
_tool_jira_issue_changelogs c "+
                        "WHERE c.connection_id = i.connection_id AND c.issue_id 
= i.issue_id) < i.changelog_total)",
                apiCollector.GetSince()))
   }
   ```
   
   Workaround: after upgrading, trigger the blueprint once with "Full Sync".
   
   To find affected issues:
   
   ```sql
   SELECT t.connection_id, t.issue_key, t.changelog_total, 
COUNT(c.changelog_id) AS stored_changelogs
   FROM _tool_jira_issues t
   LEFT JOIN _tool_jira_issue_changelogs c ON c.connection_id = t.connection_id 
AND c.issue_id = t.issue_id
   GROUP BY t.connection_id, t.issue_id, t.issue_key, t.changelog_total
   HAVING COUNT(c.changelog_id) < t.changelog_total;
   ```
   
   ### Version
   
   v1.0.3-beta19 (#9377 is on `main` only)
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to