RyoKusnadi opened a new pull request, #9324: URL: https://github.com/apache/devlake/pull/9324
### ⚠️ Pre Checklist - [x] I have read through the [Contributing Documentation](https://devlake.apache.org/community/). - [x] I have added relevant tests. - [ ] I have added relevant documentation. - [x] I will add labels to the PR, such as `pr-type/bug-fix`, `pr-type/feature-development`, etc. ### Summary **Problem:** For a Jira issue with more than 20 worklogs, `collectWorklogs` sends the same request once per page. Every response holds the same worklogs, so `_raw_jira_api_worklogs` gets about `n * ceil(n/50)` rows instead of `n`. The raw table grows fast and `extractWorklogs` has to read all of it. The reporter saw a 44 GB raw table and a 13 h extractor. **How it happens:** The collector only picks issues with `worklog_total > 20`. For these it reads `total` from the first response, so `GetTotalPagesFromResponse` returns `ceil(total/50)`. `fetchPagesDetermined` then sends one request per page. The URL template does not use the pager and `Query` returned `nil`, so every page request is `api/2/issue/{id}/worklog` with no parameters. Any issue with 21 or more worklogs triggers this. **Cause:** `Query` in `backend/plugins/jira/tasks/worklog_collector.go` returned `nil, nil` (added in #7466 with the comment that the API returns all worklogs). The paging loop was kept, but nothing told Jira which page to return. **Fix:** `Query` now sets `startAt` and `maxResults` from `reqData.Pager`, the same way `issue_changelog_collector.go` does. I moved it into a named function, `buildWorklogQuery`, so a unit test can call it. **Scope:** - Other Jira collectors that page already set `startAt`; I did not find another one with an empty `Query`. - Jira Server/DC: I could not check whether it honours `startAt` on this endpoint. If it ignores it, the requests stay identical, as they are today, so nothing gets worse. - Not changed: the idea of switching `extractWorklogs` to a stateful extractor. It is a separate change. - Rows already in the raw table are not cleaned up. **Testing:** - New `TestBuildWorklogQuery` checks `startAt` and `maxResults` for pages 1, 2 and 10. - With `Query` returning `nil` (the old behaviour) it fails: `startAt = "", want "0"`. With the fix it passes. - Ran `go vet ./plugins/jira/tasks/`, `go test ./plugins/jira/tasks/ -run TestBuildWorklogQuery` and `gofmt -l` on the two changed files. - Not run: the rest of the package tests, the e2e tests, and a live Jira. I did not call a real Jira API. **Risk:** Low. Only the query string of the worklog request changes, and only for issues with more than 20 worklogs. To roll back, revert the commit. ### Does this close any open issues? Closes #9323 ### Screenshots N/A ### Other Information None. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
