ShivamSharma43 opened a new issue, #9340: URL: https://github.com/apache/devlake/issues/9340
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/devlake/issues?q=is%3Aissue) and found no similar issues. ### What happened The Taiga `collectEpics`, `collectIssues`, `collectTasks` and `collectUserStories` subtasks only collect the first 30 items of a project. The four collectors set `PageSize: 1000` but their `Query` only sends `project`: ```go query.Set("project", fmt.Sprintf("%d", data.Options.ProjectId)) // Don't specify page - get all results ``` Taiga list endpoints are paginated unless the `x-disable-pagination` header is sent. taiga-back's `PaginationMixin` uses `PAGINATE_BY: 30` (`settings/common.py`) when no `page_size` is given, and returns page 1 when no `page` is given. So each request gets the first 30 items. 30 < `PageSize`, so the `ApiCollector` treats it as the last page and stops. There is a second effect. With no `GetTotalPages`/`GetNextPageCustomData`, the collector fetches pages 1..10 concurrently. All ten requests are identical, so the raw tables get the same 30 rows ten times. ### What do you expect to happen All epics, issues, tasks and user stories of the project are collected. ### How to reproduce 1. Add a Taiga project with more than 30 issues (or user stories, tasks, epics) as a data scope. 2. Run the blueprint. 3. `_tool_taiga_issues` holds 30 rows for the project, and `_raw_taiga_api_issues` holds 10 copies of the same 30 items. ### Anything else Proposed fix (I have a PR ready): send `page` and `page_size` (1000, Taiga's `MAX_PAGINATE_BY`) and stop when the response has no `X-Pagination-Next` header. Taiga answers a page past the last one with 404, so a full last page must not trigger another request. ### Version main ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
