ShivamSharma43 opened a new issue, #9340:
URL: https://github.com/apache/devlake/issues/9340

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/devlake/issues?q=is%3Aissue) and found no 
similar issues.
   
   ### What happened
   
   The Taiga `collectEpics`, `collectIssues`, `collectTasks` and 
`collectUserStories` subtasks only collect the first 30 items of a project.
   
   The four collectors set `PageSize: 1000` but their `Query` only sends 
`project`:
   
   ```go
   query.Set("project", fmt.Sprintf("%d", data.Options.ProjectId))
   // Don't specify page - get all results
   ```
   
   Taiga list endpoints are paginated unless the `x-disable-pagination` header 
is sent. taiga-back's `PaginationMixin` uses `PAGINATE_BY: 30` 
(`settings/common.py`) when no `page_size` is given, and returns page 1 when no 
`page` is given. So each request gets the first 30 items. 30 < `PageSize`, so 
the `ApiCollector` treats it as the last page and stops.
   
   There is a second effect. With no `GetTotalPages`/`GetNextPageCustomData`, 
the collector fetches pages 1..10 concurrently. All ten requests are identical, 
so the raw tables get the same 30 rows ten times.
   
   ### What do you expect to happen
   
   All epics, issues, tasks and user stories of the project are collected.
   
   ### How to reproduce
   
   1. Add a Taiga project with more than 30 issues (or user stories, tasks, 
epics) as a data scope.
   2. Run the blueprint.
   3. `_tool_taiga_issues` holds 30 rows for the project, and 
`_raw_taiga_api_issues` holds 10 copies of the same 30 items.
   
   ### Anything else
   
   Proposed fix (I have a PR ready): send `page` and `page_size` (1000, Taiga's 
`MAX_PAGINATE_BY`) and stop when the response has no `X-Pagination-Next` 
header. Taiga answers a page past the last one with 404, so a full last page 
must not trigger another request.
   
   ### Version
   
   main
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to