ShivamSharma43 opened a new issue, #9365: URL: https://github.com/apache/devlake/issues/9365
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/devlake/issues?q=is%3Aissue) and found no similar issues. ### What happened The Asana collectors don't page through list endpoints correctly. 1. `collectStory`, `collectSubtask` and `collectTag` (`plugins/asana/tasks/{story,subtask,tag}_collector.go`) send `limit=100` but have no `PageSize` and no `GetNextPageCustomData`, so the `ApiCollector` makes one request per task and never sends `offset`. Only the first 100 stories, subtasks and tags of each task are collected. Stories include system events (assignments, section moves, due date changes), so busy tasks go past 100. 2. `collectTask` and `collectUser` do page with `offset`, but their `GetNextPageCustomData` returns `nil, nil` when `next_page` is null: ```go if resp.NextPage != nil && resp.NextPage.Offset != "" { return resp.NextPage.Offset, nil } return nil, nil ``` `nil, nil` doesn't end the collection. The collector sets `CustomData` to `nil` and requests the next page, and the query then has no `offset`, so it asks for page 1 again. When the last page is full (100 items) and Asana returns `"next_page": null`, the collector keeps re-collecting the project from the start and saving the same raw rows again. From the Asana docs (https://developers.asana.com/docs/pagination): `limit` must be between 1 and 100, and "If there are no more pages available, `next_page` will be null and no offset will be provided." ### What do you expect to happen All five collectors follow `next_page.offset` and stop when `next_page` is null. ### How to reproduce - Stories: collect a project with a task that has more than 100 stories. `_raw_asana_stories` has 100 rows for that task. - Tasks: the `GetNextPageCustomData` above returns `nil, nil` for a full page with `"next_page": null`, and `fetchPagesSequentially` in `helpers/pluginhelper/api/api_collector.go` only stops on a short page or `ErrFinishCollect`. ### Anything else Proposed fix (I have a PR ready): a shared `getNextAsanaPageOffset` that returns `api.ErrFinishCollect` when `next_page` is null, and a `setAsanaPageQuery` that sets `limit` and `offset`, used by the task, user, story, subtask and tag collectors. ### Version main ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
