ShivamSharma43 opened a new issue, #9365:
URL: https://github.com/apache/devlake/issues/9365

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/devlake/issues?q=is%3Aissue) and found no 
similar issues.
   
   ### What happened
   
   The Asana collectors don't page through list endpoints correctly.
   
   1. `collectStory`, `collectSubtask` and `collectTag` 
(`plugins/asana/tasks/{story,subtask,tag}_collector.go`) send `limit=100` but 
have no `PageSize` and no `GetNextPageCustomData`, so the `ApiCollector` makes 
one request per task and never sends `offset`. Only the first 100 stories, 
subtasks and tags of each task are collected. Stories include system events 
(assignments, section moves, due date changes), so busy tasks go past 100.
   
   2. `collectTask` and `collectUser` do page with `offset`, but their 
`GetNextPageCustomData` returns `nil, nil` when `next_page` is null:
   
   ```go
   if resp.NextPage != nil && resp.NextPage.Offset != "" {
        return resp.NextPage.Offset, nil
   }
   return nil, nil
   ```
   
   `nil, nil` doesn't end the collection. The collector sets `CustomData` to 
`nil` and requests the next page, and the query then has no `offset`, so it 
asks for page 1 again. When the last page is full (100 items) and Asana returns 
`"next_page": null`, the collector keeps re-collecting the project from the 
start and saving the same raw rows again.
   
   From the Asana docs (https://developers.asana.com/docs/pagination): `limit` 
must be between 1 and 100, and "If there are no more pages available, 
`next_page` will be null and no offset will be provided."
   
   ### What do you expect to happen
   
   All five collectors follow `next_page.offset` and stop when `next_page` is 
null.
   
   ### How to reproduce
   
   - Stories: collect a project with a task that has more than 100 stories. 
`_raw_asana_stories` has 100 rows for that task.
   - Tasks: the `GetNextPageCustomData` above returns `nil, nil` for a full 
page with `"next_page": null`, and `fetchPagesSequentially` in 
`helpers/pluginhelper/api/api_collector.go` only stops on a short page or 
`ErrFinishCollect`.
   
   ### Anything else
   
   Proposed fix (I have a PR ready): a shared `getNextAsanaPageOffset` that 
returns `api.ErrFinishCollect` when `next_page` is null, and a 
`setAsanaPageQuery` that sets `limit` and `offset`, used by the task, user, 
story, subtask and tag collectors.
   
   ### Version
   
   main
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to