ShivamSharma43 opened a new pull request, #9366: URL: https://github.com/apache/devlake/pull/9366
### Summary The Asana collectors didn't page through list endpoints correctly: - `collectStory`, `collectSubtask` and `collectTag` sent `limit=100` but had no `PageSize` and no `GetNextPageCustomData`, so the `ApiCollector` made one request per task and never sent `offset`. Only the first 100 stories, subtasks and tags of each task were collected. Stories include system events, so busy tasks go past 100. - `collectTask` and `collectUser` returned `nil, nil` from `GetNextPageCustomData` when `next_page` was null. That doesn't end the collection: the collector requests another page with `CustomData` set to `nil`, the query has no `offset`, and Asana returns page 1 again. When the last page is full, the collection starts over from the first page and saves the same raw rows again. Asana's [pagination docs](https://developers.asana.com/docs/pagination): `limit` must be between 1 and 100, and "If there are no more pages available, `next_page` will be null and no offset will be provided." Changes: - `tasks/shared.go` (new): - `setAsanaPageQuery` sets `limit` from the pager and, after the first page, `offset`. - `getNextAsanaPageOffset` returns `next_page.offset`, or `api.ErrFinishCollect` when `next_page` is null. - The task and user collectors use these helpers instead of their own closures. - The story, subtask and tag collectors get `PageSize: 100` and use the same helpers, so they page through each task's list one page at a time. - `tasks/shared_test.go` (new): checks the query params, and checks that the next-page function returns the offset and stops when `next_page` is null. Request URLs (apart from `offset`), raw tables and extractors are unchanged. ### Does this close any open issues? Closes #9365 ### Screenshots N/A ### Other Information - `go test ./plugins/asana/tasks/`: pass. - `go build`, `go vet`, `gofmt` and `golangci-lint run ./plugins/asana/...`: clean. The asana e2e tests need `E2E_DB_URL`, which I didn't run locally; they cover the extractors and converters, not the collectors. - No open PR or issue touches the asana plugin. - Suggested label: `pr-type/bug-fix`. - AI assistance disclosure: this change was prepared with the help of Claude Code (Claude Opus 5.5), and I reviewed the diff. The commit carries a `Generated-by:` trailer per the ASF generative tooling guidance. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
