ShivamSharma43 opened a new pull request, #9366:
URL: https://github.com/apache/devlake/pull/9366

   ### Summary
   
   The Asana collectors didn't page through list endpoints correctly:
   
   - `collectStory`, `collectSubtask` and `collectTag` sent `limit=100` but had 
no `PageSize` and no `GetNextPageCustomData`, so the `ApiCollector` made one 
request per task and never sent `offset`. Only the first 100 stories, subtasks 
and tags of each task were collected. Stories include system events, so busy 
tasks go past 100.
   - `collectTask` and `collectUser` returned `nil, nil` from 
`GetNextPageCustomData` when `next_page` was null. That doesn't end the 
collection: the collector requests another page with `CustomData` set to `nil`, 
the query has no `offset`, and Asana returns page 1 again. When the last page 
is full, the collection starts over from the first page and saves the same raw 
rows again.
   
   Asana's [pagination docs](https://developers.asana.com/docs/pagination): 
`limit` must be between 1 and 100, and "If there are no more pages available, 
`next_page` will be null and no offset will be provided."
   
   Changes:
   
   - `tasks/shared.go` (new):
     - `setAsanaPageQuery` sets `limit` from the pager and, after the first 
page, `offset`.
     - `getNextAsanaPageOffset` returns `next_page.offset`, or 
`api.ErrFinishCollect` when `next_page` is null.
   - The task and user collectors use these helpers instead of their own 
closures.
   - The story, subtask and tag collectors get `PageSize: 100` and use the same 
helpers, so they page through each task's list one page at a time.
   - `tasks/shared_test.go` (new): checks the query params, and checks that the 
next-page function returns the offset and stops when `next_page` is null.
   
   Request URLs (apart from `offset`), raw tables and extractors are unchanged.
   
   ### Does this close any open issues?
   
   Closes #9365
   
   ### Screenshots
   
   N/A
   
   ### Other Information
   
   - `go test ./plugins/asana/tasks/`: pass.
   - `go build`, `go vet`, `gofmt` and `golangci-lint run ./plugins/asana/...`: 
clean. The asana e2e tests need `E2E_DB_URL`, which I didn't run locally; they 
cover the extractors and converters, not the collectors.
   - No open PR or issue touches the asana plugin.
   - Suggested label: `pr-type/bug-fix`.
   - AI assistance disclosure: this change was prepared with the help of Claude 
Code (Claude Opus 5.5), and I reviewed the diff. The commit carries a 
`Generated-by:` trailer per the ASF generative tooling guidance.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to