ShivamSharma43 opened a new issue, #9372:
URL: https://github.com/apache/devlake/issues/9372

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/devlake/issues?q=is%3Aissue) and found no 
similar issues.
   
   ### What happened
   
   The Bitbucket Cloud `Collect PR Commits` subtask 
(`plugins/bitbucket/tasks/pr_commit_collector.go`) collects at most 100 commits 
per pull request.
   
   The query sets a `fields` filter that lists only `values.*` keys:
   
   ```go
   query.Set("fields", 
"values.hash,values.date,values.message,values.author.raw,values.author.user.username,values.author.user.account_id,values.links.self")
   ```
   
   Bitbucket's partial response returns only the listed fields, so the response 
has no top-level `next`. The collector pages with `GetNextPageCustomData` 
(`plugins/bitbucket/tasks/api_common.go`), which reads `next` and returns 
`api.ErrFinishCollect` when it is empty. On a full first page (`pagelen=100`) 
it always finds `next` empty and stops.
   
   The other Bitbucket collectors add `page,pagelen,size` to their `fields` for 
the same reason; this one doesn't include `next`.
   
   ### What do you expect to happen
   
   All commits of a pull request are collected, following `next` until it is 
absent.
   
   ### How to reproduce
   
   Against a public repo, with `pagelen=2` so a PR with 6 commits needs more 
than one page:
   
   ```
   $ curl -s 
'https://api.bitbucket.org/2.0/repositories/atlassian/atlassian-connect-express/pullrequests/591/commits?pagelen=2&fields=values.hash'
   {"values": [{"hash": "a50685cd..."}, {"hash": "981b4fe3..."}]}
   
   $ curl -s 
'https://api.bitbucket.org/2.0/repositories/atlassian/atlassian-connect-express/pullrequests/591/commits?pagelen=2&fields=values.hash,next'
   {"values": [...], "next": 
"https://api.bitbucket.org/2.0/repositories/atlassian/atlassian-connect-express/pullrequests/591/commits?pagelen=2&fields=values.hash%2Cnext&page=E8Ux"}
   ```
   
   Without `next` in `fields` there is nothing to follow. In DevLake, a PR with 
more than 100 commits ends up with 100 rows in 
`_raw_bitbucket_api_pull_request_commits` and `pull_request_commits`.
   
   ### Anything else
   
   Proposed fix (I have a PR ready): add `next` to the `fields` list, and move 
the query into a small function so it can be unit tested.
   
   ### Version
   
   main
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to