ShivamSharma43 opened a new issue, #9372: URL: https://github.com/apache/devlake/issues/9372
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/devlake/issues?q=is%3Aissue) and found no similar issues. ### What happened The Bitbucket Cloud `Collect PR Commits` subtask (`plugins/bitbucket/tasks/pr_commit_collector.go`) collects at most 100 commits per pull request. The query sets a `fields` filter that lists only `values.*` keys: ```go query.Set("fields", "values.hash,values.date,values.message,values.author.raw,values.author.user.username,values.author.user.account_id,values.links.self") ``` Bitbucket's partial response returns only the listed fields, so the response has no top-level `next`. The collector pages with `GetNextPageCustomData` (`plugins/bitbucket/tasks/api_common.go`), which reads `next` and returns `api.ErrFinishCollect` when it is empty. On a full first page (`pagelen=100`) it always finds `next` empty and stops. The other Bitbucket collectors add `page,pagelen,size` to their `fields` for the same reason; this one doesn't include `next`. ### What do you expect to happen All commits of a pull request are collected, following `next` until it is absent. ### How to reproduce Against a public repo, with `pagelen=2` so a PR with 6 commits needs more than one page: ``` $ curl -s 'https://api.bitbucket.org/2.0/repositories/atlassian/atlassian-connect-express/pullrequests/591/commits?pagelen=2&fields=values.hash' {"values": [{"hash": "a50685cd..."}, {"hash": "981b4fe3..."}]} $ curl -s 'https://api.bitbucket.org/2.0/repositories/atlassian/atlassian-connect-express/pullrequests/591/commits?pagelen=2&fields=values.hash,next' {"values": [...], "next": "https://api.bitbucket.org/2.0/repositories/atlassian/atlassian-connect-express/pullrequests/591/commits?pagelen=2&fields=values.hash%2Cnext&page=E8Ux"} ``` Without `next` in `fields` there is nothing to follow. In DevLake, a PR with more than 100 commits ends up with 100 rows in `_raw_bitbucket_api_pull_request_commits` and `pull_request_commits`. ### Anything else Proposed fix (I have a PR ready): add `next` to the `fields` list, and move the query into a small function so it can be unit tested. ### Version main ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
