cedriclecoz opened a new issue, #9315: URL: https://github.com/apache/devlake/issues/9315
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar issues. ### What happened For one of our GitHub repositories, the `Collect Job Runs` subtask fails on **every** pipeline run. The task ends `TASK_FAILED` and the pipeline ends `TASK_PARTIAL` (we use `skip_on_fail`). The failing request is always the jobs listing of a workflow run with many jobs (about 70): ``` GET repos/<org>/<repo>/actions/runs/<run_id>/jobs?page=1&per_page=100 ``` GitHub takes about 11 seconds to build this response, which hits its own server timeout, so it returns `502 {"message": "Server Error"}`. The same request with a smaller page size succeeds: | `per_page` | Result (repeated manually with `gh api`) | |---|---| | 100 | 502 after about 11.0 s, every time (3/3 and 5/5 on two different runs) | | 50 | 200 in about 8–10 s | The page size is hard-coded in `plugins/github/tasks/cicd_job_collector.go`. ### What do you expect to happen The collector should use a page size that GitHub can serve reliably for large runs. ### How to reproduce Add a GitHub repo scope with CI/CD (workflow runs/jobs) collection enabled, where at least one workflow run has about 70 or more jobs. In our case it's a C++ project whose "Build on Linux" workflow uses a large build matrix, and the run had 3 attempts. 2. Run the blueprint (incremental or full sync). 3. `Collect Job Runs` fails with the 502 above, and so does every later run. ### Anything else - **How often:** every run, for the affected repo. - **Rate limits:** not involved. The token has plenty of rate limit left, and other repos on the same connection collect fine. ### Version v1.0.3-beta19 ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
