Startrekzky opened a new issue, #3642: URL: https://github.com/apache/incubator-devlake/issues/3642
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar feature requirement. ### Description DevLake v0.14 collects ALL GitHub Action `workflow runs` and `jobs` by default, which makes the collection speed very slow. For example, it took 4+ hours to collect data from 'apache/incubator-devlake' by 2 tokens. However, before collecting GitHub Action data, the collection speed was about 40min by 1 token. Therefore, we can see that there's a bottleneck in GitHub Action data collection. ### Use case DevLake can provide workflow as a filter to limit the size of GitHub `workflow runs` to collect, because: - from users' perspective, they don't always want to collect all CI data. They might sacrifice integrity for speed. - from DevLake's perspective, DevLake's DORA metrics are based on `deployments` in the production environment. Other CI jobs such as linter, checker and etc. are not taken into calculation, which means they're to some extent useless. Take 'apache/incubator-devlake' for example, around 30,000 requests will be posted to collect all CI data; while production deployments are all in the runs of one workflow 'Build-Images-Push-Docker'. By letting users to select this one single workflow before collection, the number of requests will be decreased to 57. Below is the prototype for how to add the workflow filter. `TBD` ### Related issues _No response_ ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
