yamoyamoto opened a new pull request, #8844: URL: https://github.com/apache/incubator-devlake/pull/8844
### ⚠️ Pre Checklist - [x] I have read through the [Contributing Documentation](https://devlake.apache.org/community/). - [x] I have added relevant tests. - [x] I have added relevant documentation. - [x] I will add labels to the PR, such as `pr-type/bug-fix`. ### Summary Repositories with more than ~40k GitHub Actions workflow runs cannot be collected today: `Collect Workflow Runs` hits GitHub's `per_page * page > 40,000` cap in unfiltered mode and fails with HTTP 422, leaving `_tool_github_runs` empty. This PR switches the collector to always use filtered mode (`created=<from>..<to>`) and adds adaptive time-window bisection to work around filtered mode's own 1,000-item-per-search cap. Leaf windows are collected through a single `ApiCollector` fed by an `Input` iterator, so the raw-table `Delete` still fires only once. `github_graphql` inherits the fix automatically via its existing `CollectRunsMeta` import. ### Does this close any open issues? Closes #8842 ### Screenshots N/A — internal collector change with no UI surface. ### Other Information - Unit tests in the new `cicd_run_collector_test.go` cover pagination edges, `total_count >= 1000` / HTTP 422 bisection triggers, the non-overlapping integer-second split rule, bootstrap from epoch, and a thin integration check that the raw-table `Delete` is invoked exactly once regardless of how many leaf windows are produced. `make build` and `make unit-test` pass locally. - Validated in our environment: we built this branch into a container image on top of `v1.0.3-beta10` and deployed it against our own DevLake instance pointing at a GitHub repository that had previously hit the 40k cap. The `Collect Workflow Runs` subtask now completes successfully and `_tool_github_runs` is populated as expected, so the change has been exercised against real GitHub traffic rather than only synthetic mocks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
