cedriclecoz opened a new issue, #9240: URL: https://github.com/apache/devlake/issues/9240
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar issues. ### What happened One of my repo failed last night pipeline on the `Collect Pull Requests` task on a PR that had been closed (not merged) for more than a month and had not been updated since. For the story GitHub returned a 422 on it. That made me take a deeper look as there was no reason to re-fetch that PR. it looks like every closed but not merged PR is being re-fetched on every incremental run. On my instance that's about ~15k closed PRs. For example, one of the oldest PR in my DB was closed in August 2019, has not been updated since, however at every run: ```sql SELECT number, state, merged, github_created_at, closed_at, github_updated_at FROM _tool_github_pull_requests WHERE state = 'closed' AND merged = 0 AND merged_at IS NULL ORDER BY github_updated_at ASC LIMIT 1; ``` ``` number,state,merged,github_created_at,closed_at,github_updated_at 29,closed,0,2019-08-02 05:39:12,2019-08-08 08:31:19,2019-08-08 08:31:19 ``` ```sql SELECT id, created_at, url FROM _raw_github_api_pull_requests WHERE url LIKE '%/[REPOSITORY]/pulls/29' ORDER BY created_at DESC LIMIT 7; ``` ``` id,created_at,url 5014408,2026-10-19 00:27:23,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29 4992046,2026-10-18 00:27:07,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29 4969613,2026-10-17 00:27:19,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29 4947192,2026-10-16 00:27:38,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29 4924806,2026-10-15 00:27:32,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29 4902464,2026-10-14 00:27:21,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29 4880192,2026-10-13 00:27:17,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29 ``` 183 identical fetches over the last 182 days of raw data we retain, for a PR untouched for 7 years. Sanitized error: ``` subtask Collect Pull Requests ended unexpectedly caused by: Retry exceeded 3 times calling repos/[ORGANIZATION]/[REPOSITORY]/pulls/29. The last error was: Http DoAsync error calling [method:GET path:repos/[ORGANIZATION]/[REPOSITORY]/pulls/29 query:map[]]. Response: {"message":"The request could not be processed because too many files changed","documentation_url":"https://docs.github.com/rest/pulls/pulls#get-a-pull-request","status":"422"} (422) ``` ``` attached stack trace -- stack trace: | github.com/apache/devlake/core/runner.RunPluginSubTasks | /app/core/runner/run_task.go:341 | [...repeated from below...] Wraps: (2) subtask Collect Pull Requests ended unexpectedly Wraps: (3) attached stack trace -- stack trace: | github.com/apache/devlake/helpers/pluginhelper/api.(*WorkerScheduler).WaitAsync | /app/helpers/pluginhelper/api/worker_scheduler.go:173 | github.com/apache/devlake/helpers/pluginhelper/api.(*ApiCollector).Execute | /app/helpers/pluginhelper/api/api_collector.go:206 | github.com/apache/devlake/helpers/pluginhelper/api.(*StatefulApiCollector).Execute | /app/helpers/pluginhelper/api/api_collector_stateful.go:97 | github.com/apache/devlake/plugins/github/tasks.CollectApiPullRequests | /app/plugins/github/tasks/pr_collector.go:141 | github.com/apache/devlake/core/runner.runSubtask | /app/core/runner/run_task.go:433 | github.com/apache/devlake/core/runner.RunPluginSubTasks | /app/core/runner/run_task.go:338 | github.com/apache/devlake/core/runner.RunPluginTask | /app/core/runner/run_task.go:173 | github.com/apache/devlake/core/runner.RunTask | /app/core/runner/run_task.go:147 | github.com/apache/devlake/server/services.runTaskStandalone | /app/server/services/task_runner.go:114 | github.com/apache/devlake/server/services.RunTasksStandalone.func1 | /app/server/services/task.go:187 | runtime.goexit | /usr/local/go/src/runtime/asm_amd64.s:1771 Wraps: (4) Retry exceeded 3 times calling repos/[ORGANIZATION]/[REPOSITORY]/pulls/29. ... Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError ### What do you expect to happen A PR should only be re-fetched individually when it has actually changed since it was last collected, else in a few years time will be fetching way more closed than opened PR. Reopening, pushing, commenting, reviewing, merging or closing a PR all bump `updated_at`, so if a closed PR gets re-opened Devlake should pick it up. ### How to reproduce 1. Collect a repository that has closed-unmerged PRs. 2. Run incremental collection again. 3. Observe one `GET /repos/{org}/{repo}/pulls/{number}` per open **and** per closed-unmerged PR in the database, on every run, including PRs untouched for years: ```sql SELECT COUNT(*) FROM _tool_github_pull_requests WHERE state != 'closed' OR merged = 0; ``` ### Anything else Happens at every collection, and I assume collection time will increase the more unmerged closed PR there are. Also DB size will become bloated with duplicated data seeing that each non-updated run stores the same payload in _raw_github_api_pull_requests. Hope I didn't make obvious mistakes in that analysis. ### Version v1.0.3-beta18 ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
