cedriclecoz opened a new issue, #9140: URL: https://github.com/apache/devlake/issues/9140
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar issues. ### What happened The GitHub Collect Pull Requests subtask fails completely when a previously collected pull request returns HTTP 404 from GitHub. The collector retries `GET /repos/{org}/{repo}/pulls/{pr_number}` three times, then terminates the subtask. Subsequent scheduled collections retry the same unavailable pull request and fail again. Newer pull requests are therefore not collected, leaving repository dashboard data stale. The affected pull request was previously present in DevLake's GitHub cache with an `open` state, but its GitHub pull-request endpoint now returns `404 Not Found`. The reason it became unavailable is unknown; it likely has been deleted or made inaccessible by GitHub on request of the author (cred leak for example). As a workaround, after taking a DB snapshot, I deleted the verified affected row from `_tool_github_pull_requests` and its linked record in `_raw_github_api_pull_requests`. The next collection succeeded and newer pull requests appeared in dashboards. ### What do you expect to happen A 404 for one previously known pull request should not permanently block collection of later pull requests in the repository. The collector should handle an unavailable/stale pull request as a non-fatal condition, record or reconcile that state, continue collecting remaining pull requests, and avoid retrying the same known 404 on every scheduled run. A supported mechanism to invalidate or reconcile one stale cached pull-request record would also be helpful. ### How to reproduce 1. Configure the GitHub connector for a repository and run collection so a pull request is cached by DevLake. 2. Make that previously collected pull request unavailable to the configured GitHub token, for example by deleting it or otherwise making GitHub return 404 for `GET /repos/{org}/{repo}/pulls/{pr_number}`. Raising a support ticket with GitHub is likely needed. 3. Run the `Collect Pull Requests` subtask. 4. Observe that DevLake retries the individual pull-request endpoint and fails the entire subtask after the retry limit. 5. Run the scheduled collection again. 6. Observe that the same pull request is retried and collection fails again, so later pull requests are not ingested. ### Anything else Sanitized error: subtask Collect Pull Requests ended unexpectedly caused by: Retry exceeded 3 times calling repos/[ORGANIZATION]/[REPOSITORY]/pulls/[PR_NUMBER]. Response: { "message": "Not Found", "documentation_url": "https://docs.github.com/rest/pulls/pulls#get-a-pull-request";, "status": "404" } The stack trace reaches `plugins/github/tasks.CollectApiPullRequests` in `plugins/github/tasks/pr_collector.go`. The workaround removed exactly one verified cached pull-request record and its linked raw API payload in a transaction, after an RDS snapshot. This is not a general solution and should not be required for a single unavailable pull request. This has happened repeatedly since early July. It is unknown whether the pull request was deleted, made private/inaccessible, or removed by GitHub for another reason. ```text attached stack trace github.com/apache/devlake/core/runner.RunPluginSubTasks /app/core/runner/run_task.go:341 github.com/apache/devlake/helpers/pluginhelper/api.(*WorkerScheduler).WaitAsync /app/helpers/pluginhelper/api/worker_scheduler.go:173 github.com/apache/devlake/helpers/pluginhelper/api.(*ApiCollector).Execute /app/helpers/pluginhelper/api/api_collector.go:206 github.com/apache/devlake/helpers/pluginhelper/api.(*StatefulApiCollector).Execute /app/helpers/pluginhelper/api/api_collector_stateful.go:97 github.com/apache/devlake/plugins/github/tasks.CollectApiPullRequests /app/plugins/github/tasks/pr_collector.go:140 github.com/apache/devlake/core/runner.runSubtask /app/core/runner/run_task.go:433 github.com/apache/devlake/core/runner.RunPluginSubTasks /app/core/runner/run_task.go:338 github.com/apache/devlake/core/runner.RunPluginTask /app/core/runner/run_task.go:173 github.com/apache/devlake/core/runner.RunTask /app/core/runner/run_task.go:147 github.com/apache/devlake/server/services.runTaskStandalone /app/server/services/task_runner.go:114 github.com/apache/devlake/server/services.RunTasksStandalone.func1 /app/server/services/task.go:187 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1771 ``` ### Version v1.0.3-beta17 ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
