stigi opened a new pull request, #9334: URL: https://github.com/apache/devlake/pull/9334
### Summary `WorkerScheduler.WaitAsync` can return `nil` although a task failed. The error is then reported by the next `WaitAsync` call on the same client, which is usually the next subtask. A failed task reports its error with a panic, and the ants pool's panic handler records it. But `SubmitBlocking` calls `waitGroup.Done()` in a defer inside the task. Go runs that defer while the panic unwinds, before ants recovers and calls the handler. So `WaitAsync` can wake up, see no errors and return `nil`. The error is recorded a moment later. When this happens to a collector, the subtask counts as successful and `StatefulApiCollector` saves its incremental state. The data that the failed requests should have collected is never collected again. On our instance the CircleCI token lost access to the project, and every request got `404 Project not found` for three weeks: - On most nights `collectPipelines` passed and moved its `since` marker forward. The 404 for `/v2/project/.../pipeline` was reported by `collectWorkflows`, which never calls that URL. - On other nights `collectPipelines` failed correctly. - After the token was fixed, the collector continued from the last marker, and two weeks of pipelines and deployments were missing. This PR recovers the panic inside the task and records it before `Done()`. <details> <summary>Sequence</summary> 1. `DoAsync` exhausts its retries and returns an error. The `SubmitBlocking` task calls `panic(err)`. 2. The task's deferred `s.waitGroup.Done()` runs, and the counter reaches 0. 3. `WaitAsync` wakes up, reads `workerErrors` (empty) and returns `nil`. 4. The ants worker recovers and calls the panic handler, and `checkError` appends the error. </details> ### Does this close any open issues? No issue filed. ### Screenshots Not applicable (helper-only change). ### Tests `helpers/pluginhelper/api/worker_scheduler_test.go`: `TestWorkerSchedulerWaitAsyncReturnsTaskError` submits a failing task 1000 times and checks that `WaitAsync` returns the error each time. Without the fix it fails within a few hundred runs. With the fix it passes, also at `-count=10`. `go test ./helpers/pluginhelper/api/ ./plugins/circleci/tasks/ ./plugins/github/tasks/` passes. <sub>🤖 Generated with [Claude Code](https://claude.com/claude-code)</sub> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
