dosubot[bot] commented on issue #8614: URL: https://github.com/apache/incubator-devlake/issues/8614#issuecomment-3410179296
<!-- Answer --> There isn't currently built-in handling in the github_graphql collector for stream errors like `CANCEL` or for large response cancellations from GitHub's servers. The collector uses pagination and retries, but if GitHub terminates the connection due to large responses, the task may hang and keep retrying without a clear error message or fail-fast behavior. This is especially likely with very large repositories, where API responses can exceed server or client limits, triggering these errors. No recent fixes or documentation address this scenario. As a workaround, you could try reducing the scope of data collection (for example, by limiting the date range or repository size), or lowering the page size if that's configurable. For long-term stability, it would be beneficial for the collector to implement clearer error handling for these cases—such as failing fast with a descriptive message when a stream error or large response cancellation is detected, and possibly exposing configuration for timeouts or retry limits. If you need more robust handling, you may want to suggest improvements to the maintainers for better support of large repositories and clearer error reporting in the github_graphql collector. References: - [job_collector.go implementation](https://github.com/apache/incubator-devlake/blob/main/backend/plugins/github_graphql/tasks/job_collector.go) - [GraphqlCollector error handling](https://github.com/apache/incubator-devlake/blob/main/backend/helpers/pluginhelper/api/graphql_collector.go) <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/39c39ab2-1df6-40f2-8762-c305b17cb3d2?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/39c39ab2-1df6-40f2-8762-c305b17cb3d2?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/39c39ab2-1df6-40f2-8762-c305b17cb3d2?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/39c39ab2-1df6-40f2-8762-c305b17cb3d2?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/39c39ab2-1df6-40f2-8762-c305b17cb3d2?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/39c39ab2-1df6-40f2-8762-c305b17cb3d2?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/39c39ab2-1df6-40f2-8762-c305b17cb3d2?feedback_type=other)</sup> [](https://app.dosu.dev/b4e8e847-d479-4541-83a8-d88d83fea5c9/ask?utm_source=githu b) [](https://go.dosu.dev/discord-bot) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/incubator-devlake/issues/8614) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
