dosubot[bot] commented on issue #8529: URL: https://github.com/apache/incubator-devlake/issues/8529#issuecomment-3163442439
<!-- Greeting --> Hi @zia-devsecops! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the incubator-devlake team. <!-- Answer --> This is a known limitation in DevLake's GitLab integration: user data collection uses offset pagination (with `page` and `per_page` parameters), which is subject to GitLab's maximum offset of 5000. When the user count exceeds this, data collection fails—keyset pagination is not implemented for GitLab users in DevLake, so the error message about switching to keyset pagination can't be addressed by configuration or environment variables right now. This has been discussed in detail in open issues ([#8523](https://github.com/apache/incubator-devlake/issues/8523), [#8283](https://github.com/apache/incubator-devlake/issues/8283)), and no recent changes or merged PRs resolve it. Workarounds include periodically deduplicating or cleaning up the `_raw_gitlab_api_users` table, or limiting how often you collect all users. There is a proposed (but not merged) solution to collect users only once per pipeline and/or use an environment variable (`GITLAB_SERVER_COLLECT_ALL_USERS`) to reduce duplication and table size, but this is not available in 1.0.2-beta5. The underlying code for user collection is here and confirms only offset pagination is used ([source](https://github.com/apache/incubator-devlake/blob/2c15ad880dce830c6512fa58039527cbf4745b8b/backend/plugins/gitlab/tasks/account_collector.go)). For now, if your user base is large, you will hit this limit until a new pagination strategy is implemented. Keep an eye on the linked issues for updates. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/ddcb4cee-f3be-4abd-8204-14a07a94effd?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/ddcb4cee-f3be-4abd-8204-14a07a94effd?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/ddcb4cee-f3be-4abd-8204-14a07a94effd?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/ddcb4cee-f3be-4abd-8204-14a07a94effd?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/ddcb4cee-f3be-4abd-8204-14a07a94effd?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/ddcb4cee-f3be-4abd-8204-14a07a94effd?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/ddcb4cee-f3be-4abd-8204-14a07a94effd?feedback_type=other)</sup> [](https://app.dosu.dev/b4e8e847-d479-4541-83a8-d88d83fea5c9/ask?utm_source=githu b) [](https://go.dosu.dev/discord-bot) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/incubator-devlake/issues/8529) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@devlake.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
