mfrancisc commented on issue #9058:
URL: https://github.com/apache/devlake/issues/9058#issuecomment-5352999563
I agree the GitLab API (or GitLab itself) is a hard ceiling. Extra DevLake
parallelism on one token would only burn the same quota faster. GitLab’s
authenticated limit is per user, not per PAT, and requests from one DevLake
instance also share one IP, which can be a second cap.
What we could do on the GitLab side, rather than pushing one token as hard
as possible:
- One GitLab connection per team/org, each with its own bot (or group access
token)
- Set rateLimitPerHour on each connection below what GitLab actually allows
(or 0 to follow RateLimit-Limit response header)
- Incremental collection after the first backfill, and keep
PIPELINE_MAX_PARALLEL modest on a given connection
That is how we could try to avoid melting GitLab.
But I guess that this does not by itself get a daily incremental cycle done
at ~6.5k repos if the limiter is still a single runner and sequential GitLab
stages
So we would still want to understand:
1. Has anyone actually run a few thousand GitLab repos on one instance, and
with what project size / PIPELINE_MAX_PARALLEL / sync policy?
2. If one process cannot finish a daily incremental cycle even when GitLab
is paced correctly, is the intended answer still “bigger machine, one process”,
or adding support for several collectors sharing one database with disjoint
scopes, one scheduler and many workers could be a potential enhancement to the
framework ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
