klesh commented on issue #8187:
URL:
https://github.com/apache/incubator-devlake/issues/8187#issuecomment-2510736397
> It has been tested, and the issue of only collecting 100 records has been
fixed.
>
> However, based on the content of the PR, it still seems to be limited to
100 pages of data:
>
> ```
> //incubator-devlake/backend/plugins/gitlab/tasks/deployment_collector.go
>
> apiCollector.InitCollector(helper.ApiCollectorArgs{
> RawDataSubTaskArgs: *rawDataSubTaskArgs,
> ApiClient: data.ApiClient,
> PageSize: 100, << here
> UrlTemplate: "projects/{{ .Params.ProjectId
}}/deployments",
> ...
> ```
>
> ```
> // backend/helpers/pluginhelper/api/api_collector.go
> reqDataCopy := RequestData{
> Pager: &Pager{
> Page: i + 1,
> Size: collector.args.PageSize, << here
> Skip: collector.args.PageSize * (i),
> },
> Input: reqData.Input,
> InputJSON: reqData.InputJSON,
> }
> ```
>
> For example:
>
> I have set up to collect data for one year on DevLake, and the total
number of records will exceed 10,000. It seems that only 10,000 records will be
collected?
>
> Is this scenario correct? My assumption is that regardless of the time
range setting, I must be able to collect all the data.
>
> I'm not sure if I understand this correctly. Please let me know if there
is any issue.
No, it won't. `Size` is ` PageSize`, it doesn't affect the total number of
records to be collected as long as the `Skip` grows. No?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
