klesh commented on issue #8187: URL: https://github.com/apache/incubator-devlake/issues/8187#issuecomment-2510736397
> It has been tested, and the issue of only collecting 100 records has been fixed. > > However, based on the content of the PR, it still seems to be limited to 100 pages of data: > > ``` > //incubator-devlake/backend/plugins/gitlab/tasks/deployment_collector.go > > apiCollector.InitCollector(helper.ApiCollectorArgs{ > RawDataSubTaskArgs: *rawDataSubTaskArgs, > ApiClient: data.ApiClient, > PageSize: 100, << here > UrlTemplate: "projects/{{ .Params.ProjectId }}/deployments", > ... > ``` > > ``` > // backend/helpers/pluginhelper/api/api_collector.go > reqDataCopy := RequestData{ > Pager: &Pager{ > Page: i + 1, > Size: collector.args.PageSize, << here > Skip: collector.args.PageSize * (i), > }, > Input: reqData.Input, > InputJSON: reqData.InputJSON, > } > ``` > > For example: > > I have set up to collect data for one year on DevLake, and the total number of records will exceed 10,000. It seems that only 10,000 records will be collected? > > Is this scenario correct? My assumption is that regardless of the time range setting, I must be able to collect all the data. > > I'm not sure if I understand this correctly. Please let me know if there is any issue. No, it won't. `Size` is ` PageSize`, it doesn't affect the total number of records to be collected as long as the `Skip` grows. No? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@devlake.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org