klesh commented on issue #8187:
URL: 
https://github.com/apache/incubator-devlake/issues/8187#issuecomment-2510736397

   > It has been tested, and the issue of only collecting 100 records has been 
fixed.
   > 
   > However, based on the content of the PR, it still seems to be limited to 
100 pages of data:
   > 
   > ```
   > //incubator-devlake/backend/plugins/gitlab/tasks/deployment_collector.go
   > 
   > apiCollector.InitCollector(helper.ApiCollectorArgs{
   >            RawDataSubTaskArgs: *rawDataSubTaskArgs,
   >            ApiClient:          data.ApiClient,
   >            PageSize:           100,  << here
   >            UrlTemplate:        "projects/{{ .Params.ProjectId 
}}/deployments",
   > ...
   > ```
   > 
   > ```
   > // backend/helpers/pluginhelper/api/api_collector.go
   >            reqDataCopy := RequestData{
   >                    Pager: &Pager{
   >                            Page: i + 1,
   >                            Size: collector.args.PageSize, << here
   >                            Skip: collector.args.PageSize * (i),
   >                    },
   >                    Input:     reqData.Input,
   >                    InputJSON: reqData.InputJSON,
   >            }
   > ```
   > 
   > For example:
   > 
   > I have set up to collect data for one year on DevLake, and the total 
number of records will exceed 10,000. It seems that only 10,000 records will be 
collected?
   > 
   > Is this scenario correct? My assumption is that regardless of the time 
range setting, I must be able to collect all the data.
   > 
   > I'm not sure if I understand this correctly. Please let me know if there 
is any issue.
   
   No, it won't. `Size` is ` PageSize`, it doesn't affect the total number of 
records to be collected as long as the `Skip` grows. No?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@devlake.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Reply via email to