kkc-tonywu commented on issue #8187:
URL: 
https://github.com/apache/incubator-devlake/issues/8187#issuecomment-2505197433

   It has been tested, and the issue of only collecting 100 records has been 
fixed.
   
   However, based on the content of the PR, it still seems to be limited to 100 
pages of data:
   ```
   //incubator-devlake/backend/plugins/gitlab/tasks
   /deployment_collector.go
   
   apiCollector.InitCollector(helper.ApiCollectorArgs{
                RawDataSubTaskArgs: *rawDataSubTaskArgs,
                ApiClient:          data.ApiClient,
                PageSize:           100,  << here
                UrlTemplate:        "projects/{{ .Params.ProjectId 
}}/deployments",
   ...
   ```
   
   ```
   // backend/helpers/pluginhelper/api/api_collector.go
                reqDataCopy := RequestData{
                        Pager: &Pager{
                                Page: i + 1,
                                Size: collector.args.PageSize, << here
                                Skip: collector.args.PageSize * (i),
                        },
                        Input:     reqData.Input,
                        InputJSON: reqData.InputJSON,
                }
   ```
   For example:
   
   I have set up to collect data for one year on DevLake, and the total number 
of records will exceed 10,000. It seems that only 10,000 records will be 
collected?
   
   Is this scenario correct? My assumption is that regardless of the time range 
setting, I must be able to collect all the data.
   
   I'm not sure if I understand this correctly. Please let me know if there is 
any issue.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@devlake.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Reply via email to