Nickcw6 commented on issue #7750: URL: https://github.com/apache/incubator-devlake/issues/7750#issuecomment-2233199234
Hey @klesh, I think I've figured out what's happening here. I've updated the original post. Only the first page of the `Get all pipelines` CircleCI API response is being collected - ie. just 20 pipelines total, which has a knock-on effect when the task attempts to collect the workflows & jobs. This is consistent with always seeing 20 rows in the `_raw_circleci_api_pipelines` table, and explains the inconsistent date range behaviour I was originally seeing. I think the issue is on [this line](https://github.com/apache/incubator-devlake/blob/main/backend/plugins/circleci/tasks/pipeline_collector.go#L64) in the pipeline collector - it's setting the query param as `page_token` but according to the [API docs](https://circleci.com/docs/api/v2/index.html#operation/listPipelinesForProject) it should be `page-token`. This is also the same for the workflow collector [here](https://github.com/apache/incubator-devlake/blob/main/backend/plugins/circleci/tasks/workflow_collector.go#L84), and the [job collector](https://github.com/apache/incubator-devlake/blob/main/backend/plugins/circleci/tasks/job_collector.go#L84). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
