dosubot[bot] commented on issue #7797:
URL:
https://github.com/apache/incubator-devlake/issues/7797#issuecomment-2258718134
<!-- Answer -->
To address the issue of CircleCI pipelines being collected from before the
specified time range, you can implement a check in the `CollectPipelines`
function to filter out pipelines based on their `created_at` property. This
will ensure that only pipelines created within the specified time range are
collected.
Here's a step-by-step guide on how to implement this:
1. **Modify the `CollectPipelines` function**:
- Add a check to filter pipelines based on the `created_at` property
before storing them in the database.
2. **Update the `CollectPipelines` function**:
- Add a parameter to pass the start date of the time range.
- Filter the pipelines based on the `created_at` property.
Here is an example of how you can modify the `CollectPipelines` function:
```go
func CollectPipelines(taskCtx plugin.SubTaskContext) errors.Error {
rawDataSubTaskArgs, data := CreateRawDataSubTaskArgs(taskCtx,
RAW_PIPELINE_TABLE)
logger := taskCtx.GetLogger()
logger.Info("collect pipelines")
// Define the start date for the time range filter
startDate := data.Options.TimeAfter
collector, err := api.NewApiCollector(api.ApiCollectorArgs{
RawDataSubTaskArgs: *rawDataSubTaskArgs,
ApiClient: data.ApiClient,
UrlTemplate: "/v2/project/{{ .Params.ProjectSlug }}/pipeline",
PageSize: int(data.Options.PageSize),
GetNextPageCustomData: func(prevReqData *api.RequestData,
prevPageResponse *http.Response) (interface{}, errors.Error) {
res := CircleciPageTokenResp[any]{}
err := api.UnmarshalResponse(prevPageResponse, &res)
if err != nil {
return nil, err
}
if res.NextPageToken == "" {
return nil, api.ErrFinishCollect
}
return res.NextPageToken, nil
},
Query: func(reqData *api.RequestData) (url.Values, errors.Error) {
query := url.Values{}
if pageToken, ok := reqData.CustomData.(string); ok && pageToken
!= "" {
query.Set("page_token", reqData.CustomData.(string))
}
return query, nil
},
ResponseParser: func(res *http.Response) ([]json.RawMessage,
errors.Error) {
data := CircleciPageTokenResp[[]json.RawMessage]{}
err := api.UnmarshalResponse(res, &data)
if err != nil {
return nil, err
}
// Filter pipelines based on the created_at property
filteredItems := []json.RawMessage{}
for _, item := range data.Items {
var pipeline map[string]interface{}
if err := json.Unmarshal(item, &pipeline); err != nil {
return nil, errors.Convert(err)
}
createdAtStr, ok := pipeline["created_at"].(string)
if !ok {
continue
}
createdAt, err := time.Parse(time.RFC3339, createdAtStr)
if err != nil {
return nil, errors.Convert(err)
}
if createdAt.After(startDate) {
filteredItems = append(filteredItems, item)
}
}
return filteredItems, nil
},
})
if err != nil {
logger.Error(err, "collect pipelines error")
return err
}
return collector.Execute()
}
```
In this example:
- The `startDate` is retrieved from the `data.Options.TimeAfter` parameter.
- The `ResponseParser` function filters the pipelines based on the
`created_at` property, ensuring that only pipelines created after the
`startDate` are collected.
This approach ensures that the CircleCI pipelines collected are within the
specified time range in the sync policy, preventing the collection of pipelines
created before the specified date.
For more details on configuring the CircleCI plugin and setting the sync
policy, you can refer to the [CircleCI configuration
documentation](https://github.com/apache/incubator-devlake-website/blob/main/docs/Configuration/CircleCI.md)
<sup>[[1]](https://github.com/apache/incubator-devlake-website/blob/main/docs/Configuration/CircleCI.md)</sup><sup>[[2]](https://github.com/apache/incubator-devlake-website/blob/main/versioned_docs/version-v1.0/Configuration/CircleCI.md)</sup>.
<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@devlake.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
