klesh opened a new issue, #6220: URL: https://github.com/apache/incubator-devlake/issues/6220
## What and why to refactor Currently, we have 2 types of ApiCollectors - `ApiCollector` - Stateless API collection helper - `ApiCollectorStateManager` - Stateful API collection helper on top of `ApiCollector` which tracks the `LastSuceededTime` Stateless` and `Stateful`) work in 4 modes - `FullSync` from `ApiCollector`: Note that it means `ApiCollector` doesn't offer any help regarding Incremental collection, and the developer must implement the feature on his/her own. - `IncrementalSync by updatedAt` from `ApiCollectorStateManager`: offers aids when API supports filtering records by the `updatedAt` field. - `IncrementalSync by createdAt` from `ApiCollectorStateManager`: offers aids when API supports filtering records by the `createdAt` or returned records are sorted by `createdAt`. **Useful for collecting Pipelines/Jobs** that can be Finalized(Can not be re-opened) and Automated Short-lived entities (no human operation involved and it will be closed within days) - `IncrementalSync by createdAt plus refresh unfinished records` from `ApiCollectorStateManager`: offers refreshing unfished records based on the previous mode. **Useful for collecting PR while the API doesn't support filtering by `updatedAt`** ## Describe the solution you'd like How to refactor? ## Related issues Please link any other ## Additional context Add any other context or screenshots about the feature request here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
