klesh opened a new issue, #4535: URL: https://github.com/apache/incubator-devlake/issues/4535
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar feature requirement. ### Use case Increase the collecting speed Improve the data consistency ### Description We are pretty limited by the Data Source API Capability in terms of Incremental Collection / Time Filter supports, ideally, API should offer a `query argument` for filtering records by `updated time`, however, the reality is cruel and many of them do NOT offer such a capability, especially for those less used. During the development of https://github.com/merico-dev/lake/blob/6a0dc70abb214abd108b96b7f2de8922fe165f6c/backend/helpers/pluginhelper/api/api_collector_with_state.go#L141, I came up with a new Collector Helper named `NewStatefulApiCollectorForFinalizableEntity` which could widen the spectrum of the support, I suggest that we should go through all existing collectors and see if it can be applied. Note that it is NOT without limitation. ### Related issues #4403 ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
