ahnazary opened a new issue, #9246: URL: https://github.com/apache/devlake/issues/9246
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar feature requests. ### Use case A project is first collected with `timeAfter = T1`. Later the user moves `timeAfter` to an earlier date `T0 < T1` to get more history. Today this restarts the collection from scratch: - `NewCollectorStateManager` (`backend/helpers/pluginhelper/api/collector_state_manager.go`) sees `syncPolicy.TimeAfter.Before(state.TimeAfter)` and drops to full-sync mode: `since = T0`, `isIncremental = false`. - Every collector then fetches `[T0, now]`, which includes `[T1, now]`, the period that was already collected. - The REST and GraphQL collectors (`api_collector.go`, `graphql_collector.go`: "flush data if not incremental collection") delete all existing raw rows for the scope before fetching. For large scopes the repeated fetch is the expensive part. GitHub workflow runs and jobs dominate API calls and run time, and the rate limit is shared by every pipeline using the same App or token. A full re-collection also leaves the scope with fewer rows until it finishes. ### Description Add a third collection mode, "backfill", used when the new `timeAfter` is earlier than the stored one: 1. Collect only `[T0, T1]`: `since = T0`, with an upper bound `until = state.TimeAfter`. 2. Do not flush the existing raw data. 3. On success, store `T0` as the new `TimeAfter` (the existing code already updates it for full syncs). 4. Apply the same rule in `SubtaskStateManager`, which duplicates the logic. Collectors need to honour the upper bound. Today only a few call `GetUntil()` (in the GitHub plugin, only the workflow runs collector); the rest use `GetSince()` alone. A backfill would add an `until` filter to each collector's request or stop condition, and fetch child records (reviews, comments, jobs) only for the newly found parents. Collectors that cannot bound the range could keep today's behaviour (a full re-collect), so this can ship incrementally, starting with the state managers and one or two collectors. ### Related issues None found. ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
