zahorniak opened a new pull request, #9313: URL: https://github.com/apache/devlake/pull/9313
### Summary `extractJobs` and `extractWorkflows` used `api.NewApiExtractor`, which reads all rows of the raw table on every run. This PR changes them to `api.NewStatefulApiExtractor`, the same as the GitHub, GitLab, Jira and YouTrack extractors: - An incremental run reads only the raw rows added since the previous run. - A full sync reads all raw rows, as before. - The extraction logic and the raw params do not change. A job or workflow can have more than one raw row (the "unfinished details" collectors add a new row when its status changes). The rows are read in `id` order, so the newest row wins, as before. On the first run after the upgrade, the state manager starts from the collector state of the same raw table, so the extractor does not do a full sync again. **Tests** - New `TestCircleciJobExtractorIncremental` and `TestCircleciWorkflowExtractorIncremental`: a full run, then an incremental run after a new raw row is added, then a full sync. They check that the incremental run reads only the new row and that the newest row wins. - All CircleCI e2e tests pass on MySQL 8 and PostgreSQL 14. The two new tests fail without the change. **Result on real data** (one instance, 2 CircleCI projects, last nightly run, before #9309 is deployed): | Raw table | All rows (read by full extractor) | Rows since the last collector run (read by incremental extractor) | | --- | --- | --- | | `_raw_circleci_api_jobs`, project A | 7,971,777 | 132,507 | | `_raw_circleci_api_jobs`, project B | 2,451,612 | 37,209 | | `_raw_circleci_api_workflows`, project A | 722,514 | 10,836 | | `_raw_circleci_api_workflows`, project B | 361,614 | 4,782 | The raw params in `_devlake_collector_latest_state` are the same as the extractor params, so the first run after the upgrade is incremental. ### Does this close any open issues? Closes #9305 ### Other Information Follows #9309 (#9304). With both changes, an incremental run in our instance reads about 24k new job rows, not the full raw table (about 10.4M rows). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
