cec-lan opened a new issue, #9421: URL: https://github.com/apache/devlake/issues/9421
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar issues. ### What happened In the **github_graphql** plugin, **Extract Jobs** (**backend/plugins/github_graphql/tasks/job_extractor.go**) is built with **api.NewApiExtractor**, which — per its own doc comment — deletes and re-saves all matching rows of the raw table on every single run, regardless of whether anything changed. This means every day's collection re-processes the entire historical raw dataset for a repo's CI jobs, not just what changed since the last run. In our production instance (one repo, ~2 years of CI history), this subtask alone reads and rewrites ~103,000 rows every day and takes ~9 minutes by itself. It also defeats the incremental logic of the downstream **Convert Jobs** subtask, which does use the newer **api.StatefulDataConverter** and filters on **_tool_github_jobs.updated_at >= since**. Because **Extract Jobs** deletes+re-inserts every row on every run, **updated_at** gets reset to "now" for the entire table each day, so **Convert Jobs**'s incremental filter ends up matching almost the whole table anyway (we observed ~103,000 of ~103,360 rows processed again). That subtask takes another ~8 minutes. Together these two subtasks account for the majority of our daily collection pipeline's runtime, and the cost grows every day as the repo's CI history grows (measured growing from ~47 min to ~77 min total pipeline time over an 11-day window). For comparison: the plain REST-based github plugin's equivalent extractor (cicd_job_extractor.go) already uses api.NewStatefulApiExtractor, and github_graphql's own cicd_run_extractor.go (Extract Workflow Runs) was already migrated in #8858 — job_extractor.go appears to be the one left behind. ### What do you expect to happen **Extract Jobs** should use **api.NewStatefulApiExtractor** (mirroring **backend/plugins/github/tasks/cicd_job_extractor.go**), so incremental runs only process rows added/changed since the last successful run. ### How to reproduce 1. Connect a GitHub repo with a non-trivial CI history (tens of thousands of workflow runs/jobs) to **github_graphql**. 2. Run the collection pipeline for this scope on two consecutive days (not full-sync). 3. **Extract Jobs** log line get data from **_raw_github_graphql_jobs where params=... and got N** — N stays close to the full historical count both days. 4. **Convert Jobs**'s finished records counter also climbs to roughly the same total each day. ### Anything else - This happens on every scheduled run (daily, in our case), consistently — not intermittent, not condition-dependent. - Same class of bug as #9304 / #9305, already found and fixed for the CircleCI plugin in #9313 ("an incremental run... reads about 24k new job rows, not the full raw table (about 10.4M rows)"). This issue reports the same pattern in the github_graphql plugin, which wasn't covered by that fix. - We cross-checked this against github_graphql's own subtask registry (backend/plugins/github_graphql/impl/impl.go, SubTaskMetas()). Several subtasks in this plugin (Extract Workflow Runs, Extract Comments, Extract PR Review Comments, and almost all Convert* subtasks including Convert Jobs) are actually reused from the REST-based github plugin's own package, and those were already migrated to the stateful pattern by #8858. The ones that are genuinely github_graphql's own extractors — issue_extractor.go, pr_extractor.go, job_extractor.go, account_extractor.go, deployment_extractor.go, release_extractor.go — all still use api.NewApiExtractor. So job_extractor.go is the one causing the most measurable impact for us, but it looks like the whole plugin's own extractor layer was out of scope for #8858, not just this one file. Maybe the other five should be included in the same fix. ### Version main (also present in v1.0.3-beta8) ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
