cec-lan opened a new issue, #9421:
URL: https://github.com/apache/devlake/issues/9421

   ### Search before asking
   
   - [x] I had searched in the 
[issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and 
found no similar issues.
   
   
   ### What happened
   
   In the **github_graphql** plugin, **Extract Jobs** 
(**backend/plugins/github_graphql/tasks/job_extractor.go**) is built with 
**api.NewApiExtractor**, which — per its own doc comment — deletes and re-saves 
all matching rows of the raw table on every single run, regardless of whether 
anything changed.
   
   This means every day's collection re-processes the entire historical raw 
dataset for a repo's CI jobs, not just what changed since the last run. In our 
production instance (one repo, ~2 years of CI history), this subtask alone 
reads and rewrites ~103,000 rows every day and takes ~9 minutes by itself.
   
   It also defeats the incremental logic of the downstream **Convert Jobs** 
subtask, which does use the newer **api.StatefulDataConverter** and filters on 
**_tool_github_jobs.updated_at >= since**. Because **Extract Jobs** 
deletes+re-inserts every row on every run, **updated_at** gets reset to "now" 
for the entire table each day, so **Convert Jobs**'s incremental filter ends up 
matching almost the whole table anyway (we observed ~103,000 of ~103,360 rows 
processed again). That subtask takes another ~8 minutes.
   
   Together these two subtasks account for the majority of our daily collection 
pipeline's runtime, and the cost grows every day as the repo's CI history grows 
(measured growing from ~47 min to ~77 min total pipeline time over an 11-day 
window).
   
   For comparison: the plain REST-based github plugin's equivalent extractor 
(cicd_job_extractor.go) already uses api.NewStatefulApiExtractor, and 
github_graphql's own cicd_run_extractor.go (Extract Workflow Runs) was already 
migrated in #8858 — job_extractor.go appears to be the one left behind.
   
   ### What do you expect to happen
   
   **Extract Jobs** should use **api.NewStatefulApiExtractor** (mirroring 
**backend/plugins/github/tasks/cicd_job_extractor.go**), so incremental runs 
only process rows added/changed since the last successful run.
   
   ### How to reproduce
   
   1. Connect a GitHub repo with a non-trivial CI history (tens of thousands of 
workflow runs/jobs) to **github_graphql**.
   2. Run the collection pipeline for this scope on two consecutive days (not 
full-sync).
   3. **Extract Jobs** log line get data from **_raw_github_graphql_jobs where 
params=... and got N** — N stays close to the full historical count both days.
   4. **Convert Jobs**'s finished records counter also climbs to roughly the 
same total each day.
   
   ### Anything else
   
   - This happens on every scheduled run (daily, in our case), consistently — 
not intermittent, not condition-dependent.
   - Same class of bug as #9304 / #9305, already found and fixed for the 
CircleCI plugin in #9313 ("an incremental run... reads about 24k new job rows, 
not the full raw table (about 10.4M rows)"). This issue reports the same 
pattern in the github_graphql plugin, which wasn't covered by that fix.
   - We cross-checked this against github_graphql's own subtask registry 
(backend/plugins/github_graphql/impl/impl.go, SubTaskMetas()). Several subtasks 
in this plugin (Extract Workflow Runs, Extract Comments, Extract PR Review 
Comments, and almost all Convert* subtasks including Convert Jobs) are actually 
reused from the REST-based github plugin's own package, and those were already 
migrated to the stateful pattern by #8858. The ones that are genuinely 
github_graphql's own extractors — issue_extractor.go, pr_extractor.go, 
job_extractor.go, account_extractor.go, deployment_extractor.go, 
release_extractor.go — all still use api.NewApiExtractor. So job_extractor.go 
is the one causing the most measurable impact for us, but it looks like the 
whole plugin's own extractor layer was out of scope for #8858, not just this 
one file. Maybe the other five should be included in the same fix.
   
   
   
   ### Version
   
   main (also present in v1.0.3-beta8)
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to