ryo-ebata opened a new pull request, #9148: URL: https://github.com/apache/devlake/pull/9148
### Summary Persist GraphQL collector progress per task, raw scope, nested collector and input so restarting DevLake can resume from the last committed page instead of clearing raw data and starting over. - Commit raw rows, transactional parser cleanup and the input checkpoint together. - Retain completed run markers across the collector/runner completion window. - Keep nested collectors from flushing each other's raw rows. - Anchor incremental watermarks to the original task start; do not skip failed/panicking subtasks or duplicate their rows after restart. - Propagate iterator, cancellation and partial-response failures; reject incompatible or superseded checkpoints. - Stabilize input ordering and remove per-page full raw COUNT scans for pure parsers. - Add disposable MySQL/PostgreSQL SIGKILL tests and CI jobs. ### Does this close any open issues? Closes #9142. ### Validation - Related helper, runner, migration, server, GitHub GraphQL and Linear packages: `go test -race -count=1`. - Reproducible integration suite: `bash backend/test/graphql-resume-crash.sh` and `DEVLAKE_CRASH_DRIVER=postgres bash backend/test/graphql-resume-crash.sh`. - Each database runs 36 top-level checks and 14 SIGKILLs, including two interruptions of the same real Jobs task, exact extracted Job assertions, initialization/page/completion crash windows, rollback and lost-commit-acknowledgement injection, resume-disabled/rerun API behavior, and real PR/Issue/Linear collector recovery. - Additional unit assertions ensure the append-only page path does not run COUNT, and failed commits do not advance the in-memory row count. ### Compatibility and limits The migration adds `_devlake_graphql_collector_states`; task-less callers retain legacy behavior. Known input/code/page-setting changes are rejected and require a new task. Unknown GraphQL partial errors no longer complete a persisted task even when IgnoreQueryErrors is set. Inputs are staged to disk before fetching to detect iterator failure and manifest drift. Raw integrity uses row counts, not content checksums. Arbitrary connection/environment changes and upstream cursor/result-set changes are not snapshot-guaranteed. The subtask progress display remains per-attempt; persisted data/cursors determine what is refetched. No live GitHub/Linear service or production database was used for the integration tests. See `backend/test/graphql-resume-coverage.md` for the full audit matrix and reproduction instructions. Local results do not imply upstream GitHub Actions approval or success. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
