klesh commented on issue #9127: URL: https://github.com/apache/devlake/issues/9127#issuecomment-5761466220
> [@klesh](https://github.com/klesh) hey! so I’ve gone through the discussion thread as well as this issue. > > The main problem seems to be this check in `cicd_run_collector.go`: > > if run.Status != "completed" { > continue > } > This drops `in_progress` runs, while `CollectorStateManager.Close()` advances `LatestSuccessStart` to `until`, so those runs may never be revisited. > > I see two possible approaches: > > 1. **Low-water-mark:** persist unfinished runs and, before each sync, query the oldest unfinished run and move `windowStart` back to it. This keeps the shared watermark untouched. > 2. **Recheck by run ID:** since `GithubRun` is already keyed by `ConnectionId + RepoId + ID` and GitHub supports fetching a specific workflow run by `run_id`, we could keep the normal incremental window and only recheck previously `in_progress` runs by ID before continuing with the new window. > > The second approach seems worth considering since it avoids re-fetching the entire historical window just to update a small number of unfinished runs. Good work. I think #1 is a good idea! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
