Thundercloud12 opened a new pull request, #9162:
URL: https://github.com/apache/devlake/pull/9162
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
## Summary
### Problem
In `cicd_run_collector.go`, the collector explicitly skipped workflow runs
that were not completed:
```go
if run.Status != "completed" {
continue
}
```
When runs were skipped while `in_progress`, `CollectorStateManager.Close()`
still advanced `LatestSuccessStart` to `until`.
Because GitHub's `/actions/runs` API only filters by **created date**
(`created=<from>..<to>`) and does not support filtering by `updated`,
subsequent incremental syncs queried:
```text
created=LatestSuccessStart+1s..until
```
Any long-running workflow that was created before `LatestSuccessStart` but
completed later was therefore permanently missed.
### Solution: Low-Water-Mark Lookback
1. **Persist unfinished runs**
Removed the `run.Status != "completed"` skip in `ResponseParser` so
non-terminal runs (`in_progress`, `queued`, `waiting`, etc.) are written to
`_raw_github_api_runs` and extracted into `_tool_github_runs`.
2. **Dynamic query start adjustment**
In `CollectRuns`, before bisecting leaf windows during incremental syncs,
query `_tool_github_runs` for the oldest unfinished run:
```sql
SELECT MIN(github_created_at)
FROM _tool_github_runs
WHERE connection_id = ?
AND repo_id = ?
AND status != 'completed'
AND github_created_at >= ?
```
If an unfinished run is found and its `github_created_at` predates
`windowStart`, `windowStart` is moved back to re-include it in the GitHub API
query range.
3. **Preserve high-water mark**
`CollectorStateManager.Close()` continues to advance `LatestSuccessStart
= until`.
The shared database watermark therefore moves forward normally, and as
soon as all unfinished runs complete, `windowStart` automatically snaps back to
`LatestSuccessStart + 1s`.
4. **Zombie run guardrail**
Added:
```go
maxUnfinishedRunLookback = 14 * 24 * time.Hour
```
This ensures abandoned or killed runs cannot hold the collection window
back indefinitely.
5. **Downstream compatibility**
Downstream tasks (`cicd_run_extractor`, `cicd_run_convertor`, and
`cicd_job_collector`) already support primary-key upserts and
`github_updated_at >= since`.
Therefore, completed runs and their jobs are converted and extracted
automatically once the workflow status becomes `completed`.
## Does this close any open issues?
Closes #9125
## Screenshots
N/A (Backend collector logic fix)
## Other Information
* Unit tests added in `cicd_run_collector_test.go`:
* `TestCicdRunLoadOldestUnfinishedRunCreatedAt_ReturnsOldestTimestamp`
* `TestCicdRunLoadOldestUnfinishedRunCreatedAt_NotFoundReturnsNil`
* `TestCicdRunLoadOldestUnfinishedRunCreatedAt_PropagatesError`
* All unit tests in `plugins/github/tasks` pass.
* **Note for existing deployments:** Workflows that were skipped in previous
DevLake versions were dropped before reaching `_tool_github_runs`. A one-time
backfill with `TimeAfter` covering the affected window will be needed to
capture those older runs. After that, the low-water-mark mechanism will track
everything automatically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
