jbsmith7741 opened a new pull request, #8956:
URL: https://github.com/apache/devlake/pull/8956
### Summary
Two bug fixes for the CircleCI workflow and job collectors:
**1. Skip HTTP 500 responses from corrupt CircleCI Server records**
CircleCI Server occasionally returns `500 Internal Server Error` for
pipelines whose internal records are corrupt or stuck (e.g. a pipeline exists
in the DB but its `/workflow` endpoint errors). Previously, this caused
`ignoreDeletedBuilds` to fall through and abort the entire collector subtask.
Both the workflow and job collectors now use a renamed
`ignoreDeletedOrBrokenBuilds` hook that treats `500` the same as `404`
(data-retention deletion) — a non-fatal, per-item skip so collection continues
past broken entries.
```
Before After
─────────────────────────────────────────
─────────────────────────────────────────
HTTP 404 → ErrIgnoreAndContinue (skip) HTTP 404 → ErrIgnoreAndContinue
(skip)
HTTP 500 → error propagated, run aborts HTTP 500 → ErrIgnoreAndContinue
(skip)
```
**2. Apply `SyncPolicy.TimeAfter` window on full sync**
The DB iterator clause that limits which pipelines/workflows trigger
downstream API calls previously only activated when `isIncremental == true`.
This meant a full sync with a `TimeAfter` policy still iterated every
historical tool-layer row and issued an API request for each one. The condition
is now `if createdAfter != nil`, so both incremental and windowed full syncs
skip rows outside the time window.
The boundary condition also changed from `>` to `>=` so the exact bookmark
timestamp is not silently dropped.
```
Before After
─────────────────────────────────────────
─────────────────────────────────────────
Full sync + TimeAfter → all rows iterated, Full sync + TimeAfter →
rows before
API call per row regardless of age window skipped, boundary
inclusive
Incremental → rows after bookmark Incremental →
unchanged
(exclusive)
```
### Does this close any open issues?
Closes #8948
### Screenshots
N/A — no UI changes.
### Other Information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
