zahorniak opened a new pull request, #9309: URL: https://github.com/apache/devlake/pull/9309
### Summary The "unfinished details" step of `collectJobs` and `collectWorkflows` selected every workflow in a non-terminal status, without an age limit. A workflow with an approval job that nobody approves stays `on_hold` for ever, so DevLake collected it again on every run. Because the step appends to the raw tables, `_raw_circleci_api_jobs` grew on every run. This PR selects only workflows created in the last 30 days: - `UnfinishedJobsInputClauses` joins `_tool_circleci_workflows` and filters on the workflow `created_date`. Jobs have no created date of their own (`created_date` is empty for all `on_hold` jobs in our instance). - New `UnfinishedWorkflowsInputClauses` applies the same filter to the workflow collector. - The window is the constant `UnfinishedDetailsLookback` (30 days). The date is computed in Go, so the query works on MySQL and PostgreSQL. A workflow that gets its approval within 30 days is still collected again, so its deploy jobs after the approval are not lost. In our instance, all approvals except 1 of about 1,860 came within 30 days. **Tests** - `TestCircleciUnfinishedJobsInputIterator` (updated) and `TestCircleciUnfinishedWorkflowsInputIterator` (new) seed a workflow older than 30 days and check that it is not selected. Both tests pass on MySQL 8 and PostgreSQL 14, and fail without the change. **Result on real data** (one instance, 2 CircleCI projects, same query as the collectors): | | Before | After | | --- | --- | --- | | Workflows selected by `collectJobs` unfinished details | 14,864 | 1,851 | | Job rows added to `_raw_circleci_api_jobs` per run | 166,485 | 24,119 | | Workflows selected by `collectWorkflows` unfinished details | 15,168 | 1,858 | ### Does this close any open issues? Closes #9304 ### Other Information - #9305 (stateful CircleCI extractors) is the second part. After both changes, an incremental run reads only the new raw rows. - Old duplicate raw rows stay until the next full sync. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
