mdelacruz-avantio opened a new pull request, #9281: URL: https://github.com/apache/devlake/pull/9281
### Summary `batchFetchDeployments` loaded the whole `cicd_deployment_commits` x `commits_diffs` join into a slice of full `CicdDeploymentCommit` rows before keeping the first row per merge commit. The join has one row per commit between consecutive production deployments for every merge commit in that range, so its size has no relation to the result. On a project with 14k merge commits it returned 936k rows and the lake container was OOMKilled on every nightly run. | Item | Change | | --- | --- | | Phase 2 query | Read through `db.Cursor` and `db.Fetch` one row at a time into a struct with `id`, `finished_date` and `merge_sha`, keeping the first row per merge commit. Joins, filters and `ORDER BY cd.commit_sha, dc.started_date ASC, dc.id ASC` are unchanged, so the first row seen is still the earliest deployment | | Phase 1 query | Selects `dc.id, dc.finished_date, dc.commit_sha` instead of `dc.*`, the fields the conversion reads | | Error handling | `cursor.Err()` is checked after the loop, so a failed `Next` returns an error instead of a silently truncated map | | Map contract | The function comment states that map values carry only `Id` and `FinishedDate` | | e2e | New pull request `pr8` whose merge commit is in the diff range of two production deployments, where the deployment with the lower id started later. The expected deployment is the earlier-started one, which an order-by-id or last-wins implementation gets wrong | Memory of the second phase on the instance above goes from about 1.6 GiB to a few MiB. The database work is the same as before. ### Does this close any open issues? Closes #9280 ### Screenshots No UI change. ### Other Information - Results are identical: same query, same ordering, same first-wins rule, phase 1 still takes precedence over phase 2. The dora e2e passes on MySQL 8.4 and PostgreSQL 18.4 with the existing fixtures and the new case. - No schema change, no migration, no option change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
