mdelacruz-avantio opened a new issue, #9280: URL: https://github.com/apache/devlake/issues/9280
### Search before asking - [x] I had searched in the issues and found no similar issues. ### What happened `batchFetchDeployments` in `backend/plugins/dora/tasks/change_lead_time_calculator.go` resolves merge commits to deployments in two phases. The second phase joins `cicd_deployment_commits` with `commits_diffs`, loads every joined row into a slice of full `CicdDeploymentCommit` structs with `db.All`, and only then keeps the first row per merge commit in Go. Memory therefore grows with the size of the join, not with the number of merge commits. The join has one row per commit between a production deployment and its previous successful one, for every merge commit in that range. On a project where deployments are far apart in commit history, or where a deployment job records a commit from a second repository with a long history, the join becomes very large. On our instance the query returns about 936,000 rows for 14,088 distinct merge commits (1,693 production deployment commits). The lake container peaks at about 1.7 GiB in this subtask, is OOMKilled at a 2 GiB limit on every nightly run, and then crash loops because each restart resumes the same pipeline at the same subtask. The row count grows every week. ### What do you expect to happen The memory used by this subtask stays proportional to the number of distinct merge commits, as the first phase already does, whatever the size of the `commits_diffs` join. ### How to reproduce 1. A project with a few hundred production deployment commits where consecutive deployments are thousands of commits apart, so `commits_diffs` holds many rows per deployment pair. 2. Run the DORA pipeline and watch the lake container memory during `calculateChangeLeadTime`, or count the rows the second phase loads: ```sql SELECT COUNT(*), COUNT(DISTINCT cd.commit_sha) FROM cicd_deployment_commits dc LEFT JOIN cicd_deployment_commits p ON dc.prev_success_deployment_commit_id = p.id INNER JOIN commits_diffs cd ON cd.new_commit_sha = dc.commit_sha AND cd.old_commit_sha = COALESCE(p.commit_sha, '') LEFT JOIN project_mapping pm ON pm.table = 'cicd_scopes' AND pm.row_id = dc.cicd_scope_id WHERE dc.prev_success_deployment_commit_id <> '' AND dc.environment = 'PRODUCTION' AND dc.result = 'SUCCESS' AND pm.project_name = '<project>'; ``` The first number is what the subtask materialises, the second is what it keeps. ### Anything else Streaming the second phase with `db.Cursor` and `db.Fetch` into a three-column struct and keeping the first row per merge commit, with the same joins, filters and ordering, makes the memory independent of the join size. I will open a pull request with that change and an e2e case for the first-wins tie-break. Raising the container memory limit and setting `GOMEMLIMIT` is a stopgap only, the peak keeps growing with the join. ### Version v1.0.3-beta18. The function is unchanged on `main` at 625e512a4. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
