tigerquoll commented on PR #1061: URL: https://github.com/apache/yunikorn-k8shim/pull/1061#issuecomment-5267515996
> I prefer to do this as "one time" activity using appropriate tools as this is something not required to run every time as we don't introduce leaks in every change that we make. I would prefer running at the end of the "code commit" process like codecov etc to ensure there is no alarming signal, probably somewhere after or before the e2e tests or somewhere through a new option in `make` command. I had come across https://www.gopherguides.com/articles/golang-goroutine-leak-profile, a new feature introduced in go1.2.7 to find the leaks. This can be hooked up into anywhere in the build process. > > Please share your thoughts. The pprof leak profile and goleak actually detect different classes of leak, so I'd like to use both rather than pick one. The runtime profile only reports goroutines blocked on sync primitives that no runnable goroutine can reach - by design it doesn't flag running loops, sleeping goroutines, or goroutines blocked on reachable channels (the article's own limitations section). All four of the production defects this PR surfaced are in that second category (an unstopped wait.Until loop, event loops on package-singleton channels, a timer-backoff retry), so a build-stage profile pass would have reported nothing here. Goleak's test-exit check is what catches lifecycle bugs like these, and it attributes the failure to the exact package and PR that introduces it, at roughly zero cost while green. Where the profile is genuinely stronger is live processes - which is exactly the gap this PR documents: the e2e suites are deliberately not goleak-instrumented. We're on Go 1.26, so it's available behind GOEXPERIMENT=goroutineleakprofile (per the article, on by default from 1.27) - the one wrinkle is that's a build-time flag, so the scheduler image used in e2e needs building with it until we're on 1.27. I'd be happy to file a follow-up to hook /debug/pprof/goroutineleak into the e2e/nightly stage as you describe - the combination covers both leak classes at both detection points: goleak catches unstopped-service bugs at the PR that introduces them, the runtime profile catches provably-stuck goroutines in real integrated runs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
