[
https://issues.apache.org/jira/browse/YUNIKORN-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dale Richardson updated YUNIKORN-3367:
--------------------------------------
Description:
Follow-up to YUNIKORN-3357 (shim goleak adoption, PR #1061); burns down the
leakcheck exemption k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext
(pkg/shim).
{{KubernetesShim.doScheduling}} starts two {{wait.Until}} goroutines
({{schedule}} and {{checkOutstandingApps}}) that share the unbuffered
{{ss.stopChan}}. {{KubernetesShim.Stop()}} *sends a single value* to
{{stopChan}} instead of closing it, so exactly one of the two loops wakes and
returns; the other keeps running for the lifetime of the process, scheduling
applications on a shim that has been told to stop. Which loop survives is a
race (both park in the same Kubernetes backoff frame).
Proposed fix: {{Stop()}} should {{close(stopChan)}} — which stops both loops —
rather than send to it, then delete the exemption in
{{pkg/common/leakcheck/leakcheck.go}}.
Note for whoever triages: that exemption is the broadest in the file. Its top
frame is a {{k8s.io/apimachinery}} implementation detail, so within pkg/shim it
exempts every {{wait.Until}}/{{wait.Forever}}/{{wait.JitterUntil}} loop,
including ones not yet written; and the frame name has already moved once
({{BackoffUntil}} -> {{BackoffUntilWithContext}}), so a dependency bump can
silently break the match and redden pkg/shim. Fixing the bug removes the need
for it. Shares the {{Stop()}} path with the failed-Run() cleanup finding below
and can be fixed together.
was:
Follow-up to YUNIKORN-3357 (shim goleak adoption, PR #1061); burns down the
leakcheck exemption
k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext (pkg/shim).
{{KubernetesShim.doScheduling}} starts two {{wait.Until}} goroutines
({{schedule}} and {{checkOutstandingApps}}) that share the unbuffered
{{ss.stopChan}}. {{KubernetesShim.Stop()}} *sends a single value* to
{{stopChan}} instead of closing it, so exactly one of the two loops wakes and
returns; the other keeps running for the lifetime of the process, scheduling
applications on a shim that has been told to stop. Which loop survives is a
race (both park in the same Kubernetes backoff frame).
Proposed fix: {{Stop()}} should {{close(stopChan)}} — which stops both loops —
rather than send to it, then delete the exemption in
{{pkg/common/leakcheck/leakcheck.go}}.
Note for whoever triages: that exemption is the broadest in the file. Its top
frame is a {{k8s.io/apimachinery}} implementation detail, so within pkg/shim it
exempts every {{wait.Until}}/{{wait.Forever}}/{{wait.JitterUntil}} loop,
including ones not yet written; and the frame name has already moved once
({{BackoffUntil}} -> {{BackoffUntilWithContext}}), so a dependency bump can
silently break the match and redden pkg/shim. Fixing the bug removes the need
for it. Shares the {{Stop()}} path with the failed-Run() cleanup finding below
and can be fixed together.
> KubernetesShim.Stop() sends to the unbuffered stopChan instead of closing it,
> leaving one doScheduling loop running after stop
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: YUNIKORN-3367
> URL: https://issues.apache.org/jira/browse/YUNIKORN-3367
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: shim - kubernetes
> Reporter: Dale Richardson
> Priority: Major
> Labels: newbie
>
> Follow-up to YUNIKORN-3357 (shim goleak adoption, PR #1061); burns down the
> leakcheck exemption k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext
> (pkg/shim).
> {{KubernetesShim.doScheduling}} starts two {{wait.Until}} goroutines
> ({{schedule}} and {{checkOutstandingApps}}) that share the unbuffered
> {{ss.stopChan}}. {{KubernetesShim.Stop()}} *sends a single value* to
> {{stopChan}} instead of closing it, so exactly one of the two loops wakes and
> returns; the other keeps running for the lifetime of the process, scheduling
> applications on a shim that has been told to stop. Which loop survives is a
> race (both park in the same Kubernetes backoff frame).
> Proposed fix: {{Stop()}} should {{close(stopChan)}} — which stops both loops
> — rather than send to it, then delete the exemption in
> {{pkg/common/leakcheck/leakcheck.go}}.
> Note for whoever triages: that exemption is the broadest in the file. Its top
> frame is a {{k8s.io/apimachinery}} implementation detail, so within pkg/shim
> it exempts every {{wait.Until}}/{{wait.Forever}}/{{wait.JitterUntil}} loop,
> including ones not yet written; and the frame name has already moved once
> ({{BackoffUntil}} -> {{BackoffUntilWithContext}}), so a dependency bump can
> silently break the match and redden pkg/shim. Fixing the bug removes the need
> for it. Shares the {{Stop()}} path with the failed-Run() cleanup finding
> below and can be fixed together.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
