dloszewski-cni commented on PR #14401: URL: https://github.com/apache/cloudstack/pull/14401#issuecomment-6426714144
Thanks @Damans227 for picking this up, and @DaanHoogland for the ping — sorry for the slow reply. On the automation question (Oct 7): no hand-written harness. The race is driven by the CloudStack CSI driver in a Kubernetes cluster — when a detach exceeds the driver’s ~300s timeout it re-submits, so several DetachVolume jobs for the same volume stack up on that VM’s work queue. Pods rescheduling across nodes then supply the attach-elsewhere step. We hit it four times in 20 days on 4.22.1.0 with KVM + Ceph RBD. The prerequisite is a slow per-VM work queue; in our case agent-side libvirt stats calls were pushing individual detaches to 3–6 minutes. On #14401: the execution-time re-check is right, and it matches what we wrote. I think it needs a second piece to close the race rather than narrow it. The check reads volume.getInstanceId() at the top of detachVolumeFromVm, but the row is cleared after the agent round-trip returns. In our environment that round-trip ran 2–3 minutes. A newer attach landing inside that window still gets its attachment cleared — which is the exact sequence behind the occurrences we reported. We made the final clear conditional as well: a VolumeDao.detachVolume(volumeId, vmId) that puts both id and instance_id in the search criteria and returns whether a row actually changed, with the caller logging and re-reading when it didn’t. That’s a compare-and-swap, and it mirrors what the attach path already does with checkAndSetAttaching. I have that as a branch against 4.22 with unit tests for both halves — the DAO-level conditional update, and the service-level skip asserting no agent round-trip or stats collection happens. Happy to open it as a PR against this one, or hand the DAO change to @Damans227 to fold in, whichever is less friction. Still open: the attach-side case (occurrence 4 in the issue) isn’t addressed by either patch, and we never root-caused it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
