bhouse-nexthop commented on PR #14189:
URL: https://github.com/apache/cloudstack/pull/14189#issuecomment-6733520062
### Test report: storage pool refcount and the false umount success, on a
KVM zone
**Environment**
- A CloudStack 24.0 (main) build with this PR applied, alongside some of my
other open PRs.
- Three KVM hosts on Ubuntu 24.04, nested virtualisation; Ceph RBD primary
storage (zone-wide); NFS secondary storage.
- 25 small qcow2 templates registered for this test, each used once, so
every deploy below is a template's first copy to primary storage. That copy
mounts the template's directory from secondary storage under `/mnt/<uuid>`, and
unmounts it when the copy is done.
- On a zone-wide pool, the management server can send the copy to any host,
whichever host the VM goes to. So every check below covers all three hosts.
**1. A busy mount is reported as a failed unmount**
- Each host ran a watcher that `cd`s into any new secondary mount under
`/mnt` and holds it for 15 minutes.
- I deployed one VM from a fresh template.
- The deploy succeeded. The agent that ran the copy logged:
```
ERROR deleteStoragePool removed pool from libvirt, but libvirt had trouble
unmounting the pool. Trying umount location /mnt/<uuid> again in a few seconds
WARN ... sleep 5 && umount /mnt/<uuid> ... umount.nfs4: /mnt/<uuid>:
device is busy
ERROR Failed to unmount /mnt/<uuid>, it is still a mount point
```
- There was no `Succeeded in unmounting` line. The busy mount was reported
as a failure, and the command still finished.
- The mount stays until something unmounts it; I unmounted it by hand once
the watcher was gone. The description says that is out of scope.
- Without the PR (from the description's production logs): `Succeeded in
unmounting`, followed about 28 ms later by an ERROR `device is busy`.
**2. Parallel first copies: no busy unmounts, no leaked mounts**
Two rounds of parallel deploys, each from its own fresh template: 8 in the
first round and 7 in the second. With nothing holding the mounts:
| Round | Deploys Running | Copies (create / remove pool), by host | Removed
cleanly | `had trouble unmounting` / `device is busy` | Mounts under `/mnt`
after |
|---|---|---|---|---|---|
| A | 8 / 8 | 8 / 8: 5, 1 and 2 | 8 | 0 / 0 | 0 new |
| B | 7 / 7 | 7 / 7: 2, 4 and 1 | 7 | 0 / 0 | 0 new |
- Every copy's pool was created, used and removed from libvirt, with no
fallback umount needed.
- The description's production baseline is 22 of 22 copies failing the
unmount.
- Copies that ran at the same time on one host each had their own mount (one
directory per template), so the refcount never had to hold a pool open across
two users here. The PR's unit tests cover a shared pool.
**Not covered: the lock-stall check.** The `jstack` snapshots I took during
the rounds are unusable because of a mistake in my capture script, so I make no
claim about cross-pool lock waits. The PR's unit tests cover the locking.
**Result:** pass.
- A busy mount is reported as a failed unmount, not a false success.
- Parallel first copies show no busy-unmount failures, and no mount is left
behind.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
