bhouse-nexthop opened a new pull request, #14199:
URL: https://github.com/apache/cloudstack/pull/14199
### Description
A VM deploy wedged a KVM agent thread for 7.8 hours and took the cluster's
VM creation with it. The thread was cloning a template on RBD and never
returned.
Stack from the agent, with the native frames underneath:
```
"AgentRequest-Handler-28" last log line: Creating volume <uuid> from
template <uuid>
KVMStorageProcessor.cloneVolumeFromBaseTemplate
LibvirtStorageAdaptor.createDiskFromTemplateOnRBD
com.ceph.rbd.Rbd.open -> JNA -> rbd_open
rbd_open + 366 librbd.so.1.19.0
std::condition_variable::wait(unique_lock&) the overload with no
timeout
__pthread_cond_wait_2_0 timeout = NULL
```
Ceph was healthy the whole time: `HEALTH_OK`, every OSD up, no watchers on
the template, no blocklisted clients, and the same template opened instantly
with `rbd` on that same host.
The user visible effect was that new VM deploys kept failing with
`InsufficientServerCapacityException`, on a cluster where every host had
terabytes of free memory. That error is what a deploy reports when the planner
ends up with no usable destination, whatever the reason, so it sent us looking
at capacity for a while before the real cause turned up in an agent thread dump.
Three things in the KVM plugin turned one stuck call into an outage:
| Problem | Detail |
| --- | --- |
| Nothing bounds the call | librados leaves `rados_osd_op_timeout` and
`rados_mon_op_timeout` at `0`, which means wait forever |
| Handles are never released | `Rados` only calls `rados_shutdown()` from
`finalize()`, and nothing in the plugin called `shutDown()`, so every handle
waited for the garbage collector |
| Cleanup only on the happy path | The clone path had no `finally` at all,
so a failure part way through leaked the open image and the IO context |
The agent runs a host's storage commands in sequence, so the stuck thread
also blocked every command queued behind it for that host.
### Fix
**1. Bound the operations.** A new `CephUtil.connect()` applies the timeouts
to every handle:
```
client_mount_timeout 30 unchanged
rados_osd_op_timeout 60 was unset, meaning forever
rados_mon_op_timeout 30 was unset, meaning forever
```
These bound a single operation, not a whole request, so a long copy or
flatten is made up of many operations that are each well inside the limit. Both
are settable in `agent.properties` as `rados.osd.op.timeout` and
`rados.mon.op.timeout`, and `0` restores the previous behaviour.
**2. Release what we open.** `CephUtil` adds non-throwing `closeQuietly`,
`ioCtxDestroyQuietly` and `shutDownQuietly`, and every path cleans up from a
`finally`.
| | before | after |
| --- | --- | --- |
| Rados handles created | 11 | 11, all through `CephUtil.connect()` |
| Rados handles released | 0 | 11 |
| Clone path cleanup | none | image, then IO context, then handle |
| Snapshot backup closed its image | no | yes |
**3. Open read only where we only read.** A read write open registers a
watcher on the image header and lets the image take its exclusive lock. A clone
parent needs neither:
```java
// before
RbdImage srcImage = rbd.open(template.getName());
// after
RbdImage srcImage = rbd.openReadOnly(template.getName());
```
Creating and protecting the base snapshot are writes, so the first clone of
a template reopens it read write for that one step. Every later clone of that
template keeps the read only handle. The same change applies to the cross
cluster copy and to reading a snapshot during backup. Paths that really do
write, such as `snapCreate`, `snapUnprotect` and `snapRemove`, are unchanged.
### What this does not fix
The thread was waiting on an internal librbd completion that never arrived,
so a per operation timeout may not reach that particular wait. This change
makes the agent recover and keeps one stuck call from blocking a host's whole
command queue. It does not fix librbd.
### Types of changes
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] build/CI
### How Has This Been Tested?
- `mvn -pl core,agent,plugins/hypervisors/kvm test` builds clean and the
existing unit tests pass.
- The behaviour above was traced on a production KVM and Ceph cluster: agent
thread dump, kernel `wchan` and `syscall` state showing an untimed futex wait,
and the librbd frames recovered by matching return addresses on the stuck
thread's stack against a disassembly of the same build.
- Defaults are conservative. A per operation limit of 60s is far above a
normal operation on a healthy cluster, and setting either property to `0`
restores exactly the old behaviour.
- Not yet exercised against a live clone on a patched agent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
