bhouse-nexthop commented on PR #14199:
URL: https://github.com/apache/cloudstack/pull/14199#issuecomment-6737449618
### Test report: Ceph outages on a KVM host, with the agent's Ceph traffic
blocked
**Environment**
- A CloudStack 24.0 (main) build with this PR applied, alongside some of my
other open PRs. That includes #14188 (merged), which looks RBD volumes up
through librbd.
- **4.22 caveat:** I ran this on 24.0, not 4.22. The KVM storage code this
PR touches (`LibvirtStorageAdaptor`, `KVMStorageProcessor`,
`KVMStoragePoolManager` and the snapshot wrappers) is the same on 4.22 and
main, so I expect the same behaviour there. I have not run it on a 4.22 build.
- Three KVM hosts on Ubuntu 24.04 (libvirt 10.0.0, QEMU 8.2.2), nested
virtualisation. Ceph squid 19.2.3, RBD primary storage, NFS secondary storage.
- The agent's defaults: `rados.osd.op.timeout=60`, `rados.mon.op.timeout=30`.
**How the fault was injected**
- An nftables rule on the KVM host drops only the traffic from the
`cloudstack-agent.service` cgroup to Ceph:
```
nft add rule inet cephblk out socket cgroupv2 level 2
'"system.slice/cloudstack-agent.service"' ip daddr <ceph> tcp dport '{
6800-7568 }' counter drop
```
Steps 1, 2 and 5 drop the OSD ports only, so `rados_connect` succeeds and
the image open waits on the OSDs. Step 4 also drops the mon ports.
- qemu runs in `machine.slice` and is not matched, so running guests keep
their disk I/O. I checked this before the run, and again after every block by
watching the guests' I/O request counters grow.
- Processes the agent starts, such as `qemu-img`, inherit its cgroup and are
blocked too.
- Ceph was `HEALTH_OK` after every step, and all hosts stayed Up throughout.
**History**
- The first lab run of this PR found that each Ceph operation was bounded,
but `KVMStoragePoolManager.getPhysicalDisk()` retried any failure 100 times, 3
s apart. A deploy with the OSDs blocked failed one open every ~63 s and would
have held the command for about 105 minutes. A snapshot behaved the same way.
- The PR now treats a Ceph error other than librbd's `-ENOENT` as final
("not retrying a Ceph error"). Everything below is on that version.
**1. OSD ports blocked, deploy a VM to the host**
- The deploy **failed after 67 s**, with `Unable to orchestrate the start of
VM instance ...`.
- The agent log shows one bounded open, and no retry:
```
07:12:24 Trying to fetch storage pool <pool> from libvirt
07:13:24 ERROR A Ceph RBD operation failed (-110). The error was: Failed
to open image <template> - Connection timed out
07:13:24 WARN [KVMStoragePoolManager] Failed to find volume:<template>
due to ...RbdException: Failed to open image <template>, not retrying a Ceph
error
```
- At 45 s, `jstack` showed the deploy's handler thread in
`rbd_open_read_only`, as expected while the open waits. A second handler was
there too, running a `GetVolumeStatsCommand` from the stats collector. That one
also came back after its single bounded open (answered at 56 s). Neither
command held the other up, and the host stayed Up.
- After I removed the block, a fresh deploy to the same host succeeded.
**2. "Without": the timeouts set to 0, which is the stock behaviour**
- I set `rados.osd.op.timeout=0` and `rados.mon.op.timeout=0` in
`agent.properties`, restarted the agent, and repeated step 1.
- The deploy's handler thread was parked in `rbd_open_read_only` at 70 s and
again at 370 s: the same thread, with its CPU time unchanged
(`AgentRequest-Handler-5 #56 ... cpu=58.26ms elapsed=96s`, then
`elapsed=397s`). At 430 s, a second handler (a later command) was parked the
same way.
- The agent process was the same throughout; nothing restarted it.
- When I removed the block after ~450 s, the parked open completed and the
VM came up Running. That is the stock "wait for as long as Ceph is away".
- I put the defaults back and restarted the agent afterwards.
**3. Snapshot of a running VM's ROOT with the OSD ports blocked**
- `createSnapshot` **failed after 62 s**. The agent logged the same single
`-110` open and the "not retrying a Ceph error" line, then `Failed take
snapshot for volume ...`.
- `listSnapshots` shows the snapshot in `Error`, not BackedUp, and `rbd snap
ls` on the image is empty, so there is no stray RBD snapshot.
- Once unblocked, a new snapshot of the same ROOT reached BackedUp in 4 s.
**4. Snapshot backup interrupted mid-copy**
- A 20 GB data disk, filled from `/dev/urandom`. I used 20 GB rather than 40
GB to stay clear of the pool's nearfull ratio.
- `snapshot.backup.to.secondary=true`. I started `createSnapshot`, and found
the copy by watching for its process on the hosts:
```
qemu-img convert -O qcow2 -U
rbd:cloudstack/<image>@<snap>:mon_host=...:client_mount_timeout=30
/mnt/<secondary>/snapshots/...
cgroup 0::/system.slice/cloudstack-agent.service
```
- With 7.0 GB copied (and growing at ~240 MB/s), I blocked all of the agent
cgroup's Ceph ports on that host for 180 s.
- The copy stalled at 7045709824 bytes for the whole window, and
`qemu-img` stayed alive.
- After the block was removed, `qemu-img` resumed and finished 85 s later.
- The snapshot ended BackedUp with a full-size file: 21478375424 bytes,
`qemu-img info` virtual size 20 GiB, matching `physical_size` in
`snapshot_store_ref`. No short file was recorded.
- **Note on scope:** this backup path is
`KVMStorageProcessor.backupSnapshot()`, which copies with `qemu-img convert`.
This PR doesn't change that code, and `qemu-img` gets no op timeout, so a
longer outage would hold the copy until `backup.snapshot.wait`.
- The PR's read-loop fix (a negative `rbd_read` is now an error instead of
end-of-image) is in `LibvirtBackupSnapshotCommandWrapper`. I found no current
management-server path that sends `BackupSnapshotCommand`, so that change is
covered by the unit tests only.
**5. Watchers on the template image (from the first run, same code for this
part)**
- I sampled `rbd status` on the template image every ~0.2 s while 6 VMs were
deployed from it at once, 3 to each of two hosts.
- No watcher appeared, and all 6 VMs were Running. The clone parent is
opened read-only.
**6. Cluster handles released (from the first run, same code for this part)**
- `ceph tell mon.* sessions` entries from the host: 22 before and 22 after
10 deploy/destroy cycles on it.
**Result:** pass.
- With the OSDs unreachable, a deploy and a snapshot now fail in about one
`rados.osd.op.timeout` (67 s and 62 s), instead of waiting for as long as Ceph
is away.
- Other commands on the host keep running, and the host stays Up.
- With the timeouts set to 0, the agent goes back to the stock behaviour,
with a handler parked in `rbd_open` for the whole outage.
- A failed snapshot leaves no RBD snapshot behind and is not recorded as
created. An interrupted backup did not record a short file.
- Clone opens hold no watchers, and the cluster handles are released.
**Not covered**
- A mon-only outage during a long copy, the internal librbd completion hang,
a cross-cluster copy, and a pool without cephx.
- `BackupSnapshotCommand` (see step 4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
