On Fri, Oct 16, 2026 at 05:42:45PM -0400, Gregory Price (Meta) wrote:
> From: Gregory Price <[email protected]>
>
> When the kernel provides daxX.Y/state, offline by writing 'unplugged'.
> The kernel offlines and removes the whole device atomically, so no
> agent can online blocks before a later unbind. On failure the device
> is left unchanged.
>
> The memory blocks are removed rather than left offline, including
> when they are all offline already; 'daxctl online-memory' restores
> them. If they are all offline and the unplug fails (e.g. no memory
> hotremove support), this remains a no-op.
>
> Unplugging a node's last memory removes its sysfs directory, so treat
> a missing node directory as zero blocks when the device is unplugged.
> 'daxctl list' likewise omits 'movable' for a device with no blocks.
>
> Signed-off-by: Gregory Price <[email protected]>
> ---
> daxctl/device.c | 51 +++++++++++++++++++++++++++++++++++++++---
> daxctl/json.c | 6 ++++-
> daxctl/lib/libdaxctl.c | 12 ++++++++--
> 3 files changed, 63 insertions(+), 6 deletions(-)
>
> diff --git a/daxctl/device.c b/daxctl/device.c
> index fd84799..99c0d96 100644
> --- a/daxctl/device.c
> +++ b/daxctl/device.c
> @@ -694,7 +694,8 @@ static int dev_offline_memory(struct daxctl_dev *dev)
> {
> struct daxctl_memory *mem = daxctl_dev_get_memory(dev);
> const char *devname = daxctl_dev_get_devname(dev);
> - int num_sections, num_on, num_off, rc;
> + int num_sections, num_on, num_off, rc, unplug;
> + enum daxctl_kmem_state state;
>
> if (!mem) {
> fprintf(stderr, "%s: failed to get the memory object\n",
> @@ -717,17 +718,61 @@ static int dev_offline_memory(struct daxctl_dev *dev)
> return num_on;
> }
>
> + /*
> + * Unplug atomically where supported, even if already offline, so
> + * no agent can online blocks before a later unbind.
> + */
> + state = daxctl_dev_get_kmem_state(dev);
> + unplug = state != DAXCTL_KMEM_UNKNOWN && state != DAXCTL_KMEM_UNPLUGGED;
> +
Hi Greg,
I see you called this out in the cover letter, and that the kernel is
deliberately combining offlining and removing memory to avoid the race
before unbind.
But, this does change the behavior of the command "daxctl offline-memory".
Previously it left memory blocks present but offline and now it removes
them entirely.
Should offline-memory keep its existing behavior of leaving memory blocks
offline, and use atomic unplug only when reconfiguring the device?
> num_off = num_sections - num_on;
> - if (num_off == num_sections) {
> + if (num_off == num_sections && !unplug) {
> fprintf(stderr, "%s: all memory sections (%d) already
> offline\n",
> devname, num_off);
> return 1;
> }
> - if (num_off)
> + if (num_off && num_on)
> fprintf(stderr, "%s: %d memory section%s already offline\n",
> devname, num_off,
> num_off == 1 ? "" : "s");
>
> + if (unplug) {
> + rc = daxctl_dev_set_kmem_state(dev, DAXCTL_KMEM_UNPLUGGED);
> + if (rc < 0 && num_on == 0) {
> + /* already offline, unplug is best effort */
When num_on == 0, another agent could online a block between the earlier
check and the unplug and that memory would then be removed.
I see this noted in the commit message. Is that worth documenting for
users?
-- Alison
