From: Emerson Busson <[email protected]> Sent: Thursday, September 17, 2026 6:40 PM > > When VMBus sub-channels (such as synthetic SCSI, network, or vsock > channels) are dynamically opened during periods of sustained memory > load or memory tier eviction, `vmbus_alloc_ring()` attempts to > allocate physically contiguous memory using > `alloc_pages(GFP_KERNEL | __GFP_ZERO, order)`. > > For standard rings (order-7, 512 KiB contiguous memory), high buddy > allocator fragmentation under memory pressure frequently causes > `alloc_pages()` to fail with -ENOMEM even when ample total virtual > memory is available. This manifests in userspace as connection > timeouts (e.g. `accept4 failed 110: Connection timed out` on WSL2 vsock > control planes). > > This patch introduces a resilient fallback mechanism: > 1. When `alloc_pages()` fails due to external fragmentation, > `vmbus_alloc_ring()` falls back to `vzalloc_node()` (or `vzalloc()`) > to satisfy the buffer allocation from virtually contiguous pages. > 2. In `hv_ringbuffer_init()`, detects `is_vmalloc_addr(virt_addr)` and > populates the `pages_wraparound` mapping array using > `vmalloc_to_page()`. > 3. In `vmbus_free_ring()`, tracks `ringbuffer_is_vmalloc` and calls > `vfree()` safely, preserving Confidential VM (CoCo) memory > re-encryption isolation checks prior to release.
The problem you describe is real, and there have been code changes in the past to remove places where high-order kernel memory allocations were needed to load and initialize a VMBus device or a PCI pass-thru devices (such as the Mellanox NIC VF). But as you noticed, the ring buffer allocations were never fixed. Unfortunately, your solution doesn't work in all CoCo VMs. In arm64 CCA VMs and in TDX VMs running without a paravisor, the set_memory_decrypted() call in __vmbus_establish_gpadl() doesn't work for vmalloc'ed memory. It only works for physically contiguous memory, and various proposals to make it work for vmalloc'ed memory have been stymied. As a result, Kameron Carr recently had a patch set [1] accepted to solve the problem. See the new function vmbus_alloc_buffer() and related. Solving this high-order ring buffer allocation problem is a prime opportunity to do a bit of refactoring: 1) Ring buffer allocation should *always* use Kameron's new mechanism, not just as a fallback if normal alloc_pages() fails. This would make things simpler, and even though Kameron's mechanism has a little more overhead, I think that's OK as ring buffer allocation/free is not a fast path. 2) The three data fields needed by Kameron's mechanism should be grouped into a structure that can be passed around. Currently the fields are separately added in the netvsc driver because that's the only place that needed a GPADL for a buffer larger than alloc_pages() could provide. But with broader usage for all ring buffers, some encapsulation into a struct such as struct vmbus_buffer would make sense. 3) We also have struct vmbus_gpadl. There's some logical duplication between the fields Kameron needed and what's in struct vmbus_gpadl. The stuff in struct vmbus_gpadl could be folded into struct vmbus_buffer and the duplication eliminated. 4) I have separately been working on a fix to making sure that memory is leaked if the GPADL can't be removed, or if the buffer memory can't be re-encrypted in a CoCo VM. I've run into messy issues that can't be cleanly solved with the current data structures. But having a struct vmbus_buffer with a "leak" flag field would be a clean solution. 5) Kameron's new code required introducing a new GPADL type HV_GPADL_BUFFER_DECRYPTED because "normal" buffers are decrypted in __vmbus_establish_gpadl(). But if vmbus_alloc_buffer() is always used, and it does the decryption in a CoCo VM instead of __vmbus_establish_gpadl(), then some of the HV_GPADL_* types can probably be eliminated. That's a quick summary of what I'm thinking. I haven't coded any of this, so maybe there are some flaws that I haven't immediately spotted, but I'm pretty sure this could all be worked out in a way that's better and simpler than what we have now, and that would avoid ring buffer allocation failures due to memory fragmentation. Michael [1] https://lore.kernel.org/linux-hyperv/[email protected]/ > > Signed-off-by: Emerson Busson <[email protected]> > --- > drivers/hv/channel.c | 46 ++++++++++++++++++++++++++++++++++----- > drivers/hv/hyperv_vmbus.h | 2 +- > drivers/hv/ring_buffer.c | 19 +++++++++++----- > include/linux/hyperv.h | 2 ++ > 4 files changed, 57 insertions(+), 12 deletions(-) > > diff --git a/drivers/hv/channel.c b/drivers/hv/channel.c > index 162d6aeec..f0fb3dd8f 100644 > --- a/drivers/hv/channel.c > +++ b/drivers/hv/channel.c > @@ -12,6 +12,7 @@ > #include <linux/sched.h> > #include <linux/wait.h> > #include <linux/mm.h> > +#include <linux/vmalloc.h> > #include <linux/slab.h> > #include <linux/module.h> > #include <linux/hyperv.h> > @@ -153,13 +154,20 @@ void vmbus_free_ring(struct vmbus_channel *channel) > hv_ringbuffer_cleanup(&channel->outbound); > hv_ringbuffer_cleanup(&channel->inbound); > > - if (channel->ringbuffer_page) { > + if (channel->ringbuffer_is_vmalloc && channel->ringbuffer_page_virt) { > + /* In a CoCo VM leak the memory if it didn't get re-encrypted */ > + if (!channel->ringbuffer_gpadlhandle.decrypted) > + vfree(channel->ringbuffer_page_virt); > + channel->ringbuffer_page_virt = NULL; > + channel->ringbuffer_is_vmalloc = false; > + } else if (channel->ringbuffer_page) { > /* In a CoCo VM leak the memory if it didn't get re-encrypted */ > if (!channel->ringbuffer_gpadlhandle.decrypted) > __free_pages(channel->ringbuffer_page, > get_order(channel->ringbuffer_pagecount > << PAGE_SHIFT)); > channel->ringbuffer_page = NULL; > + channel->ringbuffer_page_virt = NULL; > } > } > EXPORT_SYMBOL_GPL(vmbus_free_ring); > @@ -182,10 +190,26 @@ int vmbus_alloc_ring(struct vmbus_channel *newchannel, > if (!page) > page = alloc_pages(GFP_KERNEL|__GFP_ZERO, order); > > - if (!page) > - return -ENOMEM; > + if (!page) { > + /* Fallback to virtual memory allocation under buddy > fragmentation */ > + void *virt_addr = vzalloc_node(send_size + recv_size, > + > cpu_to_node(newchannel->target_cpu)); > + > + if (!virt_addr) > + virt_addr = vzalloc(send_size + recv_size); > + > + if (!virt_addr) > + return -ENOMEM; > + > + newchannel->ringbuffer_page = NULL; > + newchannel->ringbuffer_page_virt = virt_addr; > + newchannel->ringbuffer_is_vmalloc = true; > + } else { > + newchannel->ringbuffer_page = page; > + newchannel->ringbuffer_page_virt = page_address(page); > + newchannel->ringbuffer_is_vmalloc = false; > + } > > - newchannel->ringbuffer_page = page; > newchannel->ringbuffer_pagecount = (send_size + recv_size) >> > PAGE_SHIFT; > newchannel->ringbuffer_send_offset = send_size >> PAGE_SHIFT; > > @@ -639,6 +663,7 @@ static int __vmbus_open(struct vmbus_channel *newchannel, > struct vmbus_channel_open_channel *open_msg; > struct vmbus_channel_msginfo *open_info = NULL; > struct page *page = newchannel->ringbuffer_page; > + void *inbound_virt = NULL; > u32 send_pages, recv_pages; > unsigned long flags; > int err; > @@ -669,6 +694,8 @@ static int __vmbus_open(struct vmbus_channel *newchannel, > newchannel->ringbuffer_gpadlhandle.gpadl_handle = 0; > > err = __vmbus_establish_gpadl(newchannel, HV_GPADL_RING, > + newchannel->ringbuffer_page_virt ? > + newchannel->ringbuffer_page_virt : > page_address(newchannel->ringbuffer_page), > (send_pages + recv_pages) << PAGE_SHIFT, > newchannel->ringbuffer_send_offset << > PAGE_SHIFT, > @@ -677,11 +704,18 @@ static int __vmbus_open(struct vmbus_channel > *newchannel, > goto error_clean_ring; > > err = hv_ringbuffer_init(&newchannel->outbound, > - page, send_pages, 0); > + page, newchannel->ringbuffer_page_virt, > + send_pages, 0); > if (err) > goto error_free_gpadl; > > - err = hv_ringbuffer_init(&newchannel->inbound, &page[send_pages], > + if (newchannel->ringbuffer_page_virt) > + inbound_virt = newchannel->ringbuffer_page_virt + > + (send_pages << PAGE_SHIFT); > + > + err = hv_ringbuffer_init(&newchannel->inbound, > + page ? &page[send_pages] : NULL, > + inbound_virt, > recv_pages, newchannel->max_pkt_size); > if (err) > goto error_free_gpadl; > diff --git a/drivers/hv/hyperv_vmbus.h b/drivers/hv/hyperv_vmbus.h > index 34943de7d..ec06c30d2 100644 > --- a/drivers/hv/hyperv_vmbus.h > +++ b/drivers/hv/hyperv_vmbus.h > @@ -182,7 +182,7 @@ extern int hv_synic_cleanup(unsigned int cpu); > void hv_ringbuffer_pre_init(struct vmbus_channel *channel); > > int hv_ringbuffer_init(struct hv_ring_buffer_info *ring_info, > - struct page *pages, u32 pagecnt, u32 max_pkt_size); > + struct page *pages, void *virt_addr, u32 pagecnt, u32 > max_pkt_size); > > void hv_ringbuffer_cleanup(struct hv_ring_buffer_info *ring_info); > > diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c > index 23ce1fb70..e6d4cf185 100644 > --- a/drivers/hv/ring_buffer.c > +++ b/drivers/hv/ring_buffer.c > @@ -184,7 +184,7 @@ void hv_ringbuffer_pre_init(struct vmbus_channel *channel) > > /* Initialize the ring buffer. */ > int hv_ringbuffer_init(struct hv_ring_buffer_info *ring_info, > - struct page *pages, u32 page_cnt, u32 max_pkt_size) > + struct page *pages, void *virt_addr, u32 page_cnt, u32 > max_pkt_size) > { > struct page **pages_wraparound; > int i; > @@ -201,10 +201,19 @@ int hv_ringbuffer_init(struct hv_ring_buffer_info > *ring_info, > if (!pages_wraparound) > return -ENOMEM; > > - pages_wraparound[0] = pages; > - for (i = 0; i < 2 * (page_cnt - 1); i++) > - pages_wraparound[i + 1] = > - &pages[i % (page_cnt - 1) + 1]; > + if (virt_addr && is_vmalloc_addr(virt_addr)) { > + pages_wraparound[0] = vmalloc_to_page(virt_addr); > + for (i = 0; i < 2 * (page_cnt - 1); i++) { > + void *curr_virt = virt_addr + ((i % (page_cnt - 1) + 1) > << > PAGE_SHIFT); > + > + pages_wraparound[i + 1] = vmalloc_to_page(curr_virt); > + } > + } else { > + pages_wraparound[0] = pages; > + for (i = 0; i < 2 * (page_cnt - 1); i++) > + pages_wraparound[i + 1] = > + &pages[i % (page_cnt - 1) + 1]; > + } > > ring_info->ring_buffer = (struct hv_ring_buffer *) > vmap(pages_wraparound, page_cnt * 2 - 1, VM_MAP, > diff --git a/include/linux/hyperv.h b/include/linux/hyperv.h > index a76f556f5..63203df4d 100644 > --- a/include/linux/hyperv.h > +++ b/include/linux/hyperv.h > @@ -807,6 +807,8 @@ struct vmbus_channel { > > /* Allocated memory for ring buffer */ > struct page *ringbuffer_page; > + void *ringbuffer_page_virt; > + bool ringbuffer_is_vmalloc; > u32 ringbuffer_pagecount; > u32 ringbuffer_send_offset; > struct hv_ring_buffer_info outbound; /* send to parent */ > -- > 2.43.0 >
