Hi Team,
We recently upgraded from VPP 23.02 to 26.06 and observed a significant
regression in buffer pool allocation.
We'd like to understand the rationale behind the page boundary check
introduced in vlib_buffer_pool_create (in src/vlib/buffer.c).
The problem:
With the same configuration that has worked since 23.02:
buffers {
buffers-per-numa 128000
default data-size 3072
}
On VPP 23.02, buffer allocation works as expected:
vpp# show buffers
Pool Name Index NUMA Size Data Size Total Avail Cached
Used
default-numa-0 0 0 3968 3072 128000 127968 32 0
default-numa-1 1 1 3968 3072 128000 128000 0 0
default-numa-2 2 2 3968 3072 128000 128000 0 0
default-numa-3 3 3 3968 3072 128000 128000 0 0
*On VPP 26.06, only ~2032 buffers are allocated per NUMA instead of 128000:*
vpp# show buffers
Pool Name Index NUMA Size Data Size Total Avail Cached
Used
default-numa-0 0 0 3968 3072 2031 460 32
1539
default-numa-1 1 1 3968 3072 2032 2032 0 0
default-numa-2 2 2 3968 3072 2032 2032 0 0
default-numa-3 3 3 3968 3072 2031 2031 0 0
Root cause analysis:
The new code in vlib_buffer_pool_create skips any buffer that would span
across a page boundary:
* /* skip if buffer spans across page boundary */ if (((uword) p &
page_mask) != ((uword) (p + alloc_size) & page_mask)) continue;*
With 4KB pages and alloc_size = 3968, only 1 buffer fits per 4KB page (3968
< 4096), but the linear iteration with p += alloc_size causes the buffer
start offset to drift across pages.
The check then rejects ~94% of candidate slots because most will cross a
page boundary when stepping by 3968 through a 4096-aligned address space.
The memory is allocated (physmem region is sized for 128000 buffers), but
most of it goes unused — an effective utilization rate of only ~1.6%.
my questions:
1. Was the impact on non-hugepage deployments (4KB pages) considered? With
hugepages (e.g., 2MB), the check is essentially a no-op since many buffers
fit within a single page, but with 4KB pages and large
buffer sizes (>2048), the utilization drops drastically.
2. Would it be acceptable upstream to use a per-page iteration approach
instead — placing buffers at page-aligned offsets to guarantee no
cross-page spanning while maintaining full utilization?
Thanks & Regards
Amelia
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#27083): https://lists.fd.io/g/vpp-dev/message/27083
Mute This Topic: https://lists.fd.io/mt/119987044/21656
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/14379924/21656/631435203/xyzzy
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
