Can you retry with https://patchwork.kernel.org/project/dri-devel/patch/[email protected]/ appliecd to your kernel?
Thank you, Ian On Thu, Mar 12, 2026 at 11:48 AM Yuma Kakei <[email protected]> wrote: > Hi vmwgfx maintainers and DRM developers, > > I have encountered a severe Slab memory leak in the vmwgfx driver when > using a Wayland compositor (KWin) with Atomic KMS enabled. This issue > eventually leads to complete memory exhaustion and OOM within a few minutes > of normal desktop usage. > > [Environment] > Kernel: 7.0-rc3 (Mainline) and 6.17.0-14 (HWE kernel) > Distro: KDE neon Unstable (Ubuntu 24.04 LTS base) > Compositor: KWin Wayland (Latest master branch) > Guest Graphics: VMware SVGA II Adapter (vmwgfx) > > [Symptoms & Evidence] > When moving windows or triggering screen updates on the Wayland session, > system memory is rapidly consumed. > Monitoring with slabtop reveals that radix_tree_node and kmalloc-rnd-07-1k > objects are leaking aggressively per frame. > > It appears that DRM objects (likely GEM handles/IDR or framebuffers) > created during the Atomic KMS commit are not being properly freed by the > driver after the frame is rendered, leaving orphaned objects in the kernel > Slab. > > Earlier, before disabling 3D acceleration, dmesg also showed: > vmwgfx: mob memory overflow. Consider increasing guest RAM and > graphicsMemory. > However, even with 3D acceleration disabled (llvmpipe fallback), the Slab > memory leak persists as long as Atomic KMS is used. > > [Steps to Reproduce] > 1. Boot a VMware guest with vmwgfx and launch a modern Wayland compositor > (KWin) that strictly uses Atomic KMS. > 2. Move windows around to force screen redraws. > 3. Run sudo slabtop -s c and observe radix_tree_node and kmalloc-rnd-07-1k > growing indefinitely. > > [Workaround] > Disabling Atomic KMS in the compositor completely stops the memory leak. > Setting the environment variable KWIN_DRM_NO_AMS=1 forces KWin to use > legacy DRM API, which works perfectly without any Slab growth. This > strongly suggests the bug resides in the vmwgfx atomic commit/cleanup path. > > Any assistance or guidance on testing potential patches would be greatly > appreciated. I am happy to provide further logs or test patches if needed. > > Best regards, > Yuma Kakei
smime.p7s
Description: S/MIME Cryptographic Signature
