Hi vmwgfx maintainers and DRM developers, I have encountered a severe Slab memory leak in the vmwgfx driver when using a Wayland compositor (KWin) with Atomic KMS enabled. This issue eventually leads to complete memory exhaustion and OOM within a few minutes of normal desktop usage.
[Environment] Kernel: 7.0-rc3 (Mainline) and 6.17.0-14 (HWE kernel) Distro: KDE neon Unstable (Ubuntu 24.04 LTS base) Compositor: KWin Wayland (Latest master branch) Guest Graphics: VMware SVGA II Adapter (vmwgfx) [Symptoms & Evidence] When moving windows or triggering screen updates on the Wayland session, system memory is rapidly consumed. Monitoring with slabtop reveals that radix_tree_node and kmalloc-rnd-07-1k objects are leaking aggressively per frame. It appears that DRM objects (likely GEM handles/IDR or framebuffers) created during the Atomic KMS commit are not being properly freed by the driver after the frame is rendered, leaving orphaned objects in the kernel Slab. Earlier, before disabling 3D acceleration, dmesg also showed: vmwgfx: mob memory overflow. Consider increasing guest RAM and graphicsMemory. However, even with 3D acceleration disabled (llvmpipe fallback), the Slab memory leak persists as long as Atomic KMS is used. [Steps to Reproduce] 1. Boot a VMware guest with vmwgfx and launch a modern Wayland compositor (KWin) that strictly uses Atomic KMS. 2. Move windows around to force screen redraws. 3. Run sudo slabtop -s c and observe radix_tree_node and kmalloc-rnd-07-1k growing indefinitely. [Workaround] Disabling Atomic KMS in the compositor completely stops the memory leak. Setting the environment variable KWIN_DRM_NO_AMS=1 forces KWin to use legacy DRM API, which works perfectly without any Slab growth. This strongly suggests the bug resides in the vmwgfx atomic commit/cleanup path. Any assistance or guidance on testing potential patches would be greatly appreciated. I am happy to provide further logs or test patches if needed. Best regards, Yuma Kakei
