Public bug reported:
[ Impact ]
On Ubuntu 26.04 (mutter 50.1-0ubuntu2.5), a screen cast of a virtual monitor
(org.gnome.Mutter.ScreenCast.Session.RecordVirtual) makes gnome-shell write
past the end of its PipeWire buffer on every frame when
the negotiated height isn't a whole multiple of the monitor scale.
gnome-shell then crashes with SIGSEGV, and the user's session is lost.
Why: mutter 50 lays monitors out in logical pixels. A 1800x951 virtual
monitor at scale 2 gets a logical size of 900 x roundf(951 / 2) = 900x476.
meta_screen_cast_virtual_stream_src_record_to_buffer() ignores
the buffer's own width and height and calls clutter_stage_paint_to_buffer(),
which sizes both its texture and the CoglBitmap that wraps the buffer from the
view: 476 x 2 = 952 rows. The MemFd buffer mutter
allocated holds 951 rows (stride 7200 x 951), so
cogl_framebuffer_read_pixels_into_bitmap() writes one row, 7200 bytes, past its
end on every frame.
What happens next depends on what is mapped after the buffer:
- If the next page is unmapped, gnome-shell gets SIGSEGV in the GL driver.
- If it is mapped (another screen-cast buffer, or a pipewire-memfd), that
memory is corrupted silently. gnome-shell can then fail later somewhere else.
One of our crashes was a SIGSEGV with si_code SI_KERNEL
and no fault in the GL driver; we think it was such a later failure.
It happens whenever height mod scale >= scale / 2: an odd height at scale 2,
height mod 3 = 2 at scale 3, height mod 4 = 2 at scale 4. An odd width alone
overruns by only a few bytes, which normally stays
inside the last page.
Who hits it: anything that creates a virtual monitor through RecordVirtual
and sizes it to something outside its control, typically a remote-desktop
server that follows the remote client's window size. We hit
it with a remote-desktop server on Ubuntu 26.04. The scale usually comes from
the global org.gnome.desktop.interface scaling-factor (HiDPI clients). Settings
> Displays would not offer 200% for 1800x951, but
the global scaling-factor isn't checked against the monitor's mode.
[ Steps to reproduce ]
The attached script, repro-virtual-odd-height.py, needs python3-gi,
gstreamer1.0-tools and gstreamer1.0-pipewire. It runs 6 casts of 10 seconds
each, one after another. Each cast creates a virtual monitor
through RecordVirtual, pulls WIDTH x HEIGHT frames with gst-launch-1.0
pipewiresrc, and moves the pointer over that monitor through
org.gnome.Mutter.RemoteDesktop so frames keep coming. It prints PASS if
gnome-shell survives all 6 casts, otherwise FAIL. It doesn't start
gnome-shell itself; if none is running on the user's session bus, it prints
ERROR and stops.
A single cast overruns on every frame but often doesn't fault, because
the overflow lands in a mapped neighbour. That is why the script repeats
the cast. In our tests every run crashed by the 3rd cast.
The steps run a headless gnome-shell, so no display is needed. As
root:
1. Create a scratch user and start its user session:
useradd -m repro-test
loginctl enable-linger repro-test
U=$(id -u repro-test)
systemctl start user@$U.service
RU="runuser -u repro-test -- env XDG_RUNTIME_DIR=/run/user/$U
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$U/bus"
2. Set scale 2 and start a headless gnome-shell:
$RU gsettings set org.gnome.desktop.interface scaling-factor 2
$RU systemd-run --user --unit=repro-shell
--setenv=XDG_SESSION_TYPE=wayland gnome-shell --wayland --headless --no-x11
--virtual-monitor 1280x720
sleep 12
3. Run the script:
$RU python3 /path/to/repro-virtual-odd-height.py 1800 951
Actual result: the script prints FAIL, because gnome-shell crashed.
journalctl -k usually shows "gnome-shell[...]: segfault at ... error 6 in
libgallium-...so" (we also saw error 7, and some crashes left no
kernel line).
Expected result: the script prints PASS, and there is no gnome-shell segfault.
Other sizes and scales:
- Scales 3 and 4 crash too: set scaling-factor to 3 or 4, restart gnome-shell
($RU systemctl --user restart repro-shell), wait about 12 seconds, and run the
script with 1800 950.
- These don't crash: 1800 950 at scale 2, and 1800 951 at scale 1.
Clean up:
$RU gsettings reset org.gnome.desktop.interface scaling-factor
loginctl disable-linger repro-test
systemctl stop user@$U.service
userdel -r repro-test
[ Upstream fix ]
Fixed upstream by commit 93a178ea "screen-cast/stream-src: Add
meta_screen_cast_stream_src_paint_to_buffer"
(https://gitlab.gnome.org/GNOME/mutter/-/commit/93a178eaf501c881994e86b1f5246b9a96d3581e,
MR
https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/5046), on main since
2026-06-09. It is in 51.alpha and later, so the development series (tenacious,
mutter 51.0-1ubuntu3) and Ubuntu 26.10 already have
it.
It was backported to the gnome-50 branch as da684eeb
(https://gitlab.gnome.org/GNOME/mutter/-/commit/da684eebaac1d8a3fce1928b19829419c8b00778,
2026-10-26), but no 50.x release contains it yet; 50.6 was tagged
on 2026-10-13. Please consider backporting da684eeb to 26.04. It applies to
50.1-0ubuntu2.6 with line offsets only.
The new helper paints the view into an offscreen framebuffer of the buffer's
own width x height, then reads exactly width x height back into the buffer.
Rounding of the logical size can then only change which
pixels get drawn, never how many bytes get written. The commit message
describes it as a refactor and CPU saving, and doesn't mention the overflow.
[ Other Info ]
Tested on Ubuntu 26.04: a Xen VM with 2 vCPUs, no GPU (Mesa 26.0.3
llvmpipe), kernel 7.0.0-42 and gnome-shell 50.1-0ubuntu1.5, using the
steps above.
Stock libmutter-18-0 50.1-0ubuntu2.5 — the reproducer printed FAIL in 7 of 7
runs:
- scale 2 at 1800x951: 3/3, in the 2nd cast;
- scale 3 at 1800x950: 2/2, in the 2nd cast;
- scale 4 at 1800x950: 2/2, in the 2nd and 3rd casts.
Six of those crashes were the same fault: "segfault ... error 6/7 in
libgallium-26.0.3-1ubuntu1.so" at offset 0x3b4b21. The seventh was the
SI_KERNEL SIGSEGV mentioned above. A uprobe on
clutter_stage_paint_to_buffer confirmed the overrun on every frame: 900x476
at scale 2 into 951 rows, 600x317 at scale 3 into 950, 450x238 at scale 4 into
950.
With da684eeb applied to 50.1-0ubuntu2.6 (built locally, package tests not
run), the reproducer printed PASS in 10 of 10 runs of 6 casts each: scale 2 at
1800x951 4/4, scale 3 at 1800x950 2/2, scale 4 at
1800x950 2/2, and the two sizes that don't crash 2/2. There were no
gnome-shell segfaults, and the cast frames showed complete, correct pictures.
We first saw this as gnome-shell crashes under a remote-desktop server
driving RecordVirtual (2200x1151 and 1800x951 at scale 2, both crashing within
about 20 s). The stack in that case was libgallium <-
libmutter-cogl cogl_framebuffer_read_pixels_into_bitmap <-
clutter_stage_paint_to_buffer <-
meta_screen_cast_virtual_stream_src_record_to_buffer <-
clutter_stage_paint_view <- clutter_frame_clock_dispatch.
The faulting address was the exact end of the /memfd:mutter-screen-cast-memfd
mapping.
The change in 50.1-0ubuntu2.6 (multi-GPU scanout) doesn't touch this
path.
** Affects: mutter (Ubuntu)
Importance: Undecided
Status: New
** Tags: resolute
** Attachment added: "repro-virtual-odd-height.py"
https://bugs.launchpad.net/bugs/2174817/+attachment/6019054/+files/repro-virtual-odd-height.py
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2174817
Title:
gnome-shell crashes when a virtual-monitor screen cast's size isn't a
multiple of the scale
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/mutter/+bug/2174817/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs