Performing verification for Focal. I will first reproduce the problem with glib2.0 2.64.6-1~ubuntu20.04.3 from -security with the libglib2.0-0 libglib2.0-bin libglib2.0-data packages.
I deleted all existing schemas from /usr/share/glib-2.0/schemas and replaced them with a set of schemas which reproduce the problem easily from my customer. $ cd /usr/share/glib-2.0/schemas/ $ sudo rm * $ sudo cp ~/schemas/* . The gsettings.compiled from the customer has been corrupted, and when I reboot, gdm fails to start and I get a blank screen with a blinking insertion pointer. The sha256 of the customers corrupted gsettings.compiled is: $ sudo openssl sha256 /usr/share/glib-2.0/schemas/gschemas.compiled SHA256(/usr/share/glib-2.0/schemas/gschemas.compiled)= 2c98dc9a7fdbac858a8d5ca7e4dd813f16058a46dba2c54b5239cd8cdba5bb3e When I ssh back in, and recompile the file: $ sudo glib-compile-schemas /usr/share/glib-2.0/schemas Error parsing key “logout” in schema “org.gnome.settings-daemon.plugins.media-keys” as specified in override file “/usr/share/glib-2.0/schemas/50_vmware_viewagent.gschema.override”: 0-22:can not parse as value of type 'as'. Ignoring override for this key. $ sudo openssl sha256 /usr/share/glib-2.0/schemas/gschemas.compiled SHA256(/usr/share/glib-2.0/schemas/gschemas.compiled)= 78163b5f0000efbd6320ce0d355c9531bf657a4f4dc15f057d95ef144323cd56 The sha256 has changed. Doing a bindiff, I see: $ sudo cmp -l /home/ubuntu/schemas/gschemas.compiled /usr/share/glib-2.0/schemas/gschemas.compiled | gawk '{printf "%08X %02X %02X\n", $1, strtonum(0$2), strtonum(0$3)}' 0000376F E3 4F 00003771 A4 C2 We see two bytes different. These bytes are the uninitialised memory this bug is about. When I reboot, gdm starts fine, but that is because this time I got lucky and the parser for the gschema.compiled file thinks 4F and C2 are okay. But there are combinations which aren't okay, and will end up with a corrupted gschema.compiled file. Re-compiling the file again: $ sudo glib-compile-schemas /usr/share/glib-2.0/schemas Error parsing key “logout” in schema “org.gnome.settings-daemon.plugins.media-keys” as specified in override file “/usr/share/glib-2.0/schemas/50_vmware_viewagent.gschema.override”: 0-22:can not parse as value of type 'as'. Ignoring override for this key. $ sudo openssl sha256 /usr/share/glib-2.0/schemas/gschemas.compiled SHA256(/usr/share/glib-2.0/schemas/gschemas.compiled)= 460c70faca7afc26fa88a0e5918d312478e15f20ad84f4afaa5d17627a823e01 The sha256 changed, and if we bindiff, the bytes have changed: $ sudo cmp -l /home/ubuntu/schemas/gschemas.compiled /usr/share/glib-2.0/schemas/gschemas.compiled | gawk '{printf "%08X %02X %02X\n", $1, strtonum(0$2), strtonum(0$3)}' 0000376F E3 A6 00003771 A4 A1 If we run glib-compile-schemas through valgrind, it reports that we are writing to uninitialised memory: https://paste.ubuntu.com/p/sxrQtbswpw/ I then enabled -proposed and installed libglib2.0-0 libglib2.0-bin libglib2.0-data version 2.64.6-1~ubuntu20.04.4. Now, when I re-compile the gschemas.compiled file, the sha256 matches every time, meaning no more non-deterministic behaviour caused by writing unitialised memory to disk: $ sudo glib-compile-schemas /usr/share/glib-2.0/schemas Error parsing key “logout” in schema “org.gnome.settings-daemon.plugins.media-keys” as specified in override file “/usr/share/glib-2.0/schemas/50_vmware_viewagent.gschema.override”: 0-22:can not parse as value of type 'as'. Ignoring override for this key. $ sudo openssl sha256 /usr/share/glib-2.0/schemas/gschemas.compiled SHA256(/usr/share/glib-2.0/schemas/gschemas.compiled)= cd9132d18b596a304251cd1eb50b64aa6fd7511a312906f9a49e1975a319fbf1 $ sudo glib-compile-schemas /usr/share/glib-2.0/schemas Error parsing key “logout” in schema “org.gnome.settings-daemon.plugins.media-keys” as specified in override file “/usr/share/glib-2.0/schemas/50_vmware_viewagent.gschema.override”: 0-22:can not parse as value of type 'as'. Ignoring override for this key. $ sudo openssl sha256 /usr/share/glib-2.0/schemas/gschemas.compiled SHA256(/usr/share/glib-2.0/schemas/gschemas.compiled)= cd9132d18b596a304251cd1eb50b64aa6fd7511a312906f9a49e1975a319fbf1 Doing a bindiff, I see the changed bytes from before are now all zeros, which is what the patch initialises the buffer to: $ sudo cmp -l /home/ubuntu/schemas/gschemas.compiled /usr/share/glib-2.0/schemas/gschemas.compiled | gawk '{printf "%08X %02X %02X\n", $1, strtonum(0$2), strtonum(0$3)}' 0000376F E3 00 00003771 A4 00 00003772 55 00 Doing a run through valgrind, we no longer get a report about writing to uninitialised memory: https://paste.ubuntu.com/p/z52DGZcdz3/ Rebooting, the VM comes up and GDM starts properly, so glib can parse the gsettings.compiled file without any issues. Wonderful. The problem is fixed by the package in -proposed, happy to mark as verified. ** Tags removed: sts-sponsor verification-needed verification-needed-focal ** Tags added: verification-done verification-done-focal -- You received this bug notification because you are a member of Desktop Packages, which is subscribed to glib2.0 in Ubuntu. https://bugs.launchpad.net/bugs/1930359 Title: glib2.0: Uninitialised memory is written to gschema.compiled, failure to parse this file leads to gdm, gnome-shell failing to start Status in GLib: Unknown Status in glib2.0 package in Ubuntu: Fix Released Status in glib2.0 source package in Focal: Fix Committed Bug description: [Impact] A recent SRU of mutter 3.36.9-0ubuntu0.20.04.1 caused an outage for a user with 300 VDIs running Focal, where GNOME applications would fail to start, and if you reboot, gdm and gnome-shell both fail to start, and you are left with a black screen and a blinking cursor. After much investigation, mutter was not at fault. Instead, mutter- common calls the libglib2.0-0 hook on upgrade: Processing triggers for libglib2.0-0:amd64 (2.64.6-1~ubuntu20.04.3) ... This in turn calls glib-compile-schemas to recompile the gsettings gschema cache, from the files in /usr/share/glib-2.0/schemas/. The result is a binary gschemas.compiled file, which is loaded by libglib2.0 on every invocation of a GNOME application, or gdm or gnome-shell to fetch application default settings. Now, glib2.0 2.64.6-1~ubuntu20.04.3 in Focal has some non- deterministic behaviour when calling glib-compile-schemas, causing generated gschemas.compiled files to have differing contents on each run: # glib-compile-schemas /usr/share/glib-2.0/schemas # cmp -l /home/ubuntu/schemas/gschemas.compiled /usr/share/glib-2.0/schemas/gschemas.compiled | gawk '{printf "%08X %02X %02X\n", $1, strtonum(0$2), strtonum(0$3)}' 0000376F E3 D0 00003771 A4 DB # glib-compile-schemas /usr/share/glib-2.0/schemas # cmp -l /home/ubuntu/schemas/gschemas.compiled /usr/share/glib-2.0/schemas/gschemas.compiled | gawk '{printf "%08X %02X %02X\n", $1, strtonum(0$2), strtonum(0$3)}' 0000376F E3 C3 00003771 A4 98 # glib-compile-schemas /usr/share/glib-2.0/schemas # cmp -l /home/ubuntu/schemas/gschemas.compiled /usr/share/glib-2.0/schemas/gschemas.compiled | gawk '{printf "%08X %02X %02X\n", $1, strtonum(0$2), strtonum(0$3)}' 0000376F E3 68 00003771 A4 30 00003772 55 56 The bytes on the left are from a corrupted gschemas.compiled provided by an affected user. The changing bytes on the right are non- deterministic. I ran valgrind over glib-compile-schemas, and found that we are writing to uninitialised memory. https://paste.ubuntu.com/p/hvZccwdzxz/ What is happening is that a submodule of glib, gvdb, contains the logic for serialising the gschema data structures, and when it allocates a buffer to store the eventual gschemas.compiled file, it does not initialise it. When we populate the fields in the buffer, some bytes are never overwritten, and these junk bytes find themselves written to gschemas.compiled. On boot, when gdm and gnome-shell attempt to parse and load this corrupted gschemas.compiled file, it can't parse the junk bytes, and raises and error, which propagates up to a breakpoint in glib logging, but no debugger is present, so the kernel traps the breakpoint, and terminates the library, and the calling application, e.g. gdm. The result is that the user is left starting at a black screen with a blinking pointer. [Testcase] On a Focal system, simply run valgrind over glib-compile-schemas: # valgrind glib-compile-schemas /usr/share/glib-2.0/schemas You will get output like this, with the warning "Syscall param write(buf) points to uninitialised byte(s)": https://paste.ubuntu.com/p/hvZccwdzxz/ If you happen to have a large amount of gschema overrides present on your system, like my affected user does, you can save a copy of a generated gschema.compiled to your home directory and bindiff it against recompiles: # glib-compile-schemas /usr/share/glib-2.0/schemas # cp /usr/share/glib-2.0/schemas/gschema.compiled /home/ubuntu/schemas/gschemas.compiled # glib-compile-schemas /usr/share/glib-2.0/schemas # cmp -l /home/ubuntu/schemas/gschemas.compiled /usr/share/glib-2.0/schemas/gschemas.compiled | gawk '{printf "%08X %02X %02X\n", $1, strtonum(0$2), strtonum(0$3)}' 0000376F E3 C3 00003771 A4 98 If you install the test package from the following ppa: https://launchpad.net/~mruffell/+archive/ubuntu/sf311791-test When you run valgrind, it will report a clean run with no writing to uninitialised buffers, and all invocations of glib-compile-schemas will be deterministic, and generate the file same with the same sha256 hash every time. The unwritten bytes if you do a bindiff from before and after will be all set to zero. [Where problems can occur] I am doubtful that any programs are relying on buggy non-deterministic behaviour from random bytes found in uninitialised memory, so this should be a relatively safe change. Since we are updating glib, which all GNOME applications, gdm and gnome-shell link to, if we introduce an error, it could cause these applications to stop working, and at a worse case, see the symptoms this bug is trying to fix, which is a blinking cursor on a blank screen. Installing any updates to glib also causes the gsettings gschema cache to be re-generated, and from this bug, we know that libglib seems to trust the gschema.compiled file and doesn't perform much validation, if the user has bad data in their gschema files, it could lead to their systems having issues on next boot. If a regression occurs, users should first attempt to re-generate their schemas like so: glib-compile-schemas /usr/share/glib-2.0/schemas and if that fails, then they should downgrade their libglib2.0-0 libglib2.0-bin libglib2.0-data packages. [Other info] This was fixed by the commit: commit ea64c739239faea463f3cb9154a12cc4532ba525 Author: Philip Withnall <withn...@endlessm.com> Date: Wed Mar 18 09:15:59 2020 +0000 Subject: gvdb-builder: Initialise some memory to zero in the bloom filter Link: https://github.com/GNOME/glib/commit/ea64c739239faea463f3cb9154a12cc4532ba525 Only Focal needs this patch, Groovy and up are unaffected. To manage notifications about this bug go to: https://bugs.launchpad.net/glib/+bug/1930359/+subscriptions -- Mailing list: https://launchpad.net/~desktop-packages Post to : desktop-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~desktop-packages More help : https://help.launchpad.net/ListHelp