Bug#942881: Audio on Lenovo X1 Carbon 5th generation stopped working after upgrade to linux-image-5.3.0-1-amd64 ("No response from codec")
On Fri, 2019-10-25 at 10:22 +0200, Ansgar Burchardt wrote: > Ben Hutchings writes: > > The code signing service logs every file it signs, along with a hash of > > the detached signature, but I don't know where the logs are so I can't > > comapre with that. > > I checked the audit log, but I don't think it will help much. It > currently records that: > > - 2019-10-21 07:20:03.898781: >decided to sign > linux-image-5.3.0-1-amd64-unsigned_5.3.7-1_amd64/[...]/snd-hda-codec-hdmi.ko >with sha256sum > 3fe77a308b28825f0d18717e073b411246aea9bb753f76f6071b3fc4e60c6005 > > - 2019-10-21 07:20:04.175379: >signature for the file logged >with sha256sum > c2a36f35867ae92b8664f4bd2193e70370eb3b92013ea53f3573d2508d3da4cb >(which matches snd-hda-codec-hdmi.ko.sig in src:linux-signed-amd64) Thanks. > So linux' sign-file likely produced a truncated file for some reason; > note that ftp-master still uses linux-kbuild-4.9/4.9.189-3+deb9u1. sign-file has only changed very slightly since then, and the changes don't affect its use of OpenSSL. So this version should still be fine. Ben. -- Ben Hutchings Humans are not rational beings; they are rationalising beings. signature.asc Description: This is a digitally signed message part
Bug#942881: Audio on Lenovo X1 Carbon 5th generation stopped working after upgrade to linux-image-5.3.0-1-amd64 ("No response from codec")
On Fri, 2019-10-25 at 11:21 +0200, Tomas Janousek wrote: > Cool, thanks for reproducing the issue! Just one question: when you say > production key, does that mean a hardware security module like Ben mentioned, > or can you reproduce this with a fully software implementation? The production key is on a YubiKey, i.e. a hardware security module. Trying with the test key (software) didn't show the problem. > Provided the latter, that means there exists an input to sign-file that > produces an invalid (shorter) signature, and it's likely we can find another > combination of key/module that also fails, and that can be made public (as > opposed to the Debian production key). I don't have the computing resources > for this, but if we're sure the reproducer exists, someone at LKML might. > > Otherwise I'm afraid you might need to dig a bit deeper. :-) Sadly it looks like this requires more digging. I'll try later :/ At least it is an interesting problem. Ansgar
Bug#942881: Audio on Lenovo X1 Carbon 5th generation stopped working after upgrade to linux-image-5.3.0-1-amd64 ("No response from codec")
Hi Ansgar, On Fri, Oct 25, 2019 at 10:59:09AM +0200, Ansgar wrote: > I tried running `sign-file` manually and can reproduce the truncated > file with Debian's production key. I also tried signing the same key > with a test key instead of the production key: then the signature is 256 > bytes long, just as with any other file... > > `strace -e write sign-file` reports only a single call to `write()` > which writes the entire file in one go. The return value also matches > the number of bytes asked to be written in every case. Cool, thanks for reproducing the issue! Just one question: when you say production key, does that mean a hardware security module like Ben mentioned, or can you reproduce this with a fully software implementation? Provided the latter, that means there exists an input to sign-file that produces an invalid (shorter) signature, and it's likely we can find another combination of key/module that also fails, and that can be made public (as opposed to the Debian production key). I don't have the computing resources for this, but if we're sure the reproducer exists, someone at LKML might. Otherwise I'm afraid you might need to dig a bit deeper. :-) -- Tomáš Janoušek, a.k.a. Pivník, a.k.a. Liskni_si, http://work.lisk.in/
Bug#942881: Audio on Lenovo X1 Carbon 5th generation stopped working after upgrade to linux-image-5.3.0-1-amd64 ("No response from codec")
Tomas Janousek writes: > On Fri, Oct 25, 2019 at 09:45:55AM +0200, Ansgar wrote: >> Tomas Janousek suggested in https://bugs.debian.org/942881#41 that the >> file might be truncated and two bytes missing. I think that might be >> the problem, but with three bytes missing: >> >> src:linux-signed-amd64/5.3.7+1 has for linux-image-5.3.0-1-amd64 a total >> of 3568 detached signatures: one is 1378 bytes (kernel itself), then >> 3566 module signatures at 396 bytes each, then one module signature for >> snd-hda-codec-hdmi.ko.sig which is only 393 bytes. That is very >> suspicious... > > Not really. That's just the ASN.1. For 256 byte octet string, the length field > is one byte longer than for 255 or 254 bytes. Ah, I see: the asn1parse output has hl=2 vs. hl=3. > Yesterday I got one more idea: we've ruled out padding, but maybe a zero byte > in the middle would somehow get lost. So I tried all the ways one could place > two zero bytes into the 254 byte string, and got nothing. I tried running `sign-file` manually and can reproduce the truncated file with Debian's production key. I also tried signing the same key with a test key instead of the production key: then the signature is 256 bytes long, just as with any other file... `strace -e write sign-file` reports only a single call to `write()` which writes the entire file in one go. The return value also matches the number of bytes asked to be written in every case. Ansgar
Bug#942881: Audio on Lenovo X1 Carbon 5th generation stopped working after upgrade to linux-image-5.3.0-1-amd64 ("No response from codec")
Ben Hutchings writes: > The code signing service logs every file it signs, along with a hash of > the detached signature, but I don't know where the logs are so I can't > comapre with that. I checked the audit log, but I don't think it will help much. It currently records that: - 2019-10-21 07:20:03.898781: decided to sign linux-image-5.3.0-1-amd64-unsigned_5.3.7-1_amd64/[...]/snd-hda-codec-hdmi.ko with sha256sum 3fe77a308b28825f0d18717e073b411246aea9bb753f76f6071b3fc4e60c6005 - 2019-10-21 07:20:04.175379: signature for the file logged with sha256sum c2a36f35867ae92b8664f4bd2193e70370eb3b92013ea53f3573d2508d3da4cb (which matches snd-hda-codec-hdmi.ko.sig in src:linux-signed-amd64) So linux' sign-file likely produced a truncated file for some reason; note that ftp-master still uses linux-kbuild-4.9/4.9.189-3+deb9u1. Ansgar
Bug#942881: Audio on Lenovo X1 Carbon 5th generation stopped working after upgrade to linux-image-5.3.0-1-amd64 ("No response from codec")
Hi, On Fri, Oct 25, 2019 at 09:45:55AM +0200, Ansgar wrote: > Tomas Janousek suggested in https://bugs.debian.org/942881#41 that the > file might be truncated and two bytes missing. I think that might be > the problem, but with three bytes missing: > > src:linux-signed-amd64/5.3.7+1 has for linux-image-5.3.0-1-amd64 a total > of 3568 detached signatures: one is 1378 bytes (kernel itself), then > 3566 module signatures at 396 bytes each, then one module signature for > snd-hda-codec-hdmi.ko.sig which is only 393 bytes. That is very > suspicious... Not really. That's just the ASN.1. For 256 byte octet string, the length field is one byte longer than for 255 or 254 bytes. Yesterday I got one more idea: we've ruled out padding, but maybe a zero byte in the middle would somehow get lost. So I tried all the ways one could place two zero bytes into the 254 byte string, and got nothing. -- Tomáš Janoušek, a.k.a. Pivník, a.k.a. Liskni_si, http://work.lisk.in/
Bug#942881: Audio on Lenovo X1 Carbon 5th generation stopped working after upgrade to linux-image-5.3.0-1-amd64 ("No response from codec")
Hi, Ben Hutchings writes: > You can also find the detached signatures in the source package, > linux-signed-amd64. For this module, the signature is: > > debian/signatures/linux-image-5.3.0-1-amd64-unsigned/lib/modules/5.3.0-1-amd64/kernel/sound/pci/hda/snd-hda-codec-hdmi.ko.sig Tomas Janousek suggested in https://bugs.debian.org/942881#41 that the file might be truncated and two bytes missing. I think that might be the problem, but with three bytes missing: src:linux-signed-amd64/5.3.7+1 has for linux-image-5.3.0-1-amd64 a total of 3568 detached signatures: one is 1378 bytes (kernel itself), then 3566 module signatures at 396 bytes each, then one module signature for snd-hda-codec-hdmi.ko.sig which is only 393 bytes. That is very suspicious... > It might be worth > adding verification to the code signing service so we can catch this if > it happens again. We could alternately verify signatures at the point > we attach them to binaries, but that would need to be implemented in > multiple places. Ack; validating the signatures when attaching them might notice when the process of attaching them causes bugs, but I'm not sure how likely that is. But then signing stuff producing truncated files also shouldn't happen... Ansgar
Bug#942881: Audio on Lenovo X1 Carbon 5th generation stopped working after upgrade to linux-image-5.3.0-1-amd64 ("No response from codec")
On Wed, 23 Oct 2019 17:24:22 +0200 Tomas Janousek wrote: > Hi all, > > On Wed, Oct 23, 2019 at 09:24:50AM +0200, Tomas Janousek wrote: > > And there's something horribly wrong with the Debian build of modules, it > > seems. I wonder what else is broken on this system because of a badly signed > > module... :-/ > > I tried to investigate what's wrong with the module signature with some help > from https://unix.stackexchange.com/a/496800/11549: Thanks for taking the time to investigate this! [...] > $ /usr/src/linux-2.6/scripts/extract-module-sig.pl -s > /lib/modules/5.3.0-1-amd64/kernel/sound/pci/hda/snd-hda-codec-hdmi.ko > >snd-hda-codec-hdmi.sig > Read 133153 bytes from module file > Found magic number at 133153 > Found PKCS#7/CMS encapsulation > Found 393 bytes of signature [3082018506092a864886f70d010702a0] > $ openssl asn1parse -inform der -in snd-hda-codec-hdmi.sig | tail -1 > 136:d=5 hl=3 l= 254 prim: OCTET STRING [HEX > DUMP]:8204A68D4EDDC44A0FB89998D7D0FF2A37351C3E9A76D42C015BD868E7A579F95F95C145D61C73F3EC54B314A297DA7E9AE9CACFBD01C0E489E35A3D05F882B01FCEA25754D704F83672841416B69C52760C758A60AF2A38B5706FC02CBEA8AAB1D6F4E82B3C9F0E163B41C0F06E9DFDC166531BBB25BECBFCD3F01097C4ABD32D90A84F3F07DDE5C508F6CCD56CA5BB5467C4D70BB5AC64D432478C4454654F4D77F9DB6EC957F3A70CF818268643F84C64E49DE88624D588B6ED6CCFF2AE8DBF1D8D724BE5EBFCFCAB96E0E47B7B70553268A5BAAC41CBA8AFA8E4A65B5F229038283D99CD2A2D13931D891DF911EF298F8BDC9488E4D07A29AD810330 [...] You can also find the detached signatures in the source package, linux-signed-amd64. For this module, the signature is: debian/signatures/linux-image-5.3.0-1-amd64-unsigned/lib/modules/5.3.0-1-amd64/kernel/sound/pci/hda/snd-hda-codec-hdmi.ko.sig and the content of that seems to match the final signed module. So the error occurred during creation of that source package by our code signing service, not during the following build process. The code signing service logs every file it signs, along with a hash of the detached signature, but I don't know where the logs are so I can't comapre with that. So far as I can see, all file I/O in the code signing process is checked so an error would cause the source package creation to fail. My suspicion is that something may have gone wrong in communication with the HSM which wasn't caught by the driver. It might be worth adding verification to the code signing service so we can catch this if it happens again. We could alternately verify signatures at the point we attach them to binaries, but that would need to be implemented in multiple places. Ben. -- Ben Hutchings Humans are not rational beings; they are rationalising beings. signature.asc Description: This is a digitally signed message part
Bug#942881: Audio on Lenovo X1 Carbon 5th generation stopped working after upgrade to linux-image-5.3.0-1-amd64 ("No response from codec")
Hi again, On Tue, Oct 22, 2019 at 11:59:03PM +0200, Tomas Janousek wrote: > There seems to be a workaround: > > options snd_hda_intel probe_mask=1 > > I'm not getting errors with this and audio works. I have a feeling that HDMI > audio will not work, however, because there's no "input: HDA Intel PCH HDMI" > in dmesg any more. :-( We got a reply from https://bugzilla.kernel.org/show_bug.cgi?id=205293: > Do you have HDMI HD-audio codec driver module available on your system? > The log message (showing "Generic") indicate that it's the generic driver > being bound, not the HDMI codec driver. However unlikely that suggestion sounds, it's spot on: # modprobe snd-hda-codec-hdmi modprobe: ERROR: could not insert 'snd_hda_codec_hdmi': Key was rejected by service Indeed, when I modprobe --force snd-hda-codec-hdmi and then rmmod/modprobe snd-hda-intel, everything's fine again, including HDMI profiles. So the new modprobe.d workaround is: install snd-hda-codec-hdmi /sbin/modprobe --ignore-install --force-modversion snd-hda-codec-hdmi And there's something horribly wrong with the Debian build of modules, it seems. I wonder what else is broken on this system because of a badly signed module... :-/ -- Tomáš Janoušek, a.k.a. Pivník, a.k.a. Liskni_si, http://work.lisk.in/
Bug#942881: Audio on Lenovo X1 Carbon 5th generation stopped working after upgrade to linux-image-5.3.0-1-amd64 ("No response from codec")
Hi,
On Tue, Oct 22, 2019 at 09:47:32PM +0300, Josh Triplett wrote:
> After upgrading to linux-image-5.3.0-1-amd64, I get messages like those
> seen in the log ("No response from codec, resetting bus" and "Unable to
> sync register"), and eventually a WARN. The mixer doesn't show the audio
> device at all.
>
> This didn't happen with the previous kernel, namely:
> Oct 21 21:04:28 s kernel: Linux version 5.2.0-3-amd64
> (debian-kernel@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-23)) #1 SMP
> Debian 5.2.17-1 (2019-10-06)
I can confirm the same behaviour with ThinkPad T25.
There seems to be a workaround:
options snd_hda_intel probe_mask=1
I'm not getting errors with this and audio works. I have a feeling that HDMI
audio will not work, however, because there's no "input: HDA Intel PCH HDMI"
in dmesg any more. :-(
--
Tomáš Janoušek, a.k.a. Pivník, a.k.a. Liskni_si, http://work.lisk.in/
Bug#942881: Audio on Lenovo X1 Carbon 5th generation stopped working after upgrade to linux-image-5.3.0-1-amd64 ("No response from codec")
Package: src:linux
Version: 5.3.7-1
Severity: normal
After upgrading to linux-image-5.3.0-1-amd64, I get messages like those
seen in the log ("No response from codec, resetting bus" and "Unable to
sync register"), and eventually a WARN. The mixer doesn't show the audio
device at all.
This didn't happen with the previous kernel, namely:
Oct 21 21:04:28 s kernel: Linux version 5.2.0-3-amd64
(debian-kernel@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-23)) #1 SMP
Debian 5.2.17-1 (2019-10-06)
-- Package-specific info:
** Version:
Linux version 5.3.0-1-amd64 (debian-kernel@lists.debian.org) (gcc version 9.2.1
20191008 (Debian 9.2.1-9)) #1 SMP Debian 5.3.7-1 (2019-10-19)
** Command line:
BOOT_IMAGE=/vmlinuz-5.3.0-1-amd64
root=UUID=121586a4-bf8c-49a4-9a72-ed70fcf72772 ro quiet
** Tainted: W (512)
* Taint on warning.
** Kernel log:
[ 59.442968] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20170500
[ 60.448988] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20270500
[ 61.454722] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20370500
[ 62.460306] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x201f0500
[ 63.493654] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20170500
[ 64.498615] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20270500
[ 65.507773] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20370500
[ 66.516498] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x201f0500
[ 67.557305] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20170500
[ 68.569978] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20270500
[ 69.578573] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20370500
[ 70.587072] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x201f0500
[ 71.627548] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20170500
[ 72.635462] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20270500
[ 73.639361] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20370500
[ 74.651383] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x201f0500
[ 75.687274] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20170500
[ 76.691051] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20270500
[ 77.694902] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20370500
[ 78.702874] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x201f0500
[ 79.734778] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20170500
[ 80.738684] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20270500
[ 81.742564] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20370500
[ 82.746472] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x201f0500
[ 83.786414] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20170500
[ 84.798403] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20270500
[ 85.810293] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20370500
[ 86.822260] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x201f0500
[ 87.862207] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20170500
[ 88.866163] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20270500
[ 89.874103] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20370500
[ 90.885985] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x201f0500
[ 91.926013] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20170500
[ 92.929911] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20270500
[ 93.938042] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20370500
[ 94.945841] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x201f0500
[ 95.981858] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20370740
[ 96.985920] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20370740
[ 97.997872] snd_hda_intel :00:1f.3: No response from codec, resetting
bus: last cmd=0x20270d80
[ 97.997891] snd_hda_codec_generic hdaudioC0D2: Unable to sync register
0x2f0d00. -11
[ 99.009847] snd_hda_intel
