Bug#908453: xen-utils-common: README.comet seems to no longer apply to the current Xen packages
tags 908453 + pending thanks Hi Keith, I added a change in the packaging that removed the misleading text about PVH from the README.comet: https://salsa.debian.org/xen-team/debian-xen/commit/f7f02e1b2ba0dd6464b53f134f9ca13c5a5550e8 Hans
Bug#908453: xen-utils-common: README.comet seems to no longer apply to the current Xen packages
Hi Keith, Thanks for your elaborate report. I'll context-quote to try answer a few things below. Summary of the info below: * PVH is not supposed to work with Xen 4.8. * The documentation in README.comet is wrong for the recent Xen 4.8-stable that we ship now, we should fix that. * For 4.8 and PV guests, XPTI is also available now (and enabled by default). * Silently ignoring invalid configuration is bad. On 09/10/2018 05:51 AM, Keith Bare wrote: > Package: xen-utils-common > Version: 4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u9 > Severity: important > > Dear Maintainer, > >* What led up to the situation? > > I needed to build a new Xen DomU. Since I had seen that Debian had picked > up the 4.8 "comet" changes and pvshim, I wanted to experiment with Xen's PVH > mode and have the DomU run the Linux 4.17 kernel in stretch-backports, which > should be new enough to support PVH. Parts of the PVH code have been backported. Running PVH guests yourself is still not supported or recommended with Xen 4.8. If it was temporarily, in the shim-branch, it was still not recommended. > It seemed like this should work, given instructions in > /usr/share/doc/xen-utils-common/README.comet > > * Converting a PV config to a PVH config > > If you have a kernel capable of booting PVH, then PVH mode is both > faster and more secure than PV or PVH-shim mode. > > - Remove any reference to 'builder' (e.g., `builder="generic"`) > - Add the following line: > type="pvh" > We just had a little conversation about this in #debian-xen on oftc irc, and the latest info is that the documentation is (probably) wrong. The README.comet is not present in upstream 4.8, but was copied from the security advisory into our packaging. XSA-254 mentions "PVH is supported in Xen 4.8 only with the 4.8 "Comet" security release branch." https://xenbits.xen.org/xsa/advisory-254.html However... >* What exactly did you do (or not do) that was effective (or > ineffective)? > > I ran xen-create-image (from the xen-tools package) with --dist=stretch to > build the DomU and its configuration. I also used --pygrub so Xen would > boot the kernel installed in the DomU's filesystem. > > I temporarily edited the DomU's Xen/xl configuration to use the pvshim, per > the instructions in README.comet (type = 'pvh' / pvshim = 1), started the > DomU, added the stretch-backports repository, and installed > linux-image-amd64 from backports. I then shut down the DomU, edited the > DomU's configuration to remove the pvshim = 1 line, and re-started the DomU. > > >* What was the outcome of this action? > > It appeared the DomU was running in PV mode, despite my having added type = > 'pvh' to its Xen/xl configuration. > > It was difficult to tell, but I believed the DomU was running in PV mode as > its kernel printed: > > - [0.00] Hypervisor detected: Xen PV > - [0.00] Kernel/User page tables isolation: disabled on XEN PV. Yes, that's PV. > Some Internet searching also seemed to indicate that "xl list -l " > should mention "pvh" if the DomU is running in PVH mode. However, I only > saw: > > - "type": "pv", > > There was no mention of "pvh" in the output. > > >* What outcome did you expect instead? > > I expected the DomU to run in PVH mode. I wasn't sure what this was > supposed to look like, but I did some experiments (installed the old > 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 packages and worked around > bootloader being broken for PVH) and saw that in fact, with that version of > the Xen packages, the DomU kernel prints: > > - [0.00] Hypervisor detected: Xen HVM > - [0.00] Booting paravirtualized kernel on Xen PVH > - [0.00] Kernel/User page tables isolation: enabled > > And xl list -l shows: > > - "type": "pvh", > > >* More thoughts/discussion. > > It looks like the Debian packages lost support for booting DomUs in PVH mode > with version 4.8.3+xsa262+shim4.10.0+comet3-1+deb9u6. Probably because: > > Update to new upstream version 4.8.3+xsa262+shim4.10.0+comet3. > (This is the upstream staging-4.8 branch, which is ahead of the > upstream CI-tested stable-4.8 branch by precisely the three > most recent XSA fixes. We are switching away from the special > upstream 4.8 comet branch.) ...yes, that would be the missing puzzle piece which explains it. So in that case, the README.comet which is still in our package is misleading and incorrect. > And maybe that's fine... if the mitigation comet and the pvshim provided is > also effectively provided by XPTI changes that were present in the > stable-4.8 branch, then I guess it isn't really necessary for anybody to use > (and thus PVH-mode boot) the shim anymore. Yes, using XPTI for PV would be the least bad thing now for 4.8 I guess. It's enabled by default by the way, so if you didn't see huge performance problems in your dom0, you might as well try to run PV guests just like before
Bug#908453: xen-utils-common: README.comet seems to no longer apply to the current Xen packages
Package: xen-utils-common Version: 4.8.3+xsa267+shim4.10.1+xsa267-1+deb9u9 Severity: important Dear Maintainer, * What led up to the situation? I needed to build a new Xen DomU. Since I had seen that Debian had picked up the 4.8 "comet" changes and pvshim, I wanted to experiment with Xen's PVH mode and have the DomU run the Linux 4.17 kernel in stretch-backports, which should be new enough to support PVH. It seemed like this should work, given instructions in /usr/share/doc/xen-utils-common/README.comet * Converting a PV config to a PVH config If you have a kernel capable of booting PVH, then PVH mode is both faster and more secure than PV or PVH-shim mode. - Remove any reference to 'builder' (e.g., `builder="generic"`) - Add the following line: type="pvh" * What exactly did you do (or not do) that was effective (or ineffective)? I ran xen-create-image (from the xen-tools package) with --dist=stretch to build the DomU and its configuration. I also used --pygrub so Xen would boot the kernel installed in the DomU's filesystem. I temporarily edited the DomU's Xen/xl configuration to use the pvshim, per the instructions in README.comet (type = 'pvh' / pvshim = 1), started the DomU, added the stretch-backports repository, and installed linux-image-amd64 from backports. I then shut down the DomU, edited the DomU's configuration to remove the pvshim = 1 line, and re-started the DomU. * What was the outcome of this action? It appeared the DomU was running in PV mode, despite my having added type = 'pvh' to its Xen/xl configuration. It was difficult to tell, but I believed the DomU was running in PV mode as its kernel printed: - [0.00] Hypervisor detected: Xen PV - [0.00] Kernel/User page tables isolation: disabled on XEN PV. Some Internet searching also seemed to indicate that "xl list -l " should mention "pvh" if the DomU is running in PVH mode. However, I only saw: - "type": "pv", There was no mention of "pvh" in the output. * What outcome did you expect instead? I expected the DomU to run in PVH mode. I wasn't sure what this was supposed to look like, but I did some experiments (installed the old 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5 packages and worked around bootloader being broken for PVH) and saw that in fact, with that version of the Xen packages, the DomU kernel prints: - [0.00] Hypervisor detected: Xen HVM - [0.00] Booting paravirtualized kernel on Xen PVH - [0.00] Kernel/User page tables isolation: enabled And xl list -l shows: - "type": "pvh", * More thoughts/discussion. It looks like the Debian packages lost support for booting DomUs in PVH mode with version 4.8.3+xsa262+shim4.10.0+comet3-1+deb9u6. Probably because: Update to new upstream version 4.8.3+xsa262+shim4.10.0+comet3. (This is the upstream staging-4.8 branch, which is ahead of the upstream CI-tested stable-4.8 branch by precisely the three most recent XSA fixes. We are switching away from the special upstream 4.8 comet branch.) And maybe that's fine... if the mitigation comet and the pvshim provided is also effectively provided by XPTI changes that were present in the stable-4.8 branch, then I guess it isn't really necessary for anybody to use (and thus PVH-mode boot) the shim anymore. However, if that's the case, then it probably doesn't make sense to include README.comet and continue packaging the shim in the Debian packages anymore. The thing that's nefarious (and could be grounds for increasing the bug severity), is that anybody that followed the README.comet instructions and configured DomUs to boot a PVH-capable kernel without the shim is now, probably to their surprise, running their DomU in PV mode. This means they've lost Linux's KPTI protections from Meltdown within their DomU. The underlying issue is that the xl command seems to silently ignore configuration directives it doesn't understand--which, without the 4.8 comet2 changes, includes 'type'! This isn't a huge deal for me and my deployment (at the CMU Computer Club). Our Xen infrastructure was running jessie/Xen 4.4 at best when the Meltdown/Spectre news broke. So our initial mitigation was to switch everything to run in HVM mode (and we've continued doing so since then). I was interested in exploring PVH mode though, since it looked like it was more similar to PV mode in some ways that would make it work better with various tooling (e.g., xen-tools). The fact it didn't work with the Debian packages the way I thought it would was surprising, and I figured it might be surprising for other people too. --Keith Bare -- System Information: Debian Release: 9.5 APT prefers stable-updates APT policy: (500, 'stable-updates'), (500, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 4.9.0-7-amd64 (SMP w/16 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8) Shell:
