** No longer affects: ~juergh/ubuntu/linux-firmware-hwe
** No longer affects: ~juergh/ubuntu/linux-firmware-hwe/tenacious
** Also affects: linux-firmware-hwe (Ubuntu)
Importance: Undecided
Status: New
** No longer affects: linux-firmware (Ubuntu)
** No longer affects: linux-firmware (Ubuntu Tenacious)
** Also affects: linux-firmware-hwe (Ubuntu Stonking)
Importance: Undecided
Status: New
** Also affects: linux-firmware-hwe (Ubuntu Tenacious)
Importance: Undecided
Status: New
** Also affects: linux-firmware-hwe (Ubuntu Resolute)
Importance: Undecided
Status: New
** Description changed:
[ Impact ]
Backporting new firmware for HWE kernels to LTSes is a constant source of pain
and always results in problems. To limit the risk we always try to backport
only select firmwares that we think are safe but we never get it 100% correct.
The main issues are typically:
1. Some backported firmware that the HWE kernel needs is also loaded by the GA
- kernel which can cause issues for GA kernel users. New firmware with old
- kernels is a combination that firmware vendors typically don't test and is
- not really supported.
+ kernel which can cause issues for GA kernel users. New firmware with old
+ kernels is a combination that firmware vendors typically don't test and is
+ not really supported.
2. Missing HWE firmwares. Finding the exact firmwares that a particular kernel
- wants is not trivial. We're constantly missing blobs so when LTS users
- upgrade to a newer HWE kernel they sometimes experience regressions and HW
- that used to work no longer does because firmware is missing. We also
- deliberately exclude firmwares that we *think* might cause issues for GA
- kernel users.
+ wants is not trivial. We're constantly missing blobs so when LTS users
+ upgrade to a newer HWE kernel they sometimes experience regressions and HW
+ that used to work no longer does because firmware is missing. We also
+ deliberately exclude firmwares that we *think* might cause issues for GA
+ kernel users.
3. Identifying and backporting firmwares is an expensive task and the results
- are never satisfying for both the firmware package maintainers and (more
- importantly) the users.
+ are never satisfying for both the firmware package maintainers and (more
+ importantly) the users.
+
+ Note that this is an umbrella bug report that covers all subpackages as well.
The full list is:
+ linux-firmware-hwe
+ linux-firmware-hwe-amd-graphics
+ linux-firmware-hwe-amd-misc
+ linux-firmware-hwe-broadcom-wireless
+ linux-firmware-hwe-intel-graphics
+ linux-firmware-hwe-intel-misc
+ linux-firmware-hwe-intel-wireless
+ linux-firmware-hwe-marvell-prestera
+ linux-firmware-hwe-marvell-wireless
+ linux-firmware-hwe-mediatek
+ linux-firmware-hwe-mellanox-spectrum
+ linux-firmware-hwe-misc
+ linux-firmware-hwe-netronome
+ linux-firmware-hwe-nvidia-graphics
+ linux-firmware-hwe-qlogic
+ linux-firmware-hwe-qualcomm-graphics
+ linux-firmware-hwe-qualcomm-misc
+ linux-firmware-hwe-qualcomm-wireless
+ linux-firmware-hwe-realtek
[ Fix ]
Introduce a new set of firmware packages: linux-firmware-hwe-<foo>. The idea
is that HWE kernels depend on linux-firmware-hwe (instead of linux-firmware)
which installs the same firmware binaries as the Ubuntu release where the HWE
kernel comes from. Take Resolute as an example. The first HWE kernel for
Resolute will be 7.3 from Stonking. The new set of linux-firmware-hwe for
Resolute will be a backport of linux-firmware from Stonking. The full set, no
backporting of individual blobs. That way we have the same combination of
kernel+firmware in both the interim and the LTS releases.
<foo> mirrors the linux-firmware subpackage split, e.g.
linux-firmware-hwe-amd-graphics vs linux-firmware-amd-graphics, with
linux-firmware-hwe as the umbrella metapackage pulling in the subpackage set.
There is only one HWE firmware set per release, not one per HWE kernel series.
Lifecycle per series:
* Interim releases: linux-firmware-hwe and every linux-firmware-hwe-<foo> are
- metapackages depending on their linux-firmware-<foo> counterparts. There is
- nothing to backport for interim releases (their GA kernel is the newest
- kernel) so the package names only exist to keep the namespace and dependency
- chain uniform across all series.
+ metapackages depending on their linux-firmware-<foo> counterparts. There is
+ nothing to backport for interim releases (their GA kernel is the newest
+ kernel) so the package names only exist to keep the namespace and dependency
+ chain uniform across all series.
* LTS at release time: an LTS is the newest release when it comes out, so
- there is no HWE yet. The HWE kernel is just a metapackage depending on the
- GA kernel and linux-firmware-hwe follows suit and is also just a metapackage
- depending on linux-firmware.
+ there is no HWE yet. The HWE kernel is just a metapackage depending on the
+ GA kernel and linux-firmware-hwe follows suit and is also just a metapackage
+ depending on linux-firmware.
* LTS once the first real HWE kernel lands: the HWE kernel metapackage is
- replaced by a real kernel package and linux-firmware-hwe-<foo> is replaced
- by real packages carrying the full firmware set from the HWE kernel's
- origin release. Real packages that provide firmware blobs only ever exist
- in LTS releases (Resolute and newer).
+ replaced by a real kernel package and linux-firmware-hwe-<foo> is replaced
+ by real packages carrying the full firmware set from the HWE kernel's
+ origin release. Real packages that provide firmware blobs only ever exist
+ in LTS releases (Resolute and newer).
That way anything depending on linux-firmware-hwe-<foo> (e.g. HWE kernel
packages) resolves correctly on every series without per-series special
casing, and identical kernel+firmware combinations are guaranteed for both
the interim and the LTS releases.
The caveat with this is that it's not possible to have both linux-firmware
and linux-firmware-hwe installed at the same time (we might work around that
in the future). So you either have the firmwares for the GA kernel *or* the
firmwares for the HWE kernel. But you can have both GA *and* HWE kernels
installed at the same time. In that case and if you choose to boot the GA
kernel you'll run old kernel plus new firmware which could be problematic.
Because the real linux-firmware-hwe-<foo> packages replace metapackages that
previously pulled in the linux-firmware-<foo> counterparts, they will
Conflict/Replace (and possibly Provide) those to handle the upgrade cleanly.
[ Test Case ]
All verifications are package-level: package states via dpkg/apt, file lists
via dpkg -L, file contents via dpkg -V or debsums, and dependency resolution
via apt-get -s dry runs.
Note: no released LTS carries these packages yet, so the cases below need to
be run against test PPAs containing fake packages that mimic the real ones.
1. Dependency resolution
- kernel/firmware: GA/GA -> GA/HWE-shim
-
- Install the HWE kernel metapackage. Confirm that it Depends on
- linux-firmware-hwe and that linux-firmware-hwe resolves to
- linux-firmware. Complete the install and verify the expected package
- set is present.
-
- Steps:
- a. Start from a system with the GA kernel and linux-firmware
- installed and no HWE packages present.
- b. Dry-run: apt-get -s install <hwe-kernel-meta>
- Expected: resolves <hwe-kernel-meta> -> linux-firmware-hwe ->
- linux-firmware, with no linux-firmware removal.
- c. Install: apt-get install <hwe-kernel-meta>
- d. Verify: dpkg -l shows the HWE kernel metapackage,
- linux-firmware-hwe and linux-firmware-hwe-<foo> subpackages all
- installed, and linux-firmware still installed (not replaced).
- e. Verify: dpkg -L linux-firmware-hwe shows no files (pure
- metapackage) and the linux-firmware-hwe-<foo> packages have no
- file payload of their own.
+ kernel/firmware: GA/GA -> GA/HWE-shim
+
+ Install the HWE kernel metapackage. Confirm that it Depends on
+ linux-firmware-hwe and that linux-firmware-hwe resolves to
+ linux-firmware. Complete the install and verify the expected package
+ set is present.
+
+ Steps:
+ a. Start from a system with the GA kernel and linux-firmware
+ installed and no HWE packages present.
+ b. Dry-run: apt-get -s install <hwe-kernel-meta>
+ Expected: resolves <hwe-kernel-meta> -> linux-firmware-hwe ->
+ linux-firmware, with no linux-firmware removal.
+ c. Install: apt-get install <hwe-kernel-meta>
+ d. Verify: dpkg -l shows the HWE kernel metapackage,
+ linux-firmware-hwe and linux-firmware-hwe-<foo> subpackages all
+ installed, and linux-firmware still installed (not replaced).
+ e. Verify: dpkg -L linux-firmware-hwe shows no files (pure
+ metapackage) and the linux-firmware-hwe-<foo> packages have no
+ file payload of their own.
2. File inventory parity
- kernel/firmware: GA/HWE-shim (unchanged)
-
- Verify that the installed linux-firmware-hwe set is content-identical
- to a stock linux-firmware install.
-
- Steps:
- a. Compare the dpkg -L file lists of every linux-firmware-hwe-<foo>
- package against its linux-firmware-<foo> counterpart. The lists
- must be identical.
- b. Verify file checksums of the installed files (dpkg -V or
- debsums). All files must pass.
+ kernel/firmware: GA/HWE-shim (unchanged)
+
+ Verify that the installed linux-firmware-hwe set is content-identical
+ to a stock linux-firmware install.
+
+ Steps:
+ a. Compare the dpkg -L file lists of every linux-firmware-hwe-<foo>
+ package against its linux-firmware-<foo> counterpart. The lists
+ must be identical.
+ b. Verify file checksums of the installed files (dpkg -V or
+ debsums). All files must pass.
3. Removal path
- kernel/firmware: GA/HWE-shim -> GA/GA
-
- Purge the HWE kernel metapackage and linux-firmware-hwe and verify
- that the GA firmware set is not taken down with it.
-
- Steps:
- a. apt-get purge <hwe-kernel-meta> linux-firmware-hwe
- b. Dry-run: apt-get -s autoremove
- Expected: linux-firmware and its subpackages are NOT candidates
- for removal.
- c. Verify: all firmware files from dpkg -L linux-firmware are still
- present on disk.
+ kernel/firmware: GA/HWE-shim -> GA/GA
+
+ Purge the HWE kernel metapackage and linux-firmware-hwe and verify
+ that the GA firmware set is not taken down with it.
+
+ Steps:
+ a. apt-get purge <hwe-kernel-meta> linux-firmware-hwe
+ b. Dry-run: apt-get -s autoremove
+ Expected: linux-firmware and its subpackages are NOT candidates
+ for removal.
+ c. Verify: all firmware files from dpkg -L linux-firmware are still
+ present on disk.
4. Freshly released LTS
- kernel/firmware: GA/GA -> GA/HWE-shim
-
- Repeat case 1 on a freshly released LTS, where linux-firmware-hwe is
- also just a metapackage.
-
- Steps:
- a. On a fresh LTS install: apt-get -s install <hwe-kernel-meta>
- shows the same resolution as in case 1.
- b. Install and verify the package set as in case 1.
+ kernel/firmware: GA/GA -> GA/HWE-shim
+
+ Repeat case 1 on a freshly released LTS, where linux-firmware-hwe is
+ also just a metapackage.
+
+ Steps:
+ a. On a fresh LTS install: apt-get -s install <hwe-kernel-meta>
+ shows the same resolution as in case 1.
+ b. Install and verify the package set as in case 1.
5. Negative dependency test
- kernel/firmware: GA/GA (unchanged)
-
- Verify that GA kernel packages still depend on linux-firmware, not on
- linux-firmware-hwe.
-
- Steps:
- a. Inspect the GA kernel metapackage dependencies (dpkg -I or
- apt-cache show). The linux-firmware dependency must point at
- linux-firmware, not linux-firmware-hwe.
- b. Dry-run: apt-get -s install <ga-kernel-meta>
- Expected: the resolution does not include linux-firmware-hwe.
+ kernel/firmware: GA/GA (unchanged)
+
+ Verify that GA kernel packages still depend on linux-firmware, not on
+ linux-firmware-hwe.
+
+ Steps:
+ a. Inspect the GA kernel metapackage dependencies (dpkg -I or
+ apt-cache show). The linux-firmware dependency must point at
+ linux-firmware, not linux-firmware-hwe.
+ b. Dry-run: apt-get -s install <ga-kernel-meta>
+ Expected: the resolution does not include linux-firmware-hwe.
6. Opt-in install
- kernel/firmware: GA/GA -> HWE/HWE-real
-
- On an LTS with a real HWE kernel available, switching to the HWE
- kernel replaces linux-firmware with the real linux-firmware-hwe set.
-
- Steps:
- a. Start from a system with the GA kernel and linux-firmware
- installed. The HWE kernel and real linux-firmware-hwe packages
- are available from the test PPA.
- b. Dry-run: apt-get -s install <hwe-kernel-meta>
- Expected: linux-firmware is replaced by the real
- linux-firmware-hwe set, with no file conflicts reported.
- c. Install and verify the final state: the real linux-firmware-hwe
- set is installed, linux-firmware is gone, and dpkg --audit
- reports no half-configured packages.
+ kernel/firmware: GA/GA -> HWE/HWE-real
+
+ On an LTS with a real HWE kernel available, switching to the HWE
+ kernel replaces linux-firmware with the real linux-firmware-hwe set.
+
+ Steps:
+ a. Start from a system with the GA kernel and linux-firmware
+ installed. The HWE kernel and real linux-firmware-hwe packages
+ are available from the test PPA.
+ b. Dry-run: apt-get -s install <hwe-kernel-meta>
+ Expected: linux-firmware is replaced by the real
+ linux-firmware-hwe set, with no file conflicts reported.
+ c. Install and verify the final state: the real linux-firmware-hwe
+ set is installed, linux-firmware is gone, and dpkg --audit
+ reports no half-configured packages.
7. Origin parity
- kernel/firmware: HWE/HWE-real (unchanged)
-
- Verify that the real linux-firmware-hwe set matches the firmware set
- of the HWE kernel's origin release exactly.
-
- Steps:
- a. In a chroot or container of the origin release, install its
- linux-firmware-<foo> packages and capture their file lists
- (dpkg -L) and checksums (/var/lib/dpkg/info/*.md5sums).
- b. Compare against the target's linux-firmware-hwe-<foo> packages:
- identical file lists and checksums, no missing and no extra
- blobs.
+ kernel/firmware: HWE/HWE-real (unchanged)
+
+ Verify that the real linux-firmware-hwe set matches the firmware set
+ of the HWE kernel's origin release exactly.
+
+ Steps:
+ a. In a chroot or container of the origin release, install its
+ linux-firmware-<foo> packages and capture their file lists
+ (dpkg -L) and checksums (/var/lib/dpkg/info/*.md5sums).
+ b. Compare against the target's linux-firmware-hwe-<foo> packages:
+ identical file lists and checksums, no missing and no extra
+ blobs.
8. Co-install conflict
- kernel/firmware: HWE/HWE-real (unchanged, refused)
-
- Verify that linux-firmware and linux-firmware-hwe cannot be
- co-installed.
-
- Steps:
- a. With the real HWE set installed, dry-run: apt-get -s install
- linux-firmware. Expected: the resolver refuses due to the
- conflict.
- b. apt-get install linux-firmware linux-firmware-hwe: must fail
- cleanly, leaving dpkg --audit clean.
+ kernel/firmware: HWE/HWE-real (unchanged, refused)
+
+ Verify that linux-firmware and linux-firmware-hwe cannot be
+ co-installed.
+
+ Steps:
+ a. With the real HWE set installed, dry-run: apt-get -s install
+ linux-firmware. Expected: the resolver refuses due to the
+ conflict.
+ b. apt-get install linux-firmware linux-firmware-hwe: must fail
+ cleanly, leaving dpkg --audit clean.
9. Kernel coexistence
- kernel/firmware: HWE/HWE-real (unchanged)
-
- Verify that the GA kernel package survives the firmware switch and
- record how its linux-firmware dependency is handled.
-
- Steps:
- a. After case 6: verify that the GA kernel package is still
- installed and intact (dpkg -l).
- b. Inspect how the GA kernel's dependency on linux-firmware is
- satisfied once linux-firmware is gone (e.g. via Provides from
- the HWE packages). Record the outcome; it must not leave any
- dpkg --audit findings.
+ kernel/firmware: HWE/HWE-real (unchanged)
+
+ Verify that the GA kernel package survives the firmware switch and
+ record how its linux-firmware dependency is handled.
+
+ Steps:
+ a. After case 6: verify that the GA kernel package is still
+ installed and intact (dpkg -l).
+ b. Inspect how the GA kernel's dependency on linux-firmware is
+ satisfied once linux-firmware is gone (e.g. via Provides from
+ the HWE packages). Record the outcome; it must not leave any
+ dpkg --audit findings.
10. Metapackage -> real, via -updates
- kernel/firmware: GA/HWE-shim -> HWE/HWE-real
-
- An LTS system with the metapackage-era linux-firmware-hwe installed
- receives the real HWE kernel and real firmware packages via
- -updates.
-
- Steps:
- a. Start from an LTS system in the GA/HWE-shim state.
- b. Enable the test PPA with the real HWE kernel and real firmware
- packages. Dry-run: apt-get -s dist-upgrade
- Expected: the linux-firmware-<foo> counterparts are removed,
- the real HWE set and real HWE kernel are installed, no
- conflicts reported.
- c. Apply the upgrade and verify: no dpkg errors, and no stale
- firmware files under /usr/lib/firmware belonging to the removed
- counterparts (compare dpkg -L output against the on-disk file
- inventory).
+ kernel/firmware: GA/HWE-shim -> HWE/HWE-real
+
+ An LTS system with the metapackage-era linux-firmware-hwe installed
+ receives the real HWE kernel and real firmware packages via
+ -updates.
+
+ Steps:
+ a. Start from an LTS system in the GA/HWE-shim state.
+ b. Enable the test PPA with the real HWE kernel and real firmware
+ packages. Dry-run: apt-get -s dist-upgrade
+ Expected: the linux-firmware-<foo> counterparts are removed,
+ the real HWE set and real HWE kernel are installed, no
+ conflicts reported.
+ c. Apply the upgrade and verify: no dpkg errors, and no stale
+ firmware files under /usr/lib/firmware belonging to the removed
+ counterparts (compare dpkg -L output against the on-disk file
+ inventory).
11. Real -> metapackage (reversal)
- kernel/firmware: HWE/HWE-real -> GA/GA
-
- Verify a clean revert from the real HWE arrangement back to the GA
- firmware set.
-
- Steps:
- a. Remove the real HWE kernel and the linux-firmware-hwe set.
- b. Install linux-firmware.
- c. Verify: complete file inventory restored (dpkg -L vs on-disk),
- checksums pass (dpkg -V), dpkg --audit clean.
+ kernel/firmware: HWE/HWE-real -> GA/GA
+
+ Verify a clean revert from the real HWE arrangement back to the GA
+ firmware set.
+
+ Steps:
+ a. Remove the real HWE kernel and the linux-firmware-hwe set.
+ b. Install linux-firmware.
+ c. Verify: complete file inventory restored (dpkg -L vs on-disk),
+ checksums pass (dpkg -V), dpkg --audit clean.
Release upgrades -- one case per source -> target state transition of
linux-firmware-hwe. With no HWE packages installed, every path is a plain
linux-firmware release upgrade and serves as the shared baseline.
12. Shim -> shim
- kernel/firmware: (rel A) GA/HWE-shim -> (rel B) GA/HWE-shim
-
- E.g. interim -> interim. Metapackage arrangements on both sides, each
- resolving to its own release's linux-firmware.
-
- Steps:
- a. Start from a system in the GA/HWE-shim state on release A.
- b. do-release-upgrade to release B.
- Expected: the linux-firmware-hwe metapackages are present on
- both sides, the target release's linux-firmware is installed,
- no conflicts reported.
- c. Verify the final package set and file inventory on release B.
+ kernel/firmware: (rel A) GA/HWE-shim -> (rel B) GA/HWE-shim
+
+ E.g. interim -> interim. Metapackage arrangements on both sides, each
+ resolving to its own release's linux-firmware.
+
+ Steps:
+ a. Start from a system in the GA/HWE-shim state on release A.
+ b. do-release-upgrade to release B.
+ Expected: the linux-firmware-hwe metapackages are present on
+ both sides, the target release's linux-firmware is installed,
+ no conflicts reported.
+ c. Verify the final package set and file inventory on release B.
13. Shim -> real
- kernel/firmware: (interim) GA/HWE-shim -> (LTS) HWE/HWE-real
-
- Interim -> LTS after the target's HWE kernel has landed. The
- metapackage arrangement is replaced by the real packages and the
- real HWE kernel.
-
- Steps:
- a. Start from an interim system in the GA/HWE-shim state.
- b. do-release-upgrade to the LTS.
- Expected: the real linux-firmware-hwe set and real HWE kernel
- are installed, no conflicts reported.
- c. Verify the final package state as in case 6.
+ kernel/firmware: (interim) GA/HWE-shim -> (LTS) HWE/HWE-real
+
+ Interim -> LTS after the target's HWE kernel has landed. The
+ metapackage arrangement is replaced by the real packages and the
+ real HWE kernel.
+
+ Steps:
+ a. Start from an interim system in the GA/HWE-shim state.
+ b. do-release-upgrade to the LTS.
+ Expected: the real linux-firmware-hwe set and real HWE kernel
+ are installed, no conflicts reported.
+ c. Verify the final package state as in case 6.
14. Real -> shim
- kernel/firmware: (LTS) HWE/HWE-real -> (interim/LTS) GA/HWE-shim
-
- LTS -> interim, and LTS -> LTS before the target's HWE kernel has
- landed. Run on both paths. The source carries real packages that
- Conflict with linux-firmware while the target is metapackage-era and
- wants linux-firmware.
-
- Steps:
- a. Start from an LTS system in the HWE/HWE-real state.
- b. do-release-upgrade to the target release.
- Expected: the real HWE set is removed, the target's
- linux-firmware is installed, and the upgrade completes without
- conflicts.
- c. Verify that no firmware files are lost (file inventory vs the
- target release's linux-firmware).
+ kernel/firmware: (LTS) HWE/HWE-real -> (interim/LTS) GA/HWE-shim
+
+ LTS -> interim, and LTS -> LTS before the target's HWE kernel has
+ landed. Run on both paths. The source carries real packages that
+ Conflict with linux-firmware while the target is metapackage-era and
+ wants linux-firmware.
+
+ Steps:
+ a. Start from an LTS system in the HWE/HWE-real state.
+ b. do-release-upgrade to the target release.
+ Expected: the real HWE set is removed, the target's
+ linux-firmware is installed, and the upgrade completes without
+ conflicts.
+ c. Verify that no firmware files are lost (file inventory vs the
+ target release's linux-firmware).
15. Real -> real
- kernel/firmware: (LTS A) HWE/HWE-real -> (LTS B) HWE/HWE-real
-
- LTS -> LTS after the target's HWE kernel has landed. The source's
- real set is replaced by the target's real set. This is a late-cycle
- case that only becomes testable once the target LTS has its own HWE
- kernel.
-
- Steps:
- a. Start from an LTS system in the HWE/HWE-real state on release A.
- b. do-release-upgrade to release B (which already has real HWE
- packages).
- Expected: the source's real set is replaced by release B's real
- set, no conflicts reported.
- c. Verify the final package state as in cases 6 and 7.
+ kernel/firmware: (LTS A) HWE/HWE-real -> (LTS B) HWE/HWE-real
+
+ LTS -> LTS after the target's HWE kernel has landed. The source's
+ real set is replaced by the target's real set. This is a late-cycle
+ case that only becomes testable once the target LTS has its own HWE
+ kernel.
+
+ Steps:
+ a. Start from an LTS system in the HWE/HWE-real state on release A.
+ b. do-release-upgrade to release B (which already has real HWE
+ packages).
+ Expected: the source's real set is replaced by release B's real
+ set, no conflicts reported.
+ c. Verify the final package state as in cases 6 and 7.
[ Where Problems Could Occur ]
* Incorrect or missing Conflicts/Replaces/Provides relationships between the
- real linux-firmware-hwe-<foo> packages and their linux-firmware-<foo>
- counterparts could cause file conflicts, failed upgrades, or a
- half-installed package state. Worst case, a system ends up with no firmware
- installed and fails to boot or loses hardware functionality after a
- reboot. Covered by test cases 6, 8, 10 and 14.
+ real linux-firmware-hwe-<foo> packages and their linux-firmware-<foo>
+ counterparts could cause file conflicts, failed upgrades, or a
+ half-installed package state. Worst case, a system ends up with no firmware
+ installed and fails to boot or loses hardware functionality after a
+ reboot. Covered by test cases 6, 8, 10 and 14.
* If the backported firmware set is incomplete or the subpackage split
- doesn't match the origin release, firmware blobs required by the HWE kernel
- could be missing, causing hardware regressions on the HWE kernel
- (ironically the exact problem this design is meant to eliminate). Shipping
- the full, unmodified set from the origin release mitigates but does not
- fully eliminate this risk, e.g. if packaging bugs drop files. Covered by
- test case 7.
+ doesn't match the origin release, firmware blobs required by the HWE kernel
+ could be missing, causing hardware regressions on the HWE kernel
+ (ironically the exact problem this design is meant to eliminate). Shipping
+ the full, unmodified set from the origin release mitigates but does not
+ fully eliminate this risk, e.g. if packaging bugs drop files. Covered by
+ test case 7.
* Users who install the HWE firmware set but boot the GA kernel run old
- kernel + new firmware. This combination is not tested by firmware vendors
- and could cause regressions for GA kernel users, up to and including
- hardware that used to work no longer doing so.
+ kernel + new firmware. This combination is not tested by firmware vendors
+ and could cause regressions for GA kernel users, up to and including
+ hardware that used to work no longer doing so.
* A firmware version skew between the linux-firmware-hwe-<foo> packages and
- the HWE kernel's expectations (e.g. if the origin release firmware gets
- updated without a corresponding re-backport) could lead to missing or
- incompatible firmware at HWE kernel runtime.
+ the HWE kernel's expectations (e.g. if the origin release firmware gets
+ updated without a corresponding re-backport) could lead to missing or
+ incompatible firmware at HWE kernel runtime.
* Firmware may be required early (initramfs, root device, storage and
- graphics), so a botched transition between package states could leave a
- system unable to find firmware for its boot-critical devices after the
- next reboot. The lifecycle and release-upgrade transitions are covered
- explicitly by test cases 6, 10, 13 and 14.
+ graphics), so a botched transition between package states could leave a
+ system unable to find firmware for its boot-critical devices after the
+ next reboot. The lifecycle and release-upgrade transitions are covered
+ explicitly by test cases 6, 10, 13 and 14.
* HWE kernel packages declare a hard Depends on linux-firmware-hwe, so kernel
- and firmware packages are upgraded together. However, firmware can be
- required at early boot (initramfs, root device, storage and graphics), so
- there is still a window during an upgrade where a new kernel is present
- but the matching firmware is not (or vice versa). Systems rebooting inside
- that window could see regressions until the upgrade completes.
+ and firmware packages are upgraded together. However, firmware can be
+ required at early boot (initramfs, root device, storage and graphics), so
+ there is still a window during an upgrade where a new kernel is present
+ but the matching firmware is not (or vice versa). Systems rebooting inside
+ that window could see regressions until the upgrade completes.
** Description changed:
[ Impact ]
Backporting new firmware for HWE kernels to LTSes is a constant source of pain
and always results in problems. To limit the risk we always try to backport
only select firmwares that we think are safe but we never get it 100% correct.
The main issues are typically:
1. Some backported firmware that the HWE kernel needs is also loaded by the GA
kernel which can cause issues for GA kernel users. New firmware with old
kernels is a combination that firmware vendors typically don't test and is
not really supported.
2. Missing HWE firmwares. Finding the exact firmwares that a particular kernel
wants is not trivial. We're constantly missing blobs so when LTS users
upgrade to a newer HWE kernel they sometimes experience regressions and HW
that used to work no longer does because firmware is missing. We also
deliberately exclude firmwares that we *think* might cause issues for GA
kernel users.
3. Identifying and backporting firmwares is an expensive task and the results
are never satisfying for both the firmware package maintainers and (more
importantly) the users.
- Note that this is an umbrella bug report that covers all subpackages as well.
The full list is:
+ Note that this is an umbrella bug report that covers all subpackages as
+ well. The full list is:
+
linux-firmware-hwe
linux-firmware-hwe-amd-graphics
linux-firmware-hwe-amd-misc
linux-firmware-hwe-broadcom-wireless
linux-firmware-hwe-intel-graphics
linux-firmware-hwe-intel-misc
linux-firmware-hwe-intel-wireless
linux-firmware-hwe-marvell-prestera
linux-firmware-hwe-marvell-wireless
linux-firmware-hwe-mediatek
linux-firmware-hwe-mellanox-spectrum
linux-firmware-hwe-misc
linux-firmware-hwe-netronome
linux-firmware-hwe-nvidia-graphics
linux-firmware-hwe-qlogic
linux-firmware-hwe-qualcomm-graphics
linux-firmware-hwe-qualcomm-misc
linux-firmware-hwe-qualcomm-wireless
linux-firmware-hwe-realtek
[ Fix ]
Introduce a new set of firmware packages: linux-firmware-hwe-<foo>. The idea
is that HWE kernels depend on linux-firmware-hwe (instead of linux-firmware)
which installs the same firmware binaries as the Ubuntu release where the HWE
kernel comes from. Take Resolute as an example. The first HWE kernel for
Resolute will be 7.3 from Stonking. The new set of linux-firmware-hwe for
Resolute will be a backport of linux-firmware from Stonking. The full set, no
backporting of individual blobs. That way we have the same combination of
kernel+firmware in both the interim and the LTS releases.
<foo> mirrors the linux-firmware subpackage split, e.g.
linux-firmware-hwe-amd-graphics vs linux-firmware-amd-graphics, with
linux-firmware-hwe as the umbrella metapackage pulling in the subpackage set.
There is only one HWE firmware set per release, not one per HWE kernel series.
Lifecycle per series:
* Interim releases: linux-firmware-hwe and every linux-firmware-hwe-<foo> are
metapackages depending on their linux-firmware-<foo> counterparts. There is
nothing to backport for interim releases (their GA kernel is the newest
kernel) so the package names only exist to keep the namespace and dependency
chain uniform across all series.
* LTS at release time: an LTS is the newest release when it comes out, so
there is no HWE yet. The HWE kernel is just a metapackage depending on the
GA kernel and linux-firmware-hwe follows suit and is also just a metapackage
depending on linux-firmware.
* LTS once the first real HWE kernel lands: the HWE kernel metapackage is
replaced by a real kernel package and linux-firmware-hwe-<foo> is replaced
by real packages carrying the full firmware set from the HWE kernel's
origin release. Real packages that provide firmware blobs only ever exist
in LTS releases (Resolute and newer).
That way anything depending on linux-firmware-hwe-<foo> (e.g. HWE kernel
packages) resolves correctly on every series without per-series special
casing, and identical kernel+firmware combinations are guaranteed for both
the interim and the LTS releases.
The caveat with this is that it's not possible to have both linux-firmware
and linux-firmware-hwe installed at the same time (we might work around that
in the future). So you either have the firmwares for the GA kernel *or* the
firmwares for the HWE kernel. But you can have both GA *and* HWE kernels
installed at the same time. In that case and if you choose to boot the GA
kernel you'll run old kernel plus new firmware which could be problematic.
Because the real linux-firmware-hwe-<foo> packages replace metapackages that
previously pulled in the linux-firmware-<foo> counterparts, they will
Conflict/Replace (and possibly Provide) those to handle the upgrade cleanly.
[ Test Case ]
All verifications are package-level: package states via dpkg/apt, file lists
via dpkg -L, file contents via dpkg -V or debsums, and dependency resolution
via apt-get -s dry runs.
Note: no released LTS carries these packages yet, so the cases below need to
be run against test PPAs containing fake packages that mimic the real ones.
1. Dependency resolution
kernel/firmware: GA/GA -> GA/HWE-shim
Install the HWE kernel metapackage. Confirm that it Depends on
linux-firmware-hwe and that linux-firmware-hwe resolves to
linux-firmware. Complete the install and verify the expected package
set is present.
Steps:
a. Start from a system with the GA kernel and linux-firmware
installed and no HWE packages present.
b. Dry-run: apt-get -s install <hwe-kernel-meta>
Expected: resolves <hwe-kernel-meta> -> linux-firmware-hwe ->
linux-firmware, with no linux-firmware removal.
c. Install: apt-get install <hwe-kernel-meta>
d. Verify: dpkg -l shows the HWE kernel metapackage,
linux-firmware-hwe and linux-firmware-hwe-<foo> subpackages all
installed, and linux-firmware still installed (not replaced).
e. Verify: dpkg -L linux-firmware-hwe shows no files (pure
metapackage) and the linux-firmware-hwe-<foo> packages have no
file payload of their own.
2. File inventory parity
kernel/firmware: GA/HWE-shim (unchanged)
Verify that the installed linux-firmware-hwe set is content-identical
to a stock linux-firmware install.
Steps:
a. Compare the dpkg -L file lists of every linux-firmware-hwe-<foo>
package against its linux-firmware-<foo> counterpart. The lists
must be identical.
b. Verify file checksums of the installed files (dpkg -V or
debsums). All files must pass.
3. Removal path
kernel/firmware: GA/HWE-shim -> GA/GA
Purge the HWE kernel metapackage and linux-firmware-hwe and verify
that the GA firmware set is not taken down with it.
Steps:
a. apt-get purge <hwe-kernel-meta> linux-firmware-hwe
b. Dry-run: apt-get -s autoremove
Expected: linux-firmware and its subpackages are NOT candidates
for removal.
c. Verify: all firmware files from dpkg -L linux-firmware are still
present on disk.
4. Freshly released LTS
kernel/firmware: GA/GA -> GA/HWE-shim
Repeat case 1 on a freshly released LTS, where linux-firmware-hwe is
also just a metapackage.
Steps:
a. On a fresh LTS install: apt-get -s install <hwe-kernel-meta>
shows the same resolution as in case 1.
b. Install and verify the package set as in case 1.
5. Negative dependency test
kernel/firmware: GA/GA (unchanged)
Verify that GA kernel packages still depend on linux-firmware, not on
linux-firmware-hwe.
Steps:
a. Inspect the GA kernel metapackage dependencies (dpkg -I or
apt-cache show). The linux-firmware dependency must point at
linux-firmware, not linux-firmware-hwe.
b. Dry-run: apt-get -s install <ga-kernel-meta>
Expected: the resolution does not include linux-firmware-hwe.
6. Opt-in install
kernel/firmware: GA/GA -> HWE/HWE-real
On an LTS with a real HWE kernel available, switching to the HWE
kernel replaces linux-firmware with the real linux-firmware-hwe set.
Steps:
a. Start from a system with the GA kernel and linux-firmware
installed. The HWE kernel and real linux-firmware-hwe packages
are available from the test PPA.
b. Dry-run: apt-get -s install <hwe-kernel-meta>
Expected: linux-firmware is replaced by the real
linux-firmware-hwe set, with no file conflicts reported.
c. Install and verify the final state: the real linux-firmware-hwe
set is installed, linux-firmware is gone, and dpkg --audit
reports no half-configured packages.
7. Origin parity
kernel/firmware: HWE/HWE-real (unchanged)
Verify that the real linux-firmware-hwe set matches the firmware set
of the HWE kernel's origin release exactly.
Steps:
a. In a chroot or container of the origin release, install its
linux-firmware-<foo> packages and capture their file lists
(dpkg -L) and checksums (/var/lib/dpkg/info/*.md5sums).
b. Compare against the target's linux-firmware-hwe-<foo> packages:
identical file lists and checksums, no missing and no extra
blobs.
8. Co-install conflict
kernel/firmware: HWE/HWE-real (unchanged, refused)
Verify that linux-firmware and linux-firmware-hwe cannot be
co-installed.
Steps:
a. With the real HWE set installed, dry-run: apt-get -s install
linux-firmware. Expected: the resolver refuses due to the
conflict.
b. apt-get install linux-firmware linux-firmware-hwe: must fail
cleanly, leaving dpkg --audit clean.
9. Kernel coexistence
kernel/firmware: HWE/HWE-real (unchanged)
Verify that the GA kernel package survives the firmware switch and
record how its linux-firmware dependency is handled.
Steps:
a. After case 6: verify that the GA kernel package is still
installed and intact (dpkg -l).
b. Inspect how the GA kernel's dependency on linux-firmware is
satisfied once linux-firmware is gone (e.g. via Provides from
the HWE packages). Record the outcome; it must not leave any
dpkg --audit findings.
10. Metapackage -> real, via -updates
kernel/firmware: GA/HWE-shim -> HWE/HWE-real
An LTS system with the metapackage-era linux-firmware-hwe installed
receives the real HWE kernel and real firmware packages via
-updates.
Steps:
a. Start from an LTS system in the GA/HWE-shim state.
b. Enable the test PPA with the real HWE kernel and real firmware
packages. Dry-run: apt-get -s dist-upgrade
Expected: the linux-firmware-<foo> counterparts are removed,
the real HWE set and real HWE kernel are installed, no
conflicts reported.
c. Apply the upgrade and verify: no dpkg errors, and no stale
firmware files under /usr/lib/firmware belonging to the removed
counterparts (compare dpkg -L output against the on-disk file
inventory).
11. Real -> metapackage (reversal)
kernel/firmware: HWE/HWE-real -> GA/GA
Verify a clean revert from the real HWE arrangement back to the GA
firmware set.
Steps:
a. Remove the real HWE kernel and the linux-firmware-hwe set.
b. Install linux-firmware.
c. Verify: complete file inventory restored (dpkg -L vs on-disk),
checksums pass (dpkg -V), dpkg --audit clean.
Release upgrades -- one case per source -> target state transition of
linux-firmware-hwe. With no HWE packages installed, every path is a plain
linux-firmware release upgrade and serves as the shared baseline.
12. Shim -> shim
kernel/firmware: (rel A) GA/HWE-shim -> (rel B) GA/HWE-shim
E.g. interim -> interim. Metapackage arrangements on both sides, each
resolving to its own release's linux-firmware.
Steps:
a. Start from a system in the GA/HWE-shim state on release A.
b. do-release-upgrade to release B.
Expected: the linux-firmware-hwe metapackages are present on
both sides, the target release's linux-firmware is installed,
no conflicts reported.
c. Verify the final package set and file inventory on release B.
13. Shim -> real
kernel/firmware: (interim) GA/HWE-shim -> (LTS) HWE/HWE-real
Interim -> LTS after the target's HWE kernel has landed. The
metapackage arrangement is replaced by the real packages and the
real HWE kernel.
Steps:
a. Start from an interim system in the GA/HWE-shim state.
b. do-release-upgrade to the LTS.
Expected: the real linux-firmware-hwe set and real HWE kernel
are installed, no conflicts reported.
c. Verify the final package state as in case 6.
14. Real -> shim
kernel/firmware: (LTS) HWE/HWE-real -> (interim/LTS) GA/HWE-shim
LTS -> interim, and LTS -> LTS before the target's HWE kernel has
landed. Run on both paths. The source carries real packages that
Conflict with linux-firmware while the target is metapackage-era and
wants linux-firmware.
Steps:
a. Start from an LTS system in the HWE/HWE-real state.
b. do-release-upgrade to the target release.
Expected: the real HWE set is removed, the target's
linux-firmware is installed, and the upgrade completes without
conflicts.
c. Verify that no firmware files are lost (file inventory vs the
target release's linux-firmware).
15. Real -> real
kernel/firmware: (LTS A) HWE/HWE-real -> (LTS B) HWE/HWE-real
LTS -> LTS after the target's HWE kernel has landed. The source's
real set is replaced by the target's real set. This is a late-cycle
case that only becomes testable once the target LTS has its own HWE
kernel.
Steps:
a. Start from an LTS system in the HWE/HWE-real state on release A.
b. do-release-upgrade to release B (which already has real HWE
packages).
Expected: the source's real set is replaced by release B's real
set, no conflicts reported.
c. Verify the final package state as in cases 6 and 7.
[ Where Problems Could Occur ]
* Incorrect or missing Conflicts/Replaces/Provides relationships between the
real linux-firmware-hwe-<foo> packages and their linux-firmware-<foo>
counterparts could cause file conflicts, failed upgrades, or a
half-installed package state. Worst case, a system ends up with no firmware
installed and fails to boot or loses hardware functionality after a
reboot. Covered by test cases 6, 8, 10 and 14.
* If the backported firmware set is incomplete or the subpackage split
doesn't match the origin release, firmware blobs required by the HWE kernel
could be missing, causing hardware regressions on the HWE kernel
(ironically the exact problem this design is meant to eliminate). Shipping
the full, unmodified set from the origin release mitigates but does not
fully eliminate this risk, e.g. if packaging bugs drop files. Covered by
test case 7.
* Users who install the HWE firmware set but boot the GA kernel run old
kernel + new firmware. This combination is not tested by firmware vendors
and could cause regressions for GA kernel users, up to and including
hardware that used to work no longer doing so.
* A firmware version skew between the linux-firmware-hwe-<foo> packages and
the HWE kernel's expectations (e.g. if the origin release firmware gets
updated without a corresponding re-backport) could lead to missing or
incompatible firmware at HWE kernel runtime.
* Firmware may be required early (initramfs, root device, storage and
graphics), so a botched transition between package states could leave a
system unable to find firmware for its boot-critical devices after the
next reboot. The lifecycle and release-upgrade transitions are covered
explicitly by test cases 6, 10, 13 and 14.
* HWE kernel packages declare a hard Depends on linux-firmware-hwe, so kernel
and firmware packages are upgraded together. However, firmware can be
required at early boot (initramfs, root device, storage and graphics), so
there is still a window during an upgrade where a new kernel is present
but the matching firmware is not (or vice versa). Systems rebooting inside
that window could see regressions until the upgrade completes.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2170922
Title:
[needs-packaging] linux-firmware-hwe
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-firmware-hwe/+bug/2170922/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs