** Description changed: [ Impact ] - Backporting new firmware for HWE kernels to LTSes is a constant source - of pain and always results in problems. To limit the risk we always try - to backport only select firmwares that we think are safe but we never - get it 100% correct. The main issues are typically: - - 1. Some backported firmware that the HWE kernel needs is also loaded by - the GA kernel which can cause issues for GA kernel users. New firmware - with old kernels is a combination that firmware vendors typically don't - test and is not really supported. - - 2. Missing HWE firmwares. Finding the exact firmwares that a particular - kernel wants it not trivial. We're constantly missing blobs so when LTS - users upgrade to a newer HWE kernel they sometimes experience - regressions and HW that used to work no longer does because firmware is - missing. We also deliberately exclude firmwares that we *think* might - cause issues for GA kernel users. - - 3. Identifying and backporting firmwares is an expensive task and the - results are never satisfying for both the firmware package maintainers - and (more importantly) the users. + Backporting new firmware for HWE kernels to LTSes is a constant source of pain + and always results in problems. To limit the risk we always try to backport + only select firmwares that we think are safe but we never get it 100% correct. + The main issues are typically: + + 1. Some backported firmware that the HWE kernel needs is also loaded by the GA + kernel which can cause issues for GA kernel users. New firmware with old + kernels is a combination that firmware vendors typically don't test and is + not really supported. + + 2. Missing HWE firmwares. Finding the exact firmwares that a particular kernel + wants is not trivial. We're constantly missing blobs so when LTS users + upgrade to a newer HWE kernel they sometimes experience regressions and HW + that used to work no longer does because firmware is missing. We also + deliberately exclude firmwares that we *think* might cause issues for GA + kernel users. + + 3. Identifying and backporting firmwares is an expensive task and the results + are never satisfying for both the firmware package maintainers and (more + importantly) the users. [ Fix ] - Introduce a new set of firmware packages: linux-firmware-hwe-<foo>. The - idea is that HWE kernels depend on linux-firmware-hwe (instead of linux- - firmware) which installs the same firmware binaries as the Ubuntu - release where the HWE kernel comes from. Take Resolute as an example. - The first HWE kernel for Resolute will be 7.3 from Stonking. The new set - of linux-firmware-hwe for Resolute will be a backport of linux-firmware - from Stonking. The full set, no backporting of individual blobs. That - way we have the same combination of kernel+firmware in both the interim - and the LTS releases. - - The caveat with this is that it's not possible to have both linux- - firmware and linux-firmware-hwe installed at the same time (we might - work around that in the future). So you either have the firmwares for - the GA kernel *or* the firmwares for the HWE kernel. But you can have - both GA *and* HWE kernels installed at the same time. In that case and - if you choose to boot the GA kernel you'll run old kernel plus new - firmware which could be problematic. - - Note that at the time of a new Ubuntu release, linux-firmware-hwe-<foo> - will just be metapackages that depend on linux-firmware-<foo>. When the - first HWE kernel is added to an LTS, these metapackages will be replaced - with packages that provide real HWE firmwares. + Introduce a new set of firmware packages: linux-firmware-hwe-<foo>. The idea + is that HWE kernels depend on linux-firmware-hwe (instead of linux-firmware) + which installs the same firmware binaries as the Ubuntu release where the HWE + kernel comes from. Take Resolute as an example. The first HWE kernel for + Resolute will be 7.3 from Stonking. The new set of linux-firmware-hwe for + Resolute will be a backport of linux-firmware from Stonking. The full set, no + backporting of individual blobs. That way we have the same combination of + kernel+firmware in both the interim and the LTS releases. + + <foo> mirrors the linux-firmware subpackage split, e.g. + linux-firmware-hwe-amd-graphics vs linux-firmware-amd-graphics, with + linux-firmware-hwe as the umbrella metapackage pulling in the subpackage set. + There is only one HWE firmware set per release, not one per HWE kernel series. + + Lifecycle per series: + + * Interim releases: linux-firmware-hwe and every linux-firmware-hwe-<foo> are + metapackages depending on their linux-firmware-<foo> counterparts. There is + nothing to backport for interim releases (their GA kernel is the newest + kernel) so the package names only exist to keep the namespace and dependency + chain uniform across all series. + + * LTS at release time: an LTS is the newest release when it comes out, so + there is no HWE yet. The HWE kernel is just a metapackage depending on the + GA kernel and linux-firmware-hwe follows suit and is also just a metapackage + depending on linux-firmware. + + * LTS once the first real HWE kernel lands: the HWE kernel metapackage is + replaced by a real kernel package and linux-firmware-hwe-<foo> is replaced + by real packages carrying the full firmware set from the HWE kernel's + origin release. Real packages that provide firmware blobs only ever exist + in LTS releases (Resolute and newer). + + That way anything depending on linux-firmware-hwe-<foo> (e.g. HWE kernel + packages) resolves correctly on every series without per-series special + casing, and identical kernel+firmware combinations are guaranteed for both + the interim and the LTS releases. + + The caveat with this is that it's not possible to have both linux-firmware + and linux-firmware-hwe installed at the same time (we might work around that + in the future). So you either have the firmwares for the GA kernel *or* the + firmwares for the HWE kernel. But you can have both GA *and* HWE kernels + installed at the same time. In that case and if you choose to boot the GA + kernel you'll run old kernel plus new firmware which could be problematic. + Because the real linux-firmware-hwe-<foo> packages replace metapackages that + previously pulled in the linux-firmware-<foo> counterparts, they will + Conflict/Replace (and possibly Provide) those to handle the upgrade cleanly. [ Test Case ] - TBD + All verifications are package-level: package states via dpkg/apt, file lists + via dpkg -L, file contents via dpkg -V or debsums, and dependency resolution + via apt-get -s dry runs. + + Note: no released LTS carries these packages yet, so the cases below need to + be run against test PPAs containing fake packages that mimic the real ones. + + 1. Dependency resolution + kernel/firmware: GA/GA -> GA/HWE-shim + + Install the HWE kernel metapackage. Confirm that it Depends on + linux-firmware-hwe and that linux-firmware-hwe resolves to + linux-firmware. Complete the install and verify the expected package + set is present. + + Steps: + a. Start from a system with the GA kernel and linux-firmware + installed and no HWE packages present. + b. Dry-run: apt-get -s install <hwe-kernel-meta> + Expected: resolves <hwe-kernel-meta> -> linux-firmware-hwe -> + linux-firmware, with no linux-firmware removal. + c. Install: apt-get install <hwe-kernel-meta> + d. Verify: dpkg -l shows the HWE kernel metapackage, + linux-firmware-hwe and linux-firmware-hwe-<foo> subpackages all + installed, and linux-firmware still installed (not replaced). + e. Verify: dpkg -L linux-firmware-hwe shows no files (pure + metapackage) and the linux-firmware-hwe-<foo> packages have no + file payload of their own. + + 2. File inventory parity + kernel/firmware: GA/HWE-shim (unchanged) + + Verify that the installed linux-firmware-hwe set is content-identical + to a stock linux-firmware install. + + Steps: + a. Compare the dpkg -L file lists of every linux-firmware-hwe-<foo> + package against its linux-firmware-<foo> counterpart. The lists + must be identical. + b. Verify file checksums of the installed files (dpkg -V or + debsums). All files must pass. + + 3. Removal path + kernel/firmware: GA/HWE-shim -> GA/GA + + Purge the HWE kernel metapackage and linux-firmware-hwe and verify + that the GA firmware set is not taken down with it. + + Steps: + a. apt-get purge <hwe-kernel-meta> linux-firmware-hwe + b. Dry-run: apt-get -s autoremove + Expected: linux-firmware and its subpackages are NOT candidates + for removal. + c. Verify: all firmware files from dpkg -L linux-firmware are still + present on disk. + + 4. Freshly released LTS + kernel/firmware: GA/GA -> GA/HWE-shim + + Repeat case 1 on a freshly released LTS, where linux-firmware-hwe is + also just a metapackage. + + Steps: + a. On a fresh LTS install: apt-get -s install <hwe-kernel-meta> + shows the same resolution as in case 1. + b. Install and verify the package set as in case 1. + + 5. Negative dependency test + kernel/firmware: GA/GA (unchanged) + + Verify that GA kernel packages still depend on linux-firmware, not on + linux-firmware-hwe. + + Steps: + a. Inspect the GA kernel metapackage dependencies (dpkg -I or + apt-cache show). The linux-firmware dependency must point at + linux-firmware, not linux-firmware-hwe. + b. Dry-run: apt-get -s install <ga-kernel-meta> + Expected: the resolution does not include linux-firmware-hwe. + + 6. Opt-in install + kernel/firmware: GA/GA -> HWE/HWE-real + + On an LTS with a real HWE kernel available, switching to the HWE + kernel replaces linux-firmware with the real linux-firmware-hwe set. + + Steps: + a. Start from a system with the GA kernel and linux-firmware + installed. The HWE kernel and real linux-firmware-hwe packages + are available from the test PPA. + b. Dry-run: apt-get -s install <hwe-kernel-meta> + Expected: linux-firmware is replaced by the real + linux-firmware-hwe set, with no file conflicts reported. + c. Install and verify the final state: the real linux-firmware-hwe + set is installed, linux-firmware is gone, and dpkg --audit + reports no half-configured packages. + + 7. Origin parity + kernel/firmware: HWE/HWE-real (unchanged) + + Verify that the real linux-firmware-hwe set matches the firmware set + of the HWE kernel's origin release exactly. + + Steps: + a. In a chroot or container of the origin release, install its + linux-firmware-<foo> packages and capture their file lists + (dpkg -L) and checksums (/var/lib/dpkg/info/*.md5sums). + b. Compare against the target's linux-firmware-hwe-<foo> packages: + identical file lists and checksums, no missing and no extra + blobs. + + 8. Co-install conflict + kernel/firmware: HWE/HWE-real (unchanged, refused) + + Verify that linux-firmware and linux-firmware-hwe cannot be + co-installed. + + Steps: + a. With the real HWE set installed, dry-run: apt-get -s install + linux-firmware. Expected: the resolver refuses due to the + conflict. + b. apt-get install linux-firmware linux-firmware-hwe: must fail + cleanly, leaving dpkg --audit clean. + + 9. Kernel coexistence + kernel/firmware: HWE/HWE-real (unchanged) + + Verify that the GA kernel package survives the firmware switch and + record how its linux-firmware dependency is handled. + + Steps: + a. After case 6: verify that the GA kernel package is still + installed and intact (dpkg -l). + b. Inspect how the GA kernel's dependency on linux-firmware is + satisfied once linux-firmware is gone (e.g. via Provides from + the HWE packages). Record the outcome; it must not leave any + dpkg --audit findings. + + 10. Metapackage -> real, via -updates + kernel/firmware: GA/HWE-shim -> HWE/HWE-real + + An LTS system with the metapackage-era linux-firmware-hwe installed + receives the real HWE kernel and real firmware packages via + -updates. + + Steps: + a. Start from an LTS system in the GA/HWE-shim state. + b. Enable the test PPA with the real HWE kernel and real firmware + packages. Dry-run: apt-get -s dist-upgrade + Expected: the linux-firmware-<foo> counterparts are removed, + the real HWE set and real HWE kernel are installed, no + conflicts reported. + c. Apply the upgrade and verify: no dpkg errors, and no stale + firmware files under /usr/lib/firmware belonging to the removed + counterparts (compare dpkg -L output against the on-disk file + inventory). + + 11. Real -> metapackage (reversal) + kernel/firmware: HWE/HWE-real -> GA/GA + + Verify a clean revert from the real HWE arrangement back to the GA + firmware set. + + Steps: + a. Remove the real HWE kernel and the linux-firmware-hwe set. + b. Install linux-firmware. + c. Verify: complete file inventory restored (dpkg -L vs on-disk), + checksums pass (dpkg -V), dpkg --audit clean. + + Release upgrades -- one case per source -> target state transition of + linux-firmware-hwe. With no HWE packages installed, every path is a plain + linux-firmware release upgrade and serves as the shared baseline. + + 12. Shim -> shim + kernel/firmware: (rel A) GA/HWE-shim -> (rel B) GA/HWE-shim + + E.g. interim -> interim. Metapackage arrangements on both sides, each + resolving to its own release's linux-firmware. + + Steps: + a. Start from a system in the GA/HWE-shim state on release A. + b. do-release-upgrade to release B. + Expected: the linux-firmware-hwe metapackages are present on + both sides, the target release's linux-firmware is installed, + no conflicts reported. + c. Verify the final package set and file inventory on release B. + + 13. Shim -> real + kernel/firmware: (interim) GA/HWE-shim -> (LTS) HWE/HWE-real + + Interim -> LTS after the target's HWE kernel has landed. The + metapackage arrangement is replaced by the real packages and the + real HWE kernel. + + Steps: + a. Start from an interim system in the GA/HWE-shim state. + b. do-release-upgrade to the LTS. + Expected: the real linux-firmware-hwe set and real HWE kernel + are installed, no conflicts reported. + c. Verify the final package state as in case 6. + + 14. Real -> shim + kernel/firmware: (LTS) HWE/HWE-real -> (interim/LTS) GA/HWE-shim + + LTS -> interim, and LTS -> LTS before the target's HWE kernel has + landed. Run on both paths. The source carries real packages that + Conflict with linux-firmware while the target is metapackage-era and + wants linux-firmware. + + Steps: + a. Start from an LTS system in the HWE/HWE-real state. + b. do-release-upgrade to the target release. + Expected: the real HWE set is removed, the target's + linux-firmware is installed, and the upgrade completes without + conflicts. + c. Verify that no firmware files are lost (file inventory vs the + target release's linux-firmware). + + 15. Real -> real + kernel/firmware: (LTS A) HWE/HWE-real -> (LTS B) HWE/HWE-real + + LTS -> LTS after the target's HWE kernel has landed. The source's + real set is replaced by the target's real set. This is a late-cycle + case that only becomes testable once the target LTS has its own HWE + kernel. + + Steps: + a. Start from an LTS system in the HWE/HWE-real state on release A. + b. do-release-upgrade to release B (which already has real HWE + packages). + Expected: the source's real set is replaced by release B's real + set, no conflicts reported. + c. Verify the final package state as in cases 6 and 7. [ Where Problems Could Occur ] - TBD + * Incorrect or missing Conflicts/Replaces/Provides relationships between the + real linux-firmware-hwe-<foo> packages and their linux-firmware-<foo> + counterparts could cause file conflicts, failed upgrades, or a + half-installed package state. Worst case, a system ends up with no firmware + installed and fails to boot or loses hardware functionality after a + reboot. Covered by test cases 6, 8, 10 and 14. + + * If the backported firmware set is incomplete or the subpackage split + doesn't match the origin release, firmware blobs required by the HWE kernel + could be missing, causing hardware regressions on the HWE kernel + (ironically the exact problem this design is meant to eliminate). Shipping + the full, unmodified set from the origin release mitigates but does not + fully eliminate this risk, e.g. if packaging bugs drop files. Covered by + test case 7. + + * Users who install the HWE firmware set but boot the GA kernel run old + kernel + new firmware. This combination is not tested by firmware vendors + and could cause regressions for GA kernel users, up to and including + hardware that used to work no longer doing so. + + * A firmware version skew between the linux-firmware-hwe-<foo> packages and + the HWE kernel's expectations (e.g. if the origin release firmware gets + updated without a corresponding re-backport) could lead to missing or + incompatible firmware at HWE kernel runtime. + + * Firmware may be required early (initramfs, root device, storage and + graphics), so a botched transition between package states could leave a + system unable to find firmware for its boot-critical devices after the + next reboot. The lifecycle and release-upgrade transitions are covered + explicitly by test cases 6, 10, 13 and 14. + + * HWE kernel packages declare a hard Depends on linux-firmware-hwe, so kernel + and firmware packages are upgraded together. However, firmware can be + required at early boot (initramfs, root device, storage and graphics), so + there is still a window during an upgrade where a new kernel is present + but the matching firmware is not (or vice versa). Systems rebooting inside + that window could see regressions until the upgrade completes.
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2170922 Title: [needs-packaging] linux-firmware-hwe To manage notifications about this bug go to: https://bugs.launchpad.net/~juergh/+archive/ubuntu/linux-firmware-hwe/+bug/2170922/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
