** No longer affects: ~juergh/ubuntu/linux-firmware-hwe

** No longer affects: ~juergh/ubuntu/linux-firmware-hwe/tenacious

** Also affects: linux-firmware-hwe (Ubuntu)
   Importance: Undecided
       Status: New

** No longer affects: linux-firmware (Ubuntu)

** No longer affects: linux-firmware (Ubuntu Tenacious)

** Also affects: linux-firmware-hwe (Ubuntu Stonking)
   Importance: Undecided
       Status: New

** Also affects: linux-firmware-hwe (Ubuntu Tenacious)
   Importance: Undecided
       Status: New

** Also affects: linux-firmware-hwe (Ubuntu Resolute)
   Importance: Undecided
       Status: New

** Description changed:

  [ Impact ]
  
  Backporting new firmware for HWE kernels to LTSes is a constant source of pain
  and always results in problems. To limit the risk we always try to backport
  only select firmwares that we think are safe but we never get it 100% correct.
  The main issues are typically:
  
  1. Some backported firmware that the HWE kernel needs is also loaded by the GA
-    kernel which can cause issues for GA kernel users. New firmware with old
-    kernels is a combination that firmware vendors typically don't test and is
-    not really supported.
+    kernel which can cause issues for GA kernel users. New firmware with old
+    kernels is a combination that firmware vendors typically don't test and is
+    not really supported.
  
  2. Missing HWE firmwares. Finding the exact firmwares that a particular kernel
-    wants is not trivial. We're constantly missing blobs so when LTS users
-    upgrade to a newer HWE kernel they sometimes experience regressions and HW
-    that used to work no longer does because firmware is missing. We also
-    deliberately exclude firmwares that we *think* might cause issues for GA
-    kernel users.
+    wants is not trivial. We're constantly missing blobs so when LTS users
+    upgrade to a newer HWE kernel they sometimes experience regressions and HW
+    that used to work no longer does because firmware is missing. We also
+    deliberately exclude firmwares that we *think* might cause issues for GA
+    kernel users.
  
  3. Identifying and backporting firmwares is an expensive task and the results
-    are never satisfying for both the firmware package maintainers and (more
-    importantly) the users.
+    are never satisfying for both the firmware package maintainers and (more
+    importantly) the users.
+ 
+ Note that this is an umbrella bug report that covers all subpackages as well. 
The full list is:
+ linux-firmware-hwe
+ linux-firmware-hwe-amd-graphics
+ linux-firmware-hwe-amd-misc
+ linux-firmware-hwe-broadcom-wireless
+ linux-firmware-hwe-intel-graphics
+ linux-firmware-hwe-intel-misc
+ linux-firmware-hwe-intel-wireless
+ linux-firmware-hwe-marvell-prestera
+ linux-firmware-hwe-marvell-wireless
+ linux-firmware-hwe-mediatek
+ linux-firmware-hwe-mellanox-spectrum
+ linux-firmware-hwe-misc
+ linux-firmware-hwe-netronome
+ linux-firmware-hwe-nvidia-graphics
+ linux-firmware-hwe-qlogic
+ linux-firmware-hwe-qualcomm-graphics
+ linux-firmware-hwe-qualcomm-misc
+ linux-firmware-hwe-qualcomm-wireless
+ linux-firmware-hwe-realtek
  
  [ Fix ]
  
  Introduce a new set of firmware packages: linux-firmware-hwe-<foo>. The idea
  is that HWE kernels depend on linux-firmware-hwe (instead of linux-firmware)
  which installs the same firmware binaries as the Ubuntu release where the HWE
  kernel comes from. Take Resolute as an example. The first HWE kernel for
  Resolute will be 7.3 from Stonking. The new set of linux-firmware-hwe for
  Resolute will be a backport of linux-firmware from Stonking. The full set, no
  backporting of individual blobs. That way we have the same combination of
  kernel+firmware in both the interim and the LTS releases.
  
  <foo> mirrors the linux-firmware subpackage split, e.g.
  linux-firmware-hwe-amd-graphics vs linux-firmware-amd-graphics, with
  linux-firmware-hwe as the umbrella metapackage pulling in the subpackage set.
  There is only one HWE firmware set per release, not one per HWE kernel series.
  
  Lifecycle per series:
  
  * Interim releases: linux-firmware-hwe and every linux-firmware-hwe-<foo> are
-   metapackages depending on their linux-firmware-<foo> counterparts. There is
-   nothing to backport for interim releases (their GA kernel is the newest
-   kernel) so the package names only exist to keep the namespace and dependency
-   chain uniform across all series.
+   metapackages depending on their linux-firmware-<foo> counterparts. There is
+   nothing to backport for interim releases (their GA kernel is the newest
+   kernel) so the package names only exist to keep the namespace and dependency
+   chain uniform across all series.
  
  * LTS at release time: an LTS is the newest release when it comes out, so
-   there is no HWE yet. The HWE kernel is just a metapackage depending on the
-   GA kernel and linux-firmware-hwe follows suit and is also just a metapackage
-   depending on linux-firmware.
+   there is no HWE yet. The HWE kernel is just a metapackage depending on the
+   GA kernel and linux-firmware-hwe follows suit and is also just a metapackage
+   depending on linux-firmware.
  
  * LTS once the first real HWE kernel lands: the HWE kernel metapackage is
-   replaced by a real kernel package and linux-firmware-hwe-<foo> is replaced
-   by real packages carrying the full firmware set from the HWE kernel's
-   origin release. Real packages that provide firmware blobs only ever exist
-   in LTS releases (Resolute and newer).
+   replaced by a real kernel package and linux-firmware-hwe-<foo> is replaced
+   by real packages carrying the full firmware set from the HWE kernel's
+   origin release. Real packages that provide firmware blobs only ever exist
+   in LTS releases (Resolute and newer).
  
  That way anything depending on linux-firmware-hwe-<foo> (e.g. HWE kernel
  packages) resolves correctly on every series without per-series special
  casing, and identical kernel+firmware combinations are guaranteed for both
  the interim and the LTS releases.
  
  The caveat with this is that it's not possible to have both linux-firmware
  and linux-firmware-hwe installed at the same time (we might work around that
  in the future). So you either have the firmwares for the GA kernel *or* the
  firmwares for the HWE kernel. But you can have both GA *and* HWE kernels
  installed at the same time. In that case and if you choose to boot the GA
  kernel you'll run old kernel plus new firmware which could be problematic.
  Because the real linux-firmware-hwe-<foo> packages replace metapackages that
  previously pulled in the linux-firmware-<foo> counterparts, they will
  Conflict/Replace (and possibly Provide) those to handle the upgrade cleanly.
  
  [ Test Case ]
  
  All verifications are package-level: package states via dpkg/apt, file lists
  via dpkg -L, file contents via dpkg -V or debsums, and dependency resolution
  via apt-get -s dry runs.
  
  Note: no released LTS carries these packages yet, so the cases below need to
  be run against test PPAs containing fake packages that mimic the real ones.
  
  1. Dependency resolution
-    kernel/firmware: GA/GA -> GA/HWE-shim
- 
-    Install the HWE kernel metapackage. Confirm that it Depends on
-    linux-firmware-hwe and that linux-firmware-hwe resolves to
-    linux-firmware. Complete the install and verify the expected package
-    set is present.
- 
-    Steps:
-      a. Start from a system with the GA kernel and linux-firmware
-         installed and no HWE packages present.
-      b. Dry-run: apt-get -s install <hwe-kernel-meta>
-         Expected: resolves <hwe-kernel-meta> -> linux-firmware-hwe ->
-         linux-firmware, with no linux-firmware removal.
-      c. Install: apt-get install <hwe-kernel-meta>
-      d. Verify: dpkg -l shows the HWE kernel metapackage,
-         linux-firmware-hwe and linux-firmware-hwe-<foo> subpackages all
-         installed, and linux-firmware still installed (not replaced).
-      e. Verify: dpkg -L linux-firmware-hwe shows no files (pure
-         metapackage) and the linux-firmware-hwe-<foo> packages have no
-         file payload of their own.
+    kernel/firmware: GA/GA -> GA/HWE-shim
+ 
+    Install the HWE kernel metapackage. Confirm that it Depends on
+    linux-firmware-hwe and that linux-firmware-hwe resolves to
+    linux-firmware. Complete the install and verify the expected package
+    set is present.
+ 
+    Steps:
+      a. Start from a system with the GA kernel and linux-firmware
+         installed and no HWE packages present.
+      b. Dry-run: apt-get -s install <hwe-kernel-meta>
+         Expected: resolves <hwe-kernel-meta> -> linux-firmware-hwe ->
+         linux-firmware, with no linux-firmware removal.
+      c. Install: apt-get install <hwe-kernel-meta>
+      d. Verify: dpkg -l shows the HWE kernel metapackage,
+         linux-firmware-hwe and linux-firmware-hwe-<foo> subpackages all
+         installed, and linux-firmware still installed (not replaced).
+      e. Verify: dpkg -L linux-firmware-hwe shows no files (pure
+         metapackage) and the linux-firmware-hwe-<foo> packages have no
+         file payload of their own.
  
  2. File inventory parity
-    kernel/firmware: GA/HWE-shim (unchanged)
- 
-    Verify that the installed linux-firmware-hwe set is content-identical
-    to a stock linux-firmware install.
- 
-    Steps:
-      a. Compare the dpkg -L file lists of every linux-firmware-hwe-<foo>
-         package against its linux-firmware-<foo> counterpart. The lists
-         must be identical.
-      b. Verify file checksums of the installed files (dpkg -V or
-         debsums). All files must pass.
+    kernel/firmware: GA/HWE-shim (unchanged)
+ 
+    Verify that the installed linux-firmware-hwe set is content-identical
+    to a stock linux-firmware install.
+ 
+    Steps:
+      a. Compare the dpkg -L file lists of every linux-firmware-hwe-<foo>
+         package against its linux-firmware-<foo> counterpart. The lists
+         must be identical.
+      b. Verify file checksums of the installed files (dpkg -V or
+         debsums). All files must pass.
  
  3. Removal path
-    kernel/firmware: GA/HWE-shim -> GA/GA
- 
-    Purge the HWE kernel metapackage and linux-firmware-hwe and verify
-    that the GA firmware set is not taken down with it.
- 
-    Steps:
-      a. apt-get purge <hwe-kernel-meta> linux-firmware-hwe
-      b. Dry-run: apt-get -s autoremove
-         Expected: linux-firmware and its subpackages are NOT candidates
-         for removal.
-      c. Verify: all firmware files from dpkg -L linux-firmware are still
-         present on disk.
+    kernel/firmware: GA/HWE-shim -> GA/GA
+ 
+    Purge the HWE kernel metapackage and linux-firmware-hwe and verify
+    that the GA firmware set is not taken down with it.
+ 
+    Steps:
+      a. apt-get purge <hwe-kernel-meta> linux-firmware-hwe
+      b. Dry-run: apt-get -s autoremove
+         Expected: linux-firmware and its subpackages are NOT candidates
+         for removal.
+      c. Verify: all firmware files from dpkg -L linux-firmware are still
+         present on disk.
  
  4. Freshly released LTS
-    kernel/firmware: GA/GA -> GA/HWE-shim
- 
-    Repeat case 1 on a freshly released LTS, where linux-firmware-hwe is
-    also just a metapackage.
- 
-    Steps:
-      a. On a fresh LTS install: apt-get -s install <hwe-kernel-meta>
-         shows the same resolution as in case 1.
-      b. Install and verify the package set as in case 1.
+    kernel/firmware: GA/GA -> GA/HWE-shim
+ 
+    Repeat case 1 on a freshly released LTS, where linux-firmware-hwe is
+    also just a metapackage.
+ 
+    Steps:
+      a. On a fresh LTS install: apt-get -s install <hwe-kernel-meta>
+         shows the same resolution as in case 1.
+      b. Install and verify the package set as in case 1.
  
  5. Negative dependency test
-    kernel/firmware: GA/GA (unchanged)
- 
-    Verify that GA kernel packages still depend on linux-firmware, not on
-    linux-firmware-hwe.
- 
-    Steps:
-      a. Inspect the GA kernel metapackage dependencies (dpkg -I or
-         apt-cache show). The linux-firmware dependency must point at
-         linux-firmware, not linux-firmware-hwe.
-      b. Dry-run: apt-get -s install <ga-kernel-meta>
-         Expected: the resolution does not include linux-firmware-hwe.
+    kernel/firmware: GA/GA (unchanged)
+ 
+    Verify that GA kernel packages still depend on linux-firmware, not on
+    linux-firmware-hwe.
+ 
+    Steps:
+      a. Inspect the GA kernel metapackage dependencies (dpkg -I or
+         apt-cache show). The linux-firmware dependency must point at
+         linux-firmware, not linux-firmware-hwe.
+      b. Dry-run: apt-get -s install <ga-kernel-meta>
+         Expected: the resolution does not include linux-firmware-hwe.
  
  6. Opt-in install
-    kernel/firmware: GA/GA -> HWE/HWE-real
- 
-    On an LTS with a real HWE kernel available, switching to the HWE
-    kernel replaces linux-firmware with the real linux-firmware-hwe set.
- 
-    Steps:
-      a. Start from a system with the GA kernel and linux-firmware
-         installed. The HWE kernel and real linux-firmware-hwe packages
-         are available from the test PPA.
-      b. Dry-run: apt-get -s install <hwe-kernel-meta>
-         Expected: linux-firmware is replaced by the real
-         linux-firmware-hwe set, with no file conflicts reported.
-      c. Install and verify the final state: the real linux-firmware-hwe
-         set is installed, linux-firmware is gone, and dpkg --audit
-         reports no half-configured packages.
+    kernel/firmware: GA/GA -> HWE/HWE-real
+ 
+    On an LTS with a real HWE kernel available, switching to the HWE
+    kernel replaces linux-firmware with the real linux-firmware-hwe set.
+ 
+    Steps:
+      a. Start from a system with the GA kernel and linux-firmware
+         installed. The HWE kernel and real linux-firmware-hwe packages
+         are available from the test PPA.
+      b. Dry-run: apt-get -s install <hwe-kernel-meta>
+         Expected: linux-firmware is replaced by the real
+         linux-firmware-hwe set, with no file conflicts reported.
+      c. Install and verify the final state: the real linux-firmware-hwe
+         set is installed, linux-firmware is gone, and dpkg --audit
+         reports no half-configured packages.
  
  7. Origin parity
-    kernel/firmware: HWE/HWE-real (unchanged)
- 
-    Verify that the real linux-firmware-hwe set matches the firmware set
-    of the HWE kernel's origin release exactly.
- 
-    Steps:
-      a. In a chroot or container of the origin release, install its
-         linux-firmware-<foo> packages and capture their file lists
-         (dpkg -L) and checksums (/var/lib/dpkg/info/*.md5sums).
-      b. Compare against the target's linux-firmware-hwe-<foo> packages:
-         identical file lists and checksums, no missing and no extra
-         blobs.
+    kernel/firmware: HWE/HWE-real (unchanged)
+ 
+    Verify that the real linux-firmware-hwe set matches the firmware set
+    of the HWE kernel's origin release exactly.
+ 
+    Steps:
+      a. In a chroot or container of the origin release, install its
+         linux-firmware-<foo> packages and capture their file lists
+         (dpkg -L) and checksums (/var/lib/dpkg/info/*.md5sums).
+      b. Compare against the target's linux-firmware-hwe-<foo> packages:
+         identical file lists and checksums, no missing and no extra
+         blobs.
  
  8. Co-install conflict
-    kernel/firmware: HWE/HWE-real (unchanged, refused)
- 
-    Verify that linux-firmware and linux-firmware-hwe cannot be
-    co-installed.
- 
-    Steps:
-      a. With the real HWE set installed, dry-run: apt-get -s install
-         linux-firmware. Expected: the resolver refuses due to the
-         conflict.
-      b. apt-get install linux-firmware linux-firmware-hwe: must fail
-         cleanly, leaving dpkg --audit clean.
+    kernel/firmware: HWE/HWE-real (unchanged, refused)
+ 
+    Verify that linux-firmware and linux-firmware-hwe cannot be
+    co-installed.
+ 
+    Steps:
+      a. With the real HWE set installed, dry-run: apt-get -s install
+         linux-firmware. Expected: the resolver refuses due to the
+         conflict.
+      b. apt-get install linux-firmware linux-firmware-hwe: must fail
+         cleanly, leaving dpkg --audit clean.
  
  9. Kernel coexistence
-    kernel/firmware: HWE/HWE-real (unchanged)
- 
-    Verify that the GA kernel package survives the firmware switch and
-    record how its linux-firmware dependency is handled.
- 
-    Steps:
-      a. After case 6: verify that the GA kernel package is still
-         installed and intact (dpkg -l).
-      b. Inspect how the GA kernel's dependency on linux-firmware is
-         satisfied once linux-firmware is gone (e.g. via Provides from
-         the HWE packages). Record the outcome; it must not leave any
-         dpkg --audit findings.
+    kernel/firmware: HWE/HWE-real (unchanged)
+ 
+    Verify that the GA kernel package survives the firmware switch and
+    record how its linux-firmware dependency is handled.
+ 
+    Steps:
+      a. After case 6: verify that the GA kernel package is still
+         installed and intact (dpkg -l).
+      b. Inspect how the GA kernel's dependency on linux-firmware is
+         satisfied once linux-firmware is gone (e.g. via Provides from
+         the HWE packages). Record the outcome; it must not leave any
+         dpkg --audit findings.
  
  10. Metapackage -> real, via -updates
-     kernel/firmware: GA/HWE-shim -> HWE/HWE-real
- 
-     An LTS system with the metapackage-era linux-firmware-hwe installed
-     receives the real HWE kernel and real firmware packages via
-     -updates.
- 
-     Steps:
-       a. Start from an LTS system in the GA/HWE-shim state.
-       b. Enable the test PPA with the real HWE kernel and real firmware
-          packages. Dry-run: apt-get -s dist-upgrade
-          Expected: the linux-firmware-<foo> counterparts are removed,
-          the real HWE set and real HWE kernel are installed, no
-          conflicts reported.
-       c. Apply the upgrade and verify: no dpkg errors, and no stale
-          firmware files under /usr/lib/firmware belonging to the removed
-          counterparts (compare dpkg -L output against the on-disk file
-          inventory).
+     kernel/firmware: GA/HWE-shim -> HWE/HWE-real
+ 
+     An LTS system with the metapackage-era linux-firmware-hwe installed
+     receives the real HWE kernel and real firmware packages via
+     -updates.
+ 
+     Steps:
+       a. Start from an LTS system in the GA/HWE-shim state.
+       b. Enable the test PPA with the real HWE kernel and real firmware
+          packages. Dry-run: apt-get -s dist-upgrade
+          Expected: the linux-firmware-<foo> counterparts are removed,
+          the real HWE set and real HWE kernel are installed, no
+          conflicts reported.
+       c. Apply the upgrade and verify: no dpkg errors, and no stale
+          firmware files under /usr/lib/firmware belonging to the removed
+          counterparts (compare dpkg -L output against the on-disk file
+          inventory).
  
  11. Real -> metapackage (reversal)
-     kernel/firmware: HWE/HWE-real -> GA/GA
- 
-     Verify a clean revert from the real HWE arrangement back to the GA
-     firmware set.
- 
-     Steps:
-       a. Remove the real HWE kernel and the linux-firmware-hwe set.
-       b. Install linux-firmware.
-       c. Verify: complete file inventory restored (dpkg -L vs on-disk),
-          checksums pass (dpkg -V), dpkg --audit clean.
+     kernel/firmware: HWE/HWE-real -> GA/GA
+ 
+     Verify a clean revert from the real HWE arrangement back to the GA
+     firmware set.
+ 
+     Steps:
+       a. Remove the real HWE kernel and the linux-firmware-hwe set.
+       b. Install linux-firmware.
+       c. Verify: complete file inventory restored (dpkg -L vs on-disk),
+          checksums pass (dpkg -V), dpkg --audit clean.
  
  Release upgrades -- one case per source -> target state transition of
  linux-firmware-hwe. With no HWE packages installed, every path is a plain
  linux-firmware release upgrade and serves as the shared baseline.
  
  12. Shim -> shim
-     kernel/firmware: (rel A) GA/HWE-shim -> (rel B) GA/HWE-shim
- 
-     E.g. interim -> interim. Metapackage arrangements on both sides, each
-     resolving to its own release's linux-firmware.
- 
-     Steps:
-       a. Start from a system in the GA/HWE-shim state on release A.
-       b. do-release-upgrade to release B.
-          Expected: the linux-firmware-hwe metapackages are present on
-          both sides, the target release's linux-firmware is installed,
-          no conflicts reported.
-       c. Verify the final package set and file inventory on release B.
+     kernel/firmware: (rel A) GA/HWE-shim -> (rel B) GA/HWE-shim
+ 
+     E.g. interim -> interim. Metapackage arrangements on both sides, each
+     resolving to its own release's linux-firmware.
+ 
+     Steps:
+       a. Start from a system in the GA/HWE-shim state on release A.
+       b. do-release-upgrade to release B.
+          Expected: the linux-firmware-hwe metapackages are present on
+          both sides, the target release's linux-firmware is installed,
+          no conflicts reported.
+       c. Verify the final package set and file inventory on release B.
  
  13. Shim -> real
-     kernel/firmware: (interim) GA/HWE-shim -> (LTS) HWE/HWE-real
- 
-     Interim -> LTS after the target's HWE kernel has landed. The
-     metapackage arrangement is replaced by the real packages and the
-     real HWE kernel.
- 
-     Steps:
-       a. Start from an interim system in the GA/HWE-shim state.
-       b. do-release-upgrade to the LTS.
-          Expected: the real linux-firmware-hwe set and real HWE kernel
-          are installed, no conflicts reported.
-       c. Verify the final package state as in case 6.
+     kernel/firmware: (interim) GA/HWE-shim -> (LTS) HWE/HWE-real
+ 
+     Interim -> LTS after the target's HWE kernel has landed. The
+     metapackage arrangement is replaced by the real packages and the
+     real HWE kernel.
+ 
+     Steps:
+       a. Start from an interim system in the GA/HWE-shim state.
+       b. do-release-upgrade to the LTS.
+          Expected: the real linux-firmware-hwe set and real HWE kernel
+          are installed, no conflicts reported.
+       c. Verify the final package state as in case 6.
  
  14. Real -> shim
-     kernel/firmware: (LTS) HWE/HWE-real -> (interim/LTS) GA/HWE-shim
- 
-     LTS -> interim, and LTS -> LTS before the target's HWE kernel has
-     landed. Run on both paths. The source carries real packages that
-     Conflict with linux-firmware while the target is metapackage-era and
-     wants linux-firmware.
- 
-     Steps:
-       a. Start from an LTS system in the HWE/HWE-real state.
-       b. do-release-upgrade to the target release.
-          Expected: the real HWE set is removed, the target's
-          linux-firmware is installed, and the upgrade completes without
-          conflicts.
-       c. Verify that no firmware files are lost (file inventory vs the
-          target release's linux-firmware).
+     kernel/firmware: (LTS) HWE/HWE-real -> (interim/LTS) GA/HWE-shim
+ 
+     LTS -> interim, and LTS -> LTS before the target's HWE kernel has
+     landed. Run on both paths. The source carries real packages that
+     Conflict with linux-firmware while the target is metapackage-era and
+     wants linux-firmware.
+ 
+     Steps:
+       a. Start from an LTS system in the HWE/HWE-real state.
+       b. do-release-upgrade to the target release.
+          Expected: the real HWE set is removed, the target's
+          linux-firmware is installed, and the upgrade completes without
+          conflicts.
+       c. Verify that no firmware files are lost (file inventory vs the
+          target release's linux-firmware).
  
  15. Real -> real
-     kernel/firmware: (LTS A) HWE/HWE-real -> (LTS B) HWE/HWE-real
- 
-     LTS -> LTS after the target's HWE kernel has landed. The source's
-     real set is replaced by the target's real set. This is a late-cycle
-     case that only becomes testable once the target LTS has its own HWE
-     kernel.
- 
-     Steps:
-       a. Start from an LTS system in the HWE/HWE-real state on release A.
-       b. do-release-upgrade to release B (which already has real HWE
-          packages).
-          Expected: the source's real set is replaced by release B's real
-          set, no conflicts reported.
-       c. Verify the final package state as in cases 6 and 7.
+     kernel/firmware: (LTS A) HWE/HWE-real -> (LTS B) HWE/HWE-real
+ 
+     LTS -> LTS after the target's HWE kernel has landed. The source's
+     real set is replaced by the target's real set. This is a late-cycle
+     case that only becomes testable once the target LTS has its own HWE
+     kernel.
+ 
+     Steps:
+       a. Start from an LTS system in the HWE/HWE-real state on release A.
+       b. do-release-upgrade to release B (which already has real HWE
+          packages).
+          Expected: the source's real set is replaced by release B's real
+          set, no conflicts reported.
+       c. Verify the final package state as in cases 6 and 7.
  
  [ Where Problems Could Occur ]
  
  * Incorrect or missing Conflicts/Replaces/Provides relationships between the
-   real linux-firmware-hwe-<foo> packages and their linux-firmware-<foo>
-   counterparts could cause file conflicts, failed upgrades, or a
-   half-installed package state. Worst case, a system ends up with no firmware
-   installed and fails to boot or loses hardware functionality after a
-   reboot. Covered by test cases 6, 8, 10 and 14.
+   real linux-firmware-hwe-<foo> packages and their linux-firmware-<foo>
+   counterparts could cause file conflicts, failed upgrades, or a
+   half-installed package state. Worst case, a system ends up with no firmware
+   installed and fails to boot or loses hardware functionality after a
+   reboot. Covered by test cases 6, 8, 10 and 14.
  
  * If the backported firmware set is incomplete or the subpackage split
-   doesn't match the origin release, firmware blobs required by the HWE kernel
-   could be missing, causing hardware regressions on the HWE kernel
-   (ironically the exact problem this design is meant to eliminate). Shipping
-   the full, unmodified set from the origin release mitigates but does not
-   fully eliminate this risk, e.g. if packaging bugs drop files. Covered by
-   test case 7.
+   doesn't match the origin release, firmware blobs required by the HWE kernel
+   could be missing, causing hardware regressions on the HWE kernel
+   (ironically the exact problem this design is meant to eliminate). Shipping
+   the full, unmodified set from the origin release mitigates but does not
+   fully eliminate this risk, e.g. if packaging bugs drop files. Covered by
+   test case 7.
  
  * Users who install the HWE firmware set but boot the GA kernel run old
-   kernel + new firmware. This combination is not tested by firmware vendors
-   and could cause regressions for GA kernel users, up to and including
-   hardware that used to work no longer doing so.
+   kernel + new firmware. This combination is not tested by firmware vendors
+   and could cause regressions for GA kernel users, up to and including
+   hardware that used to work no longer doing so.
  
  * A firmware version skew between the linux-firmware-hwe-<foo> packages and
-   the HWE kernel's expectations (e.g. if the origin release firmware gets
-   updated without a corresponding re-backport) could lead to missing or
-   incompatible firmware at HWE kernel runtime.
+   the HWE kernel's expectations (e.g. if the origin release firmware gets
+   updated without a corresponding re-backport) could lead to missing or
+   incompatible firmware at HWE kernel runtime.
  
  * Firmware may be required early (initramfs, root device, storage and
-   graphics), so a botched transition between package states could leave a
-   system unable to find firmware for its boot-critical devices after the
-   next reboot. The lifecycle and release-upgrade transitions are covered
-   explicitly by test cases 6, 10, 13 and 14.
+   graphics), so a botched transition between package states could leave a
+   system unable to find firmware for its boot-critical devices after the
+   next reboot. The lifecycle and release-upgrade transitions are covered
+   explicitly by test cases 6, 10, 13 and 14.
  
  * HWE kernel packages declare a hard Depends on linux-firmware-hwe, so kernel
-   and firmware packages are upgraded together. However, firmware can be
-   required at early boot (initramfs, root device, storage and graphics), so
-   there is still a window during an upgrade where a new kernel is present
-   but the matching firmware is not (or vice versa). Systems rebooting inside
-   that window could see regressions until the upgrade completes.
+   and firmware packages are upgraded together. However, firmware can be
+   required at early boot (initramfs, root device, storage and graphics), so
+   there is still a window during an upgrade where a new kernel is present
+   but the matching firmware is not (or vice versa). Systems rebooting inside
+   that window could see regressions until the upgrade completes.

** Description changed:

  [ Impact ]
  
  Backporting new firmware for HWE kernels to LTSes is a constant source of pain
  and always results in problems. To limit the risk we always try to backport
  only select firmwares that we think are safe but we never get it 100% correct.
  The main issues are typically:
  
  1. Some backported firmware that the HWE kernel needs is also loaded by the GA
     kernel which can cause issues for GA kernel users. New firmware with old
     kernels is a combination that firmware vendors typically don't test and is
     not really supported.
  
  2. Missing HWE firmwares. Finding the exact firmwares that a particular kernel
     wants is not trivial. We're constantly missing blobs so when LTS users
     upgrade to a newer HWE kernel they sometimes experience regressions and HW
     that used to work no longer does because firmware is missing. We also
     deliberately exclude firmwares that we *think* might cause issues for GA
     kernel users.
  
  3. Identifying and backporting firmwares is an expensive task and the results
     are never satisfying for both the firmware package maintainers and (more
     importantly) the users.
  
- Note that this is an umbrella bug report that covers all subpackages as well. 
The full list is:
+ Note that this is an umbrella bug report that covers all subpackages as
+ well. The full list is:
+ 
  linux-firmware-hwe
  linux-firmware-hwe-amd-graphics
  linux-firmware-hwe-amd-misc
  linux-firmware-hwe-broadcom-wireless
  linux-firmware-hwe-intel-graphics
  linux-firmware-hwe-intel-misc
  linux-firmware-hwe-intel-wireless
  linux-firmware-hwe-marvell-prestera
  linux-firmware-hwe-marvell-wireless
  linux-firmware-hwe-mediatek
  linux-firmware-hwe-mellanox-spectrum
  linux-firmware-hwe-misc
  linux-firmware-hwe-netronome
  linux-firmware-hwe-nvidia-graphics
  linux-firmware-hwe-qlogic
  linux-firmware-hwe-qualcomm-graphics
  linux-firmware-hwe-qualcomm-misc
  linux-firmware-hwe-qualcomm-wireless
  linux-firmware-hwe-realtek
  
  [ Fix ]
  
  Introduce a new set of firmware packages: linux-firmware-hwe-<foo>. The idea
  is that HWE kernels depend on linux-firmware-hwe (instead of linux-firmware)
  which installs the same firmware binaries as the Ubuntu release where the HWE
  kernel comes from. Take Resolute as an example. The first HWE kernel for
  Resolute will be 7.3 from Stonking. The new set of linux-firmware-hwe for
  Resolute will be a backport of linux-firmware from Stonking. The full set, no
  backporting of individual blobs. That way we have the same combination of
  kernel+firmware in both the interim and the LTS releases.
  
  <foo> mirrors the linux-firmware subpackage split, e.g.
  linux-firmware-hwe-amd-graphics vs linux-firmware-amd-graphics, with
  linux-firmware-hwe as the umbrella metapackage pulling in the subpackage set.
  There is only one HWE firmware set per release, not one per HWE kernel series.
  
  Lifecycle per series:
  
  * Interim releases: linux-firmware-hwe and every linux-firmware-hwe-<foo> are
    metapackages depending on their linux-firmware-<foo> counterparts. There is
    nothing to backport for interim releases (their GA kernel is the newest
    kernel) so the package names only exist to keep the namespace and dependency
    chain uniform across all series.
  
  * LTS at release time: an LTS is the newest release when it comes out, so
    there is no HWE yet. The HWE kernel is just a metapackage depending on the
    GA kernel and linux-firmware-hwe follows suit and is also just a metapackage
    depending on linux-firmware.
  
  * LTS once the first real HWE kernel lands: the HWE kernel metapackage is
    replaced by a real kernel package and linux-firmware-hwe-<foo> is replaced
    by real packages carrying the full firmware set from the HWE kernel's
    origin release. Real packages that provide firmware blobs only ever exist
    in LTS releases (Resolute and newer).
  
  That way anything depending on linux-firmware-hwe-<foo> (e.g. HWE kernel
  packages) resolves correctly on every series without per-series special
  casing, and identical kernel+firmware combinations are guaranteed for both
  the interim and the LTS releases.
  
  The caveat with this is that it's not possible to have both linux-firmware
  and linux-firmware-hwe installed at the same time (we might work around that
  in the future). So you either have the firmwares for the GA kernel *or* the
  firmwares for the HWE kernel. But you can have both GA *and* HWE kernels
  installed at the same time. In that case and if you choose to boot the GA
  kernel you'll run old kernel plus new firmware which could be problematic.
  Because the real linux-firmware-hwe-<foo> packages replace metapackages that
  previously pulled in the linux-firmware-<foo> counterparts, they will
  Conflict/Replace (and possibly Provide) those to handle the upgrade cleanly.
  
  [ Test Case ]
  
  All verifications are package-level: package states via dpkg/apt, file lists
  via dpkg -L, file contents via dpkg -V or debsums, and dependency resolution
  via apt-get -s dry runs.
  
  Note: no released LTS carries these packages yet, so the cases below need to
  be run against test PPAs containing fake packages that mimic the real ones.
  
  1. Dependency resolution
     kernel/firmware: GA/GA -> GA/HWE-shim
  
     Install the HWE kernel metapackage. Confirm that it Depends on
     linux-firmware-hwe and that linux-firmware-hwe resolves to
     linux-firmware. Complete the install and verify the expected package
     set is present.
  
     Steps:
       a. Start from a system with the GA kernel and linux-firmware
          installed and no HWE packages present.
       b. Dry-run: apt-get -s install <hwe-kernel-meta>
          Expected: resolves <hwe-kernel-meta> -> linux-firmware-hwe ->
          linux-firmware, with no linux-firmware removal.
       c. Install: apt-get install <hwe-kernel-meta>
       d. Verify: dpkg -l shows the HWE kernel metapackage,
          linux-firmware-hwe and linux-firmware-hwe-<foo> subpackages all
          installed, and linux-firmware still installed (not replaced).
       e. Verify: dpkg -L linux-firmware-hwe shows no files (pure
          metapackage) and the linux-firmware-hwe-<foo> packages have no
          file payload of their own.
  
  2. File inventory parity
     kernel/firmware: GA/HWE-shim (unchanged)
  
     Verify that the installed linux-firmware-hwe set is content-identical
     to a stock linux-firmware install.
  
     Steps:
       a. Compare the dpkg -L file lists of every linux-firmware-hwe-<foo>
          package against its linux-firmware-<foo> counterpart. The lists
          must be identical.
       b. Verify file checksums of the installed files (dpkg -V or
          debsums). All files must pass.
  
  3. Removal path
     kernel/firmware: GA/HWE-shim -> GA/GA
  
     Purge the HWE kernel metapackage and linux-firmware-hwe and verify
     that the GA firmware set is not taken down with it.
  
     Steps:
       a. apt-get purge <hwe-kernel-meta> linux-firmware-hwe
       b. Dry-run: apt-get -s autoremove
          Expected: linux-firmware and its subpackages are NOT candidates
          for removal.
       c. Verify: all firmware files from dpkg -L linux-firmware are still
          present on disk.
  
  4. Freshly released LTS
     kernel/firmware: GA/GA -> GA/HWE-shim
  
     Repeat case 1 on a freshly released LTS, where linux-firmware-hwe is
     also just a metapackage.
  
     Steps:
       a. On a fresh LTS install: apt-get -s install <hwe-kernel-meta>
          shows the same resolution as in case 1.
       b. Install and verify the package set as in case 1.
  
  5. Negative dependency test
     kernel/firmware: GA/GA (unchanged)
  
     Verify that GA kernel packages still depend on linux-firmware, not on
     linux-firmware-hwe.
  
     Steps:
       a. Inspect the GA kernel metapackage dependencies (dpkg -I or
          apt-cache show). The linux-firmware dependency must point at
          linux-firmware, not linux-firmware-hwe.
       b. Dry-run: apt-get -s install <ga-kernel-meta>
          Expected: the resolution does not include linux-firmware-hwe.
  
  6. Opt-in install
     kernel/firmware: GA/GA -> HWE/HWE-real
  
     On an LTS with a real HWE kernel available, switching to the HWE
     kernel replaces linux-firmware with the real linux-firmware-hwe set.
  
     Steps:
       a. Start from a system with the GA kernel and linux-firmware
          installed. The HWE kernel and real linux-firmware-hwe packages
          are available from the test PPA.
       b. Dry-run: apt-get -s install <hwe-kernel-meta>
          Expected: linux-firmware is replaced by the real
          linux-firmware-hwe set, with no file conflicts reported.
       c. Install and verify the final state: the real linux-firmware-hwe
          set is installed, linux-firmware is gone, and dpkg --audit
          reports no half-configured packages.
  
  7. Origin parity
     kernel/firmware: HWE/HWE-real (unchanged)
  
     Verify that the real linux-firmware-hwe set matches the firmware set
     of the HWE kernel's origin release exactly.
  
     Steps:
       a. In a chroot or container of the origin release, install its
          linux-firmware-<foo> packages and capture their file lists
          (dpkg -L) and checksums (/var/lib/dpkg/info/*.md5sums).
       b. Compare against the target's linux-firmware-hwe-<foo> packages:
          identical file lists and checksums, no missing and no extra
          blobs.
  
  8. Co-install conflict
     kernel/firmware: HWE/HWE-real (unchanged, refused)
  
     Verify that linux-firmware and linux-firmware-hwe cannot be
     co-installed.
  
     Steps:
       a. With the real HWE set installed, dry-run: apt-get -s install
          linux-firmware. Expected: the resolver refuses due to the
          conflict.
       b. apt-get install linux-firmware linux-firmware-hwe: must fail
          cleanly, leaving dpkg --audit clean.
  
  9. Kernel coexistence
     kernel/firmware: HWE/HWE-real (unchanged)
  
     Verify that the GA kernel package survives the firmware switch and
     record how its linux-firmware dependency is handled.
  
     Steps:
       a. After case 6: verify that the GA kernel package is still
          installed and intact (dpkg -l).
       b. Inspect how the GA kernel's dependency on linux-firmware is
          satisfied once linux-firmware is gone (e.g. via Provides from
          the HWE packages). Record the outcome; it must not leave any
          dpkg --audit findings.
  
  10. Metapackage -> real, via -updates
      kernel/firmware: GA/HWE-shim -> HWE/HWE-real
  
      An LTS system with the metapackage-era linux-firmware-hwe installed
      receives the real HWE kernel and real firmware packages via
      -updates.
  
      Steps:
        a. Start from an LTS system in the GA/HWE-shim state.
        b. Enable the test PPA with the real HWE kernel and real firmware
           packages. Dry-run: apt-get -s dist-upgrade
           Expected: the linux-firmware-<foo> counterparts are removed,
           the real HWE set and real HWE kernel are installed, no
           conflicts reported.
        c. Apply the upgrade and verify: no dpkg errors, and no stale
           firmware files under /usr/lib/firmware belonging to the removed
           counterparts (compare dpkg -L output against the on-disk file
           inventory).
  
  11. Real -> metapackage (reversal)
      kernel/firmware: HWE/HWE-real -> GA/GA
  
      Verify a clean revert from the real HWE arrangement back to the GA
      firmware set.
  
      Steps:
        a. Remove the real HWE kernel and the linux-firmware-hwe set.
        b. Install linux-firmware.
        c. Verify: complete file inventory restored (dpkg -L vs on-disk),
           checksums pass (dpkg -V), dpkg --audit clean.
  
  Release upgrades -- one case per source -> target state transition of
  linux-firmware-hwe. With no HWE packages installed, every path is a plain
  linux-firmware release upgrade and serves as the shared baseline.
  
  12. Shim -> shim
      kernel/firmware: (rel A) GA/HWE-shim -> (rel B) GA/HWE-shim
  
      E.g. interim -> interim. Metapackage arrangements on both sides, each
      resolving to its own release's linux-firmware.
  
      Steps:
        a. Start from a system in the GA/HWE-shim state on release A.
        b. do-release-upgrade to release B.
           Expected: the linux-firmware-hwe metapackages are present on
           both sides, the target release's linux-firmware is installed,
           no conflicts reported.
        c. Verify the final package set and file inventory on release B.
  
  13. Shim -> real
      kernel/firmware: (interim) GA/HWE-shim -> (LTS) HWE/HWE-real
  
      Interim -> LTS after the target's HWE kernel has landed. The
      metapackage arrangement is replaced by the real packages and the
      real HWE kernel.
  
      Steps:
        a. Start from an interim system in the GA/HWE-shim state.
        b. do-release-upgrade to the LTS.
           Expected: the real linux-firmware-hwe set and real HWE kernel
           are installed, no conflicts reported.
        c. Verify the final package state as in case 6.
  
  14. Real -> shim
      kernel/firmware: (LTS) HWE/HWE-real -> (interim/LTS) GA/HWE-shim
  
      LTS -> interim, and LTS -> LTS before the target's HWE kernel has
      landed. Run on both paths. The source carries real packages that
      Conflict with linux-firmware while the target is metapackage-era and
      wants linux-firmware.
  
      Steps:
        a. Start from an LTS system in the HWE/HWE-real state.
        b. do-release-upgrade to the target release.
           Expected: the real HWE set is removed, the target's
           linux-firmware is installed, and the upgrade completes without
           conflicts.
        c. Verify that no firmware files are lost (file inventory vs the
           target release's linux-firmware).
  
  15. Real -> real
      kernel/firmware: (LTS A) HWE/HWE-real -> (LTS B) HWE/HWE-real
  
      LTS -> LTS after the target's HWE kernel has landed. The source's
      real set is replaced by the target's real set. This is a late-cycle
      case that only becomes testable once the target LTS has its own HWE
      kernel.
  
      Steps:
        a. Start from an LTS system in the HWE/HWE-real state on release A.
        b. do-release-upgrade to release B (which already has real HWE
           packages).
           Expected: the source's real set is replaced by release B's real
           set, no conflicts reported.
        c. Verify the final package state as in cases 6 and 7.
  
  [ Where Problems Could Occur ]
  
  * Incorrect or missing Conflicts/Replaces/Provides relationships between the
    real linux-firmware-hwe-<foo> packages and their linux-firmware-<foo>
    counterparts could cause file conflicts, failed upgrades, or a
    half-installed package state. Worst case, a system ends up with no firmware
    installed and fails to boot or loses hardware functionality after a
    reboot. Covered by test cases 6, 8, 10 and 14.
  
  * If the backported firmware set is incomplete or the subpackage split
    doesn't match the origin release, firmware blobs required by the HWE kernel
    could be missing, causing hardware regressions on the HWE kernel
    (ironically the exact problem this design is meant to eliminate). Shipping
    the full, unmodified set from the origin release mitigates but does not
    fully eliminate this risk, e.g. if packaging bugs drop files. Covered by
    test case 7.
  
  * Users who install the HWE firmware set but boot the GA kernel run old
    kernel + new firmware. This combination is not tested by firmware vendors
    and could cause regressions for GA kernel users, up to and including
    hardware that used to work no longer doing so.
  
  * A firmware version skew between the linux-firmware-hwe-<foo> packages and
    the HWE kernel's expectations (e.g. if the origin release firmware gets
    updated without a corresponding re-backport) could lead to missing or
    incompatible firmware at HWE kernel runtime.
  
  * Firmware may be required early (initramfs, root device, storage and
    graphics), so a botched transition between package states could leave a
    system unable to find firmware for its boot-critical devices after the
    next reboot. The lifecycle and release-upgrade transitions are covered
    explicitly by test cases 6, 10, 13 and 14.
  
  * HWE kernel packages declare a hard Depends on linux-firmware-hwe, so kernel
    and firmware packages are upgraded together. However, firmware can be
    required at early boot (initramfs, root device, storage and graphics), so
    there is still a window during an upgrade where a new kernel is present
    but the matching firmware is not (or vice versa). Systems rebooting inside
    that window could see regressions until the upgrade completes.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2170922

Title:
  [needs-packaging] linux-firmware-hwe

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-firmware-hwe/+bug/2170922/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to