Launchpad has imported 11 comments from the remote bug at https://bugzilla.kernel.org/show_bug.cgi?id=217947.
If you reply to an imported comment from within Launchpad, your comment will be sent to the remote bug automatically. Read more about Launchpad's inter-bugtracker facilities at https://help.launchpad.net/InterBugTracking. ------------------------------------------------------------------------ On 2023-09-25T09:12:13+00:00 jirislaby wrote: Downstream report: https://bugzilla.suse.com/show_bug.cgi?id=1215602 LKML link: https://lore.kernel.org/all/047d3c51-0a9e-4c3e-beef-625a7aa4f...@kernel.org/ according to logs, since 6.3 (up to 6.5.4 now), I repeatedly see: > WARNING: CPU: 14 PID: 962 at drivers/acpi/platform_profile.c:74 > platform_profile_show+0xb1/0x100 [platform_profile] > Modules linked in: ccm michael_mic ... > CPU: 14 PID: 962 Comm: power-profiles- Kdump: loaded Not tainted > 6.5.4-6-default #1 openSUSE Tumbleweed (unreleased) > dd37106c593be78644bb80e3c1534d801bf4cb36 > Hardware name: LENOVO 21CRS0K83K/21CRS0K83K, BIOS R22ET60W (1.30 ) 02/09/2023 > RIP: 0010:platform_profile_show+0xb1/0x100 [platform_profile] > Code: d0 a8 ... > RSP: 0018:ffff9c1ac0b97db0 EFLAGS: 00010296 > RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000008fc35be0 > RDX: 0000000000000000 RSI: ffff9c1ac0b97db4 RDI: ffffffffc0a8b0a0 > RBP: ffff8955ca540000 R08: ffff895b9f1ed180 R09: ffff895559ea1bc0 > R10: 00000000031a400e R11: 000000000003f680 R12: ffff895b9f1ed180 > R13: ffff9c1ac0b97e50 R14: 0000000000000001 R15: ffff9c1ac0b97ee8 > FS: 00007f71b0e71900(0000) GS:ffff895b9f100000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 00007fe402ea3400 CR3: 000000012004c000 CR4: 0000000000750ee0 > PKRU: 55555554 > Call Trace: > <TASK> > sysfs_kf_seq_show+0xab/0x100 > seq_read_iter+0x123/0x480 > vfs_read+0x1b8/0x300 It's: WARN_ON((profile < 0) || (profile >= ARRAY_SIZE(profile_names)))) So I put there one more print: dev_warn(dev, "profile=%d profile_get=%ps\n", profile, cur_profile->profile_get); and I see: : profile=-1883022368 profile_get=dytc_profile_get [thinkpad_acpi] : profile=-1510173440 profile_get=dytc_profile_get [thinkpad_acpi] : profile=-1510173440 profile_get=dytc_profile_get [thinkpad_acpi] : profile=-966231712 profile_get=dytc_profile_get [thinkpad_acpi] : profile=-1578420592 profile_get=dytc_profile_get [thinkpad_acpi] : profile=-1578420592 profile_get=dytc_profile_get [thinkpad_acpi] : profile=-1578420592 profile_get=dytc_profile_get [thinkpad_acpi] I wonder about dev passed to dytc_profile_get() having empty name (nothing before colon above)? Is that expected? Ah, convert_dytc_to_profile()'s retval is not checked in dytc_profile_refresh(). Adding: --- a/drivers/platform/x86/thinkpad_acpi.c +++ b/drivers/platform/x86/thinkpad_acpi.c @@ -10418,7 +10418,14 @@ static void dytc_profile_refresh(void) return; perfmode = (output >> DYTC_GET_MODE_BIT) & 0xF; - convert_dytc_to_profile(funcmode, perfmode, &profile); + err = convert_dytc_to_profile(funcmode, perfmode, &profile); + if (err) { + pr_warn("%s: mmc=%u psc=%u mmc_get=%u funcmode=%d output=0x%x perfmode=%d\n", + __func__, !!(dytc_capabilities & BIT(DYTC_FC_MMC)), + !!(dytc_capabilities & BIT(DYTC_FC_PSC)), + dytc_mmc_get_available, funcmode, output, perfmode); + return; + } if (profile != dytc_current_profile) { dytc_current_profile = profile; platform_profile_notify(); fixes the warning, of course. The output is: dytc_profile_refresh: mmc=0 psc=1 mmc_get=0 funcmode=0 output=0x1f001 perfmode=15 So using psc mode, it retrieves DYTC (0x1f001) as follows: 0x1.... -> VSTD=1 -> STD mode 0x.f... -> CICM=0xf, DYTC_GET_MODE_BIT(12) -> dunno what it is in STD 0x..0.. -> CICF=0x0, DYTC_GET_FUNCTION_BIT(8) -> DYTC_FUNCTION_STD 0x....1 -> GOOD But convert_dytc_to_profile() doesn't handle this at all. Do I have a newer DYTC interface? Or a broken one? Case (0x00) { Local1 = 0x0100 Local1 |= 0x80000000 Local1 |= 0x00 Local1 |= 0x01 } I.e. version 8.0, it seems. My DYTC for CMD_GET looks like: Case (0x02) { Local5 = VSTD /* \VSTD */ Local5 |= (VCQL << 0x01) Local5 |= (VSTP << 0x04) Local5 |= (VADM << 0x07) Local5 |= (VTMS << 0x09) Local5 |= (VDLS << 0x0A) Local5 |= (VMSC << 0x0C) Local5 |= (VPSC << 0x0D) Local1 = (CICF << 0x08) If ((CICF == 0x03)) { CICM = SMYH /* \SMYH */ } ElseIf ((CICF == 0x0B)) { CICM = SMMC /* \SMMC */ } ElseIf ((CICF == 0x0D)) { CICM = SPSC /* \SPSC */ } ElseIf ((CICF == 0x0F)) { CICM = SAMT /* \SAMT */ } ElseIf ((CICF == 0x07)) { CICM = SADM /* \SADM */ } Else { CICM = 0x0F } Local1 |= (CICM << 0x0C) Local1 |= (Local5 << 0x10) Local1 |= 0x01 } Reply at: https://bugs.launchpad.net/ubuntu/+source/linux-signed- hwe-6.2/+bug/2040102/comments/0 ------------------------------------------------------------------------ On 2023-09-25T09:12:31+00:00 jirislaby wrote: Created attachment 305147 DSDT Reply at: https://bugs.launchpad.net/ubuntu/+source/linux-signed- hwe-6.2/+bug/2040102/comments/1 ------------------------------------------------------------------------ On 2023-09-25T12:54:13+00:00 mpearson-lenovo wrote: Hi, Can I confirm this is on a T14s G3 AMD platform? Thanks for the detailed report - very helpful and very much appreciated. I'll need to reproduce this but I don't have one of these platforms myself so will need to ask a colleague for help. It looks like this line is causing problems on this platform: funcmode = (output >> DYTC_GET_FUNCTION_BIT) & 0xF; Normally that should return PSC mode (in this case - unless you have AMT mode, which you don't on that gen platform). But it's saying it is in standard mode instead....which is peculiar. The FW is doing something non-standard to other platforms. Do you still get the warning when: - you're in performance or low-power modes? - you're in lap mode? As a quick test could you try adding: /* Check if we are PSC mode, or have AMT enabled */ funcmode = (output >> DYTC_GET_FUNCTION_BIT) & 0xF; +if (funcmode == DYTC_FUNCTION_STD) + funcmode = DYTC_FUNCTION_PSC; To see if it behaves correctly please? I'll have to check with the FW team and find out why this change in behaviour...I suspect I need to do a kernel patch to better handle this AMT case though. Mark Reply at: https://bugs.launchpad.net/ubuntu/+source/linux-signed- hwe-6.2/+bug/2040102/comments/2 ------------------------------------------------------------------------ On 2023-09-26T06:57:50+00:00 jirislaby wrote: (In reply to Mark Pearson from comment #2) > Can I confirm this is on a T14s G3 AMD platform? Yes. > It looks like this line is causing problems on this platform: > funcmode = (output >> DYTC_GET_FUNCTION_BIT) & 0xF; > > Normally that should return PSC mode (in this case - unless you have AMT > mode, which you don't on that gen platform). > But it's saying it is in standard mode instead....which is peculiar. The FW > is doing something non-standard to other platforms. Interestingly, it happens only initially and during suspend. Other than that, it's in PSC mode likely. > Do you still get the warning when: > - you're in performance or low-power modes? I am on always in low-power mode. This notebook is crap, it's noisy and very hot in balanced or performance modes even when idling. > - you're in lap mode? What is that? > As a quick test could you try adding: > > /* Check if we are PSC mode, or have AMT enabled */ > funcmode = (output >> DYTC_GET_FUNCTION_BIT) & 0xF; > +if (funcmode == DYTC_FUNCTION_STD) > + funcmode = DYTC_FUNCTION_PSC; > > To see if it behaves correctly please? Will try. Reply at: https://bugs.launchpad.net/ubuntu/+source/linux-signed- hwe-6.2/+bug/2040102/comments/3 ------------------------------------------------------------------------ On 2023-09-26T10:32:09+00:00 jirislaby wrote: (In reply to Jiri Slaby from comment #3) > > As a quick test could you try adding: > > > > /* Check if we are PSC mode, or have AMT enabled */ > > funcmode = (output >> DYTC_GET_FUNCTION_BIT) & 0xF; > > +if (funcmode == DYTC_FUNCTION_STD) > > + funcmode = DYTC_FUNCTION_PSC; > > > > To see if it behaves correctly please? > > Will try. It only changes the error to EINVAL. 0xf is not handled as a correct state in PSC. BTW the first (bad) log: thinkpad_acpi: dytc_profile_refresh: err=-22 mmc=0 psc=1 mmc_get=0 funcmode=13 output=0x1f001 perfmode=15 subsequent refreshes (good): thinkpad_acpi: dytc_profile_refresh: err=0 mmc=0 psc=1 mmc_get=0 funcmode=13 output=0x20013d01 perfmode=3 Reply at: https://bugs.launchpad.net/ubuntu/+source/linux-signed- hwe-6.2/+bug/2040102/comments/4 ------------------------------------------------------------------------ On 2023-09-27T00:06:00+00:00 mpearson-lenovo wrote: Ah yeah - need to add: case DYTC_MODE_MMC_BALANCE: underneath case DYTC_MODE_PSC_BALANCE: in function convert_dytc_to_profile (same as is done for the MMC block above) I think that will cure that error. Still tracking down HW here :( Mark Reply at: https://bugs.launchpad.net/ubuntu/+source/linux-signed- hwe-6.2/+bug/2040102/comments/5 ------------------------------------------------------------------------ On 2023-09-27T06:34:04+00:00 jirislaby wrote: (In reply to Mark Pearson from comment #5) > Ah yeah - need to add: > case DYTC_MODE_MMC_BALANCE: > underneath > case DYTC_MODE_PSC_BALANCE: > in function convert_dytc_to_profile > (same as is done for the MMC block above) > > I think that will cure that error. Of course: dytc_profile_refresh: err=0 mmc=0 psc=1 mmc_get=0 funcmode=13 output=0x1f001 perfmode=15 Reply at: https://bugs.launchpad.net/ubuntu/+source/linux-signed- hwe-6.2/+bug/2040102/comments/6 ------------------------------------------------------------------------ On 2023-09-27T17:09:26+00:00 mpearson-lenovo wrote: OK - so (just to be sure) with those changes it is working correctly? Mark Reply at: https://bugs.launchpad.net/ubuntu/+source/linux-signed- hwe-6.2/+bug/2040102/comments/7 ------------------------------------------------------------------------ On 2023-09-28T06:10:40+00:00 jirislaby wrote: (In reply to Mark Pearson from comment #7) > OK - so (just to be sure) with those changes it is working correctly? It appears so. Reply at: https://bugs.launchpad.net/ubuntu/+source/linux-signed- hwe-6.2/+bug/2040102/comments/8 ------------------------------------------------------------------------ On 2023-10-26T08:59:01+00:00 jirislaby wrote: Gentle ping: any updates on this? Reply at: https://bugs.launchpad.net/ubuntu/+source/linux-signed- hwe-6.2/+bug/2040102/comments/12 ------------------------------------------------------------------------ On 2023-10-26T17:21:41+00:00 mpearson-lenovo wrote: Thanks for the ping - I had completely forgotten about this one. I'll check on the status. My notes (internal ticket LO-2242) say we're waiting on the FW team for a fix, they did do one version but it didn't work. I've no idea why it's taking so long. Mark Reply at: https://bugs.launchpad.net/ubuntu/+source/linux-signed- hwe-6.2/+bug/2040102/comments/13 ** Changed in: linux Status: Unknown => Confirmed ** Changed in: linux Importance: Unknown => Medium ** Bug watch added: bugzilla.suse.com/ #1215602 https://bugzilla.suse.com/show_bug.cgi?id=1215602 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-signed-hwe-6.2 in Ubuntu. https://bugs.launchpad.net/bugs/2040102 Title: kernel warning when open the lid Status in Linux: Confirmed Status in linux-signed-hwe-6.2 package in Ubuntu: New Bug description: I always put my thinkpad neo 14 to idle, close the lid, but not sleep/suspend, when I open the lid and login, I can see below warning from kernel in journal. If my memory is correct, when kernel is still around 5.15 or previous hwe(might be 6.0), I just see the first message about psmouse, no kernel warning. Oct 22 19:52:52 kernel: psmouse serio1: TrackPoint at isa0060/serio1/input0 lost synchronization, throwing 1 bytes away. Oct 22 19:52:52 kernel: ------------[ cut here ]------------ Oct 22 19:52:52 kernel: WARNING: CPU: 15 PID: 1674 at drivers/acpi/platform_profile.c:74 platform_profile_show+0xae/0xf0 [platform_profile] Oct 22 19:52:52 kernel: Modules linked in: btrfs blake2b_generic xor raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs cpuid tls tcp_diag inet_diag pppoe pppox rfcomm veth nft_masq dummy bridge stp llc nvme_fabrics nf_conntrack_netbio s_ns nf_conntrack_broadcast ccm vhost_vsock vmw_vsock_virtio_transport_common vhost vhost_iotlb vsock nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_co nntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables libcrc32c nfnetlink cmac algif_hash algif_skcipher af_alg overlay bnep binfmt_misc zstd snd_ctl_led snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_soc_hdac_hdmi snd_sof_probes sn d_hda_codec_hdmi snd_hda_codec_conexant snd_hda_codec_generic btusb uvcvideo btrtl videobuf2_vmalloc btbcm videobuf2_memops btintel videobuf2_v4l2 btmtk videodev bluetooth videobuf2_common ecdh_generic mc ecc snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel Oct 22 19:52:52 kernel: soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus intel_tcc_cooling snd_soc_core x86_pkg_temp_thermal snd_compress intel_powerclamp ac97_bus snd_pcm_dmaengine coretemp snd_hda_intel kvm_intel snd_intel_dspcfg zfs(PO) i915 snd_intel_sdw_acpi kvm snd_hda_codec zunicode(PO) snd_hda_core snd_hwdep zzst d(O) irqbypass thinkpad_acpi iwlmvm drm_buddy snd_pcm crct10dif_pclmul nvram zlua(O) ttm polyval_clmulni polyval_generic snd_seq_midi drm_display_helper zavl(PO) mac80211 ghash_clmulni_intel snd_seq_midi_event cec sha512_ssse3 icp(PO) snd_rawmidi nls_iso8859_1 rc_core aesni_intel snd_seq processor_thermal_device_pci libarc4 drm_kms_helper crypto_simd zcommon(PO) snd_seq_device processor_thermal_device i2c_algo_bit cryptd mei_hdcp iwlwifi mei_pxp snd_timer znvpair(PO) proces sor_thermal_rfim syscopyarea cmdlinepart pmt_telemetry Oct 22 19:52:52 kernel: rapl spi_nor snd intel_rapl_msr pmt_class input_leds processor_thermal_mbox sysfillrect mei_me think_lmi spl(O) cfg80211 processor_thermal_rapl intel_cstate serio_raw firmware_attributes_class wmi_bmof mtd sysimgbl t soundcore mei intel_rapl_common igen6_edac intel_vsec ledtrig_audio mac_hid int3403_thermal int3400_thermal intel_hid soc_button_array platform_profile int340x_thermal_zone acpi_thermal_rel acpi_tad sparse_keymap acpi_pad sch_fq_codel ms r parport_pc lz4 lz4_compress ppdev lp drm zram parport efi_pstore autofs4 nvme ucsi_acpi spi_intel_pci i2c_i801 xhci_pci typec_ucsi nvme_core video crc32_pclmul psmouse thunderbolt spi_intel i2c_smbus xhci_pci_renesas typec nvme_common wm i pinctrl_tigerlake Oct 22 19:52:52 kernel: CPU: 15 PID: 1674 Comm: power-profiles- Tainted: P W O 6.2.0-34-generic #34~22.04.1-Ubuntu Oct 22 19:52:52 kernel: Hardware name: LENOVO 21DN001ACD/21DN001ACD, BIOS R1RET48W(1.20) 03/06/2023 Oct 22 19:52:52 kernel: RIP: 0010:platform_profile_show+0xae/0xf0 [platform_profile] Oct 22 19:52:52 kernel: Code: e7 e8 f6 bb 80 df 48 98 48 8b 55 e8 65 48 2b 14 25 28 00 00 00 75 33 48 83 c4 10 5b 41 5c 5d 31 d2 31 f6 31 ff c3 cc cc cc cc <0f> 0b 48 c7 c0 fb ff ff ff eb d3 48 c7 c7 60 61 54 c0 e8 6b 0e 36 Oct 22 19:52:52 kernel: RSP: 0018:ffffa5a5c635bce0 EFLAGS: 00010282 Oct 22 19:52:52 kernel: RAX: 00000000ed0fbddb RBX: 0000000000000000 RCX: 0000000000000000 Oct 22 19:52:52 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 Oct 22 19:52:52 kernel: RBP: ffffa5a5c635bd00 R08: ffff8cf841f7a880 R09: 0000000000000001 Oct 22 19:52:52 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8cf9ddb43000 Oct 22 19:52:52 kernel: R13: 0000000000000001 R14: ffff8cf8426d40a0 R15: ffff8cf8426d4078 Oct 22 19:52:52 kernel: FS: 00007f11f27d98c0(0000) GS:ffff8cfb8f5c0000(0000) knlGS:0000000000000000 Oct 22 19:52:52 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Oct 22 19:52:52 kernel: CR2: 000000c0005fd010 CR3: 000000010953a000 CR4: 0000000000750ee0 Oct 22 19:52:52 kernel: PKRU: 55555554 Oct 22 19:52:52 kernel: Call Trace: Oct 22 19:52:52 kernel: <TASK> Oct 22 19:52:52 kernel: ? show_regs+0x72/0x90 Oct 22 19:52:52 kernel: ? platform_profile_show+0xae/0xf0 [platform_profile] Oct 22 19:52:52 kernel: ? __warn+0x8d/0x160 Oct 22 19:52:52 kernel: ? platform_profile_show+0xae/0xf0 [platform_profile] Oct 22 19:52:52 kernel: ? report_bug+0x1bb/0x1d0 Oct 22 19:52:52 kernel: ? handle_bug+0x46/0x90 Oct 22 19:52:52 kernel: ? exc_invalid_op+0x19/0x80 Oct 22 19:52:52 kernel: ? asm_exc_invalid_op+0x1b/0x20 Oct 22 19:52:52 kernel: ? platform_profile_show+0xae/0xf0 [platform_profile] Oct 22 19:52:52 kernel: kobj_attr_show+0xf/0x30 Oct 22 19:52:52 kernel: sysfs_kf_seq_show+0xa4/0x150 Oct 22 19:52:52 kernel: kernfs_seq_show+0x24/0x40 Oct 22 19:52:52 kernel: seq_read_iter+0x132/0x4e0 Oct 22 19:52:52 kernel: ? fsnotify_perm.part.0+0x76/0x1a0 Oct 22 19:52:52 kernel: ? fsnotify_perm.part.0+0x76/0x1a0 Oct 22 19:52:52 kernel: kernfs_fop_read_iter+0x34/0x40 Oct 22 19:52:52 kernel: vfs_read+0x219/0x2f0 Oct 22 19:52:52 kernel: ksys_read+0x67/0xf0 Oct 22 19:52:52 kernel: __x64_sys_read+0x19/0x30 Oct 22 19:52:52 kernel: do_syscall_64+0x59/0x90 Oct 22 19:52:52 kernel: ? exit_to_user_mode_prepare+0x3b/0xd0 Oct 22 19:52:52 kernel: ? syscall_exit_to_user_mode+0x38/0x60 Oct 22 19:52:52 kernel: ? do_syscall_64+0x69/0x90 Oct 22 19:52:52 kernel: ? do_syscall_64+0x69/0x90 Oct 22 19:52:52 kernel: ? do_syscall_64+0x69/0x90 Oct 22 19:52:52 kernel: ? do_syscall_64+0x69/0x90 Oct 22 19:52:52 kernel: entry_SYSCALL_64_after_hwframe+0x73/0xdd Oct 22 19:52:52 kernel: RIP: 0033:0x7f11f2f14a0c Oct 22 19:52:52 kernel: Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 f9 bf f7 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 48 89 44 24 08 e8 3f c0 f7 ff 48 Oct 22 19:52:52 kernel: RSP: 002b:00007ffda5aee7d0 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 Oct 22 19:52:52 kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f11f2f14a0c Oct 22 19:52:52 kernel: RDX: 0000000000001000 RSI: 0000564a866eb720 RDI: 0000000000000009 Oct 22 19:52:52 kernel: RBP: 0000564a86685eb0 R08: 0000000000000000 R09: 0000000000001000 Oct 22 19:52:52 kernel: R10: 0000564a8667ca10 R11: 0000000000000246 R12: 00007ffda5aee908 Oct 22 19:52:52 kernel: R13: 0000000000000000 R14: 00007ffda5aee910 R15: 0000000000000009 Oct 22 19:52:52 kernel: </TASK> Oct 22 19:52:52 kernel: ---[ end trace 0000000000000000 ]--- ProblemType: Bug DistroRelease: Ubuntu 22.04 Package: linux-image-6.2.0-34-generic 6.2.0-34.34~22.04.1 ProcVersionSignature: Ubuntu 6.2.0-34.34~22.04.1-generic 6.2.16 Uname: Linux 6.2.0-34-generic x86_64 NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair ApportVersion: 2.20.11-0ubuntu82.5 Architecture: amd64 CasperMD5CheckResult: pass CurrentDesktop: ubuntu:GNOME Date: Sun Oct 22 20:12:23 2023 InstallationDate: Installed on 2023-03-23 (212 days ago) InstallationMedia: Ubuntu 22.04.2 LTS "Jammy Jellyfish" - Release amd64 (20230223) SourcePackage: linux-signed-hwe-6.2 UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/2040102/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
