Re: Boot regression in Linux v6.4-rc3
On Wed, May 31, 2023 at 11:16 AM Frank Scheiner wrote: > Looking forward to the next occasion - for your sake maybe on another > architecture, but can't promise... ;-) I think it would be prudent for Song to also ask you to test his future upcoming modules patches on ia64 given how hard it is to procure such hardware too. Luis
Re: Boot regression in Linux v6.4-rc3
On 31.05.23 21:14, Luis Chamberlain wrote: On Wed, May 31, 2023 at 11:16 AM Frank Scheiner wrote: Looking forward to the next occasion - for your sake maybe on another architecture, but can't promise... ;-) I think it would be prudent for Song to also ask you to test his future upcoming modules patches on ia64 given how hard it is to procure such hardware too. Sure! Cheers, Frank
Re: Boot regression in Linux v6.4-rc3
Hi Linus, hi Song, On 29.05.23 00:46, Song Liu wrote: [...] Thanks for running the test! I will send the official patch. Thanks, Song With the fix merged and to conclude this, I'd like to add that it was a pleasure to work with you on this problem, although I didn't do much. Looking forward to the next occasion - for your sake maybe on another architecture, but can't promise... ;-) Cheers, Frank
Re: Boot regression in Linux v6.4-rc3
On Tue, May 30, 2023 at 05:04:56PM -0400, Linus Torvalds wrote:
> On Tue, May 30, 2023 at 4:21 PM Konstantin Ryabitsev
> wrote:
> >
> > We only add things to lore when someone asks, and nobody's asked. :) I guess
> > I'll consider this an ask and put it on the radar.
>
> Thanks. It would probably be good to see if there are any other
> vger.kernel.org lists with any appreciable traffic that aren't on
> lore.
Yes, that will auto-fix itself as we continue to migrate things over to
subspace ("new vger").
-K
Re: Boot regression in Linux v6.4-rc3
On Tue, May 30, 2023 at 4:21 PM Konstantin Ryabitsev wrote: > > We only add things to lore when someone asks, and nobody's asked. :) I guess > I'll consider this an ask and put it on the radar. Thanks. It would probably be good to see if there are any other vger.kernel.org lists with any appreciable traffic that aren't on lore. Linus
Re: Boot regression in Linux v6.4-rc3
On Sat, May 27, 2023 at 10:08:24AM -0700, Linus Torvalds wrote: > This does make it clear just how great a mailing list archive lore is. > Konstantin, is there any particular reason why > linux-i...@vger.kernel.org isn't in lore? Is it just a rational hatred > of all things itanium? We only add things to lore when someone asks, and nobody's asked. :) I guess I'll consider this an ask and put it on the radar. Regards, -K
Re: Boot regression in Linux v6.4-rc3
On Sun, May 28, 2023 at 1:10 AM Frank Scheiner wrote: > > Hi again, > > On 28.05.23 09:30, Frank Scheiner wrote: > > [...] > > Thanks, that patch (as -patch4 on top of v6.4-rc3) fixes the boot > > regression for me on the rx2620: > > > > [...] > > > > Great! I'll give it a try on my rx2800-i2, too, but assume it wil work > > there, too. > > Indeed, -patch4 also makes it work on the rx2800-i2: Thanks for running the test! I will send the official patch. Thanks, Song > > ``` > ELILO v3.16 for EFI/IA-64 > .. > Uncompressing Linux... done > Loading file AC10027B.initrd.img...done > [0.00] Linux version > 6.4.0-rc3-44c026a73be8038f03dbdeef028b642880cf1511-patch4 (root@x4270) > (ia64-linux-gcc (GCC) 12.2.0, GNU ld (GNU Binutils) 2.39) #1 SMP Sun May > 28 09:08:44 CEST 2023 > [0.00] efi: EFI v2.1 by HP > [0.00] efi: SALsystab=0xdfdd63a18 ESI=0xdfdd63f18 ACPI > 2.0=0x3d3c4014 HCDP=0xd8798 SMBIOS=0x3d368000 > [0.00] PCDP: v3 at 0xd8798 > [0.00] earlycon: uart8250 at I/O port 0x4000 (options '115200n8') > [0.00] printk: bootconsole [uart8250] enabled > [0.00] ACPI: Early table checksum verification disabled > [0.00] ACPI: RSDP 0x3D3C4014 24 (v02 HP) > [0.00] ACPI: XSDT 0x3D3C4580 000124 (v01 HP RX2800-2 > 0001 0113) > [...] > [ 36.649531] Run /init as init process > Loading, please wait... > Starting systemd-udevd version 252.6-1 > [ 36.865635] pps_core: LinuxPPS API ver. 1 registered > [ 36.869321] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 > Rodolfo Giometti > [ 36.885025] PTP clock support registered > [ 36.910852] ACPI: bus type USB registered > [ 36.918198] usbcore: registered new interface driver usbfs > [ 36.924762] usbcore: registered new interface driver hub > [ 36.922133] igb: Intel(R) Gigabit Ethernet Network Driver > [ 36.931386] usbcore: registered new device driver usb > [ 36.938149] igb: Copyright (c) 2007-2014 Intel Corporation. > [...] > [ OK ] Finished apt-daily-upgrade… apt upgrade and clean activities. > > Debian GNU/Linux 12 rx2800-i2 - > > rx2800-i2 login: > ``` > > I'll try to test this on other machines (rx4640 w/Madison, rx2660 > w/Montecito/Montvale, rx6600 w/Montvale) as well, starting on Tuesday if > possible. > > > > There is an issue - only for the rx2800-i2 - seemingly related to it's > PCIe NIC(s) and MSIs, but this is already there in 6.3.x (see first part > of [1]) and **not related to the problem of this thread (AFAICT)** and - > more important - doesn't affect operation: The machine is working > diskless using one of those interfaces so it can't be that bad. I'll > look into bisecting that issue as well. > > [1]: https://lists.debian.org/debian-ia64/2023/05/msg00010.html > > Cheers, > Frank
Re: Boot regression in Linux v6.4-rc3
Hi!
On Sun, 2023-05-28 at 09:30 +0200, Frank Scheiner wrote:
> > Frank, could you please give it a try?
> >
> > Thanks,
> > Song
> >
> > diff --git i/kernel/module/main.c w/kernel/module/main.c
> > index 0f9183f1ca9f..e4e723e1eb21 100644
> > --- i/kernel/module/main.c
> > +++ w/kernel/module/main.c
> > @@ -1514,14 +1514,14 @@ static void __layout_sections(struct module
> > *mod, struct load_info *info, bool i
> > MOD_RODATA,
> > MOD_RO_AFTER_INIT,
> > MOD_DATA,
> > - MOD_INVALID,/* This is needed to match the masks array
> > */
> > + MOD_DATA,
> > };
> > static const int init_m_to_mem_type[] = {
> > MOD_INIT_TEXT,
> > MOD_INIT_RODATA,
> > MOD_INVALID,
> > MOD_INIT_DATA,
> > - MOD_INVALID,/* This is needed to match the masks array
> > */
> > + MOD_INIT_DATA,
> > };
> >
> > for (m = 0; m < ARRAY_SIZE(masks); ++m) {
>
> Thanks, that patch (as -patch4 on top of v6.4-rc3) fixes the boot
> regression for me on the rx2620:
Sounds good. Would be great to get this into 6.4 before release.
Adrian
--
.''`. John Paul Adrian Glaubitz
: :' : Debian Developer
`. `' Physicist
`-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: Boot regression in Linux v6.4-rc3
Hi again, On 28.05.23 09:30, Frank Scheiner wrote: [...] Thanks, that patch (as -patch4 on top of v6.4-rc3) fixes the boot regression for me on the rx2620: [...] Great! I'll give it a try on my rx2800-i2, too, but assume it wil work there, too. Indeed, -patch4 also makes it work on the rx2800-i2: ``` ELILO v3.16 for EFI/IA-64 .. Uncompressing Linux... done Loading file AC10027B.initrd.img...done [0.00] Linux version 6.4.0-rc3-44c026a73be8038f03dbdeef028b642880cf1511-patch4 (root@x4270) (ia64-linux-gcc (GCC) 12.2.0, GNU ld (GNU Binutils) 2.39) #1 SMP Sun May 28 09:08:44 CEST 2023 [0.00] efi: EFI v2.1 by HP [0.00] efi: SALsystab=0xdfdd63a18 ESI=0xdfdd63f18 ACPI 2.0=0x3d3c4014 HCDP=0xd8798 SMBIOS=0x3d368000 [0.00] PCDP: v3 at 0xd8798 [0.00] earlycon: uart8250 at I/O port 0x4000 (options '115200n8') [0.00] printk: bootconsole [uart8250] enabled [0.00] ACPI: Early table checksum verification disabled [0.00] ACPI: RSDP 0x3D3C4014 24 (v02 HP) [0.00] ACPI: XSDT 0x3D3C4580 000124 (v01 HP RX2800-2 0001 0113) [...] [ 36.649531] Run /init as init process Loading, please wait... Starting systemd-udevd version 252.6-1 [ 36.865635] pps_core: LinuxPPS API ver. 1 registered [ 36.869321] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 36.885025] PTP clock support registered [ 36.910852] ACPI: bus type USB registered [ 36.918198] usbcore: registered new interface driver usbfs [ 36.924762] usbcore: registered new interface driver hub [ 36.922133] igb: Intel(R) Gigabit Ethernet Network Driver [ 36.931386] usbcore: registered new device driver usb [ 36.938149] igb: Copyright (c) 2007-2014 Intel Corporation. [...] [ OK ] Finished apt-daily-upgrade… apt upgrade and clean activities. Debian GNU/Linux 12 rx2800-i2 - rx2800-i2 login: ``` I'll try to test this on other machines (rx4640 w/Madison, rx2660 w/Montecito/Montvale, rx6600 w/Montvale) as well, starting on Tuesday if possible. There is an issue - only for the rx2800-i2 - seemingly related to it's PCIe NIC(s) and MSIs, but this is already there in 6.3.x (see first part of [1]) and **not related to the problem of this thread (AFAICT)** and - more important - doesn't affect operation: The machine is working diskless using one of those interfaces so it can't be that bad. I'll look into bisecting that issue as well. [1]: https://lists.debian.org/debian-ia64/2023/05/msg00010.html Cheers, Frank
Re: Boot regression in Linux v6.4-rc3
Hi Song, Linus,
On 28.05.23 07:24, Song Liu wrote:
AFAICT, .got should go to rodata, while .sdata and .sbss should go
to (rw)data. However, reading the code before the module_memory
change, I think they were all copied to (rw)data, which is not ideal but
most likely OK.
To match the behavior before the module_memory change, I think
we need something like the following.
Frank, could you please give it a try?
Thanks,
Song
diff --git i/kernel/module/main.c w/kernel/module/main.c
index 0f9183f1ca9f..e4e723e1eb21 100644
--- i/kernel/module/main.c
+++ w/kernel/module/main.c
@@ -1514,14 +1514,14 @@ static void __layout_sections(struct module
*mod, struct load_info *info, bool i
MOD_RODATA,
MOD_RO_AFTER_INIT,
MOD_DATA,
- MOD_INVALID,/* This is needed to match the masks array */
+ MOD_DATA,
};
static const int init_m_to_mem_type[] = {
MOD_INIT_TEXT,
MOD_INIT_RODATA,
MOD_INVALID,
MOD_INIT_DATA,
- MOD_INVALID,/* This is needed to match the masks array */
+ MOD_INIT_DATA,
};
for (m = 0; m < ARRAY_SIZE(masks); ++m) {
Thanks, that patch (as -patch4 on top of v6.4-rc3) fixes the boot
regression for me on the rx2620:
```
ELILO v3.16 for EFI/IA-64
..
Uncompressing Linux... done
Loading file AC100221.initrd.img...done
[0.00] Linux version
6.4.0-rc3-44c026a73be8038f03dbdeef028b642880cf1511-patch4 (root@x4270)
(ia64-linux-gcc (GCC) 12.2.0, GNU ld (GNU Binutils) 2.39) #1 SMP Sun May
28 09:08:44 CEST 2023
[0.00] efi: EFI v1.1 by HP
[0.00] efi: SALsystab=0x3ee7a000 ACPI 2.0=0x3fe2a000
ESI=0x3ee7b000 SMBIOS=0x3ee7c000 HCDP=0x3fe28000
[0.00] PCDP: v3 at 0x3fe28000
[0.00] earlycon: uart8250 at MMIO 0xf405 (options
'9600n8')
[0.00] printk: bootconsole [uart8250] enabled
[0.00] ACPI: Early table checksum verification disabled
[0.00] ACPI: RSDP 0x3FE2A000 28 (v02 HP)
[0.00] ACPI: XSDT 0x3FE2A02C CC (v01 HP rx2620
HP )
[...]
[3.810346] Run /init as init process
Loading, please wait...
Starting systemd-udevd version 252.6-1
[3.985378] e1000: Intel(R) PRO/1000 Network Driver
[3.989378] e1000: Copyright (c) 1999-2006 Intel Corporation.
[3.993375] GSI 29 (level, low) -> CPU 7 (0x0700) vector 53
[4.030382] ACPI: bus type USB registered
[4.030382] usbcore: registered new interface driver usbfs
[4.034382] usbcore: registered new interface driver hub
[4.034382] usbcore: registered new device driver usb
[4.040609] GSI 18 (level, low) -> CPU 0 (0x) vector 54
[4.040621] ehci-pci :00:01.2: EHCI Host Controller
[...]
[ OK ] Finished systemd-update-ut… - Record Runlevel Change in UTMP.
[ 14.568606] ioc1: LSI53C1030 C0: Capabilities={Initiator,Target}
Debian GNU/Linux 12 rx2620 -
rx2620 login:
```
Great! I'll give it a try on my rx2800-i2, too, but assume it wil work
there, too.
Cheers,
Frank
Re: Boot regression in Linux v6.4-rc3
On Sat, May 27, 2023 at 12:34 PM Linus Torvalds
wrote:
>
> On Sat, May 27, 2023 at 11:41 AM Frank Scheiner wrote:
> >
> > Ok, I put the decoded console messages on [2].
> >
> > [2]: https://pastebin.com/dLYMijfS
>
> Ugh. Apparently ia64 decoding isn't great. But at least it gives
> multiple line numbers:
>
>load_module (kernel/module/main.c:2291 kernel/module/main.c:2412
> kernel/module/main.c:2868)
>
> except your kernel obviously has those test-patches, so I still don't
> know exactly where they are.
>
> But it looks like it is in move_module(). Strange. I don't know how it
> gets to "__copy_user" from there...
>
> [ Looks at the ia64 code ]
>
> Oh.
>
> It turns out that it *says* __copy_user(), but the code is actually
> shared with the regular memcpy() function, which does
>
> GLOBAL_ENTRY(memcpy)
> and r28=0x7,in0
> and r29=0x7,in1
> mov f6=f0
> mov retval=in0
> br.cond.sptk .common_code
> ;;
>
> where that ".common_code" label is - surprise surprise - the common
> copy code, and so when the oops reports that the problem happened in
> __copy_user(), it actually is in this case just a normal memcpy.
>
> Ok, so it's probably the
>
> memcpy(dest, (void *)shdr->sh_addr, shdr->sh_size);
>
> in move_module() that takes a fault. And looking at the registers,
> the destination is in r17/r18, and your dump has
>
> unable to handle kernel paging request at virtual address 1000
> ...
> r17 : 0fff r18 : 1000
>
> so it's almost certainly that 'dest' that is bad.
Yeah, it appears we are writing to mod_mem[MOD_INVALID].
>From the log, the following sections are assigned to MOD_INVALID:
[ 4.009109] __layout_sections: section .got (sh_flags 1002)
matched to MOD_INVALID
[ 4.009109] __layout_sections: section .sdata (sh_flags 1003)
matched to MOD_INVALID
[ 4.009109] __layout_sections: section .sbss (sh_flags 1003)
matched to MOD_INVALID
AFAICT, .got should go to rodata, while .sdata and .sbss should go
to (rw)data. However, reading the code before the module_memory
change, I think they were all copied to (rw)data, which is not ideal but
most likely OK.
To match the behavior before the module_memory change, I think
we need something like the following.
Frank, could you please give it a try?
Thanks,
Song
diff --git i/kernel/module/main.c w/kernel/module/main.c
index 0f9183f1ca9f..e4e723e1eb21 100644
--- i/kernel/module/main.c
+++ w/kernel/module/main.c
@@ -1514,14 +1514,14 @@ static void __layout_sections(struct module
*mod, struct load_info *info, bool i
MOD_RODATA,
MOD_RO_AFTER_INIT,
MOD_DATA,
- MOD_INVALID,/* This is needed to match the masks array */
+ MOD_DATA,
};
static const int init_m_to_mem_type[] = {
MOD_INIT_TEXT,
MOD_INIT_RODATA,
MOD_INVALID,
MOD_INIT_DATA,
- MOD_INVALID,/* This is needed to match the masks array */
+ MOD_INIT_DATA,
};
for (m = 0; m < ARRAY_SIZE(masks); ++m) {
Re: Boot regression in Linux v6.4-rc3
Hi, On 27.05.23 21:34, Linus Torvalds wrote: On Sat, May 27, 2023 at 11:41 AM Frank Scheiner wrote: Ok, I put the decoded console messages on [2]. [2]: https://pastebin.com/dLYMijfS Ugh. Apparently ia64 decoding isn't great. But at least it gives multiple line numbers: load_module (kernel/module/main.c:2291 kernel/module/main.c:2412 kernel/module/main.c:2868) except your kernel obviously has those test-patches, so I still don't know exactly where they are. Erm, I see. I did recreate a vanilla v6.4-rc3 and ran that, decoded result is on [1] - not sure if it makes it a little better. [1]: https://pastebin.com/z5XzEnhq I did also try to build and run a SP kernel to maybe get a better picture in the traces, but that seems to require FLATMEM, which seems to not work on that machine or due to the way it is configured (and yeah, it was also the wrong commit I used for it and it was patched...): ``` [0.00] Linux version 6.4.0-rc3-933174ae28ba72ab8de5b35cb7c98fc211235096-patch3_sp (root@x4270) (ia64-linux-gcc (GCC) 12.2.0, GNU ld (GNU Binutils) 2.39) #1 Sat May 27 21:28:44 CEST 2023 [...] [0.00] ACPI: SSDT 0x3FE35BA8 00013C (v01 HP rx2620 0006 INTL 20050309) [0.00] ACPI: Local APIC address (ptrval) [0.00] 1 CPUs available, 1 CPUs total [...] [0.00] Kernel panic - not syncing: Cannot use FLATMEM with 246784MB hole [0.00] Please switch over to SPARSEMEM [0.00] ---[ end Kernel panic - not syncing: Cannot use FLATMEM with 246784MB hole [0.00] Please switch over to SPARSEMEM ]--- ``` But it looks like it is in move_module(). Strange. I don't know how it gets to "__copy_user" from there... [ Looks at the ia64 code ] Oh. It turns out that it *says* __copy_user(), but the code is actually shared with the regular memcpy() function, which does GLOBAL_ENTRY(memcpy) and r28=0x7,in0 and r29=0x7,in1 mov f6=f0 mov retval=in0 br.cond.sptk .common_code ;; where that ".common_code" label is - surprise surprise - the common copy code, and so when the oops reports that the problem happened in __copy_user(), it actually is in this case just a normal memcpy. Ok, so it's probably the memcpy(dest, (void *)shdr->sh_addr, shdr->sh_size); in move_module() that takes a fault. And looking at the registers, the destination is in r17/r18, and your dump has unable to handle kernel paging request at virtual address 1000 ... r17 : 0fff r18 : 1000 so it's almost certainly that 'dest' that is bad. Which I guess shouldn't surprise anybody. But that's where my knowledge of ia64 and the new module loader layout ends. Thanks for your help and going as far as you could, that's greatly appreciated. Running that stuff is surely easier than debugging it. :-) Cheers, Frank
Re: Boot regression in Linux v6.4-rc3
On Sat, May 27, 2023 at 11:41 AM Frank Scheiner wrote: > > Ok, I put the decoded console messages on [2]. > > [2]: https://pastebin.com/dLYMijfS Ugh. Apparently ia64 decoding isn't great. But at least it gives multiple line numbers: load_module (kernel/module/main.c:2291 kernel/module/main.c:2412 kernel/module/main.c:2868) except your kernel obviously has those test-patches, so I still don't know exactly where they are. But it looks like it is in move_module(). Strange. I don't know how it gets to "__copy_user" from there... [ Looks at the ia64 code ] Oh. It turns out that it *says* __copy_user(), but the code is actually shared with the regular memcpy() function, which does GLOBAL_ENTRY(memcpy) and r28=0x7,in0 and r29=0x7,in1 mov f6=f0 mov retval=in0 br.cond.sptk .common_code ;; where that ".common_code" label is - surprise surprise - the common copy code, and so when the oops reports that the problem happened in __copy_user(), it actually is in this case just a normal memcpy. Ok, so it's probably the memcpy(dest, (void *)shdr->sh_addr, shdr->sh_size); in move_module() that takes a fault. And looking at the registers, the destination is in r17/r18, and your dump has unable to handle kernel paging request at virtual address 1000 ... r17 : 0fff r18 : 1000 so it's almost certainly that 'dest' that is bad. Which I guess shouldn't surprise anybody. But that's where my knowledge of ia64 and the new module loader layout ends. Linus
Re: Boot regression in Linux v6.4-rc3
Hi, On 27.05.23 19:08, Linus Torvalds wrote: Anyway, the WARN_ON() is likely related, but the bug is clearly an unexpected page fault in __copy_user() when called by load_module(). The ia64 oops output is nasty, presumably because ia64 aggressively inlines things. It would help a lot if you enabled debug info (maybe you already do?) I believe it is enabled - I have at least CONFIG_DEBUG_KERNEL=y and CONFIG_DEBUG_INFO=y - my kernel config is on [1] for reference. [1]: https://pastebin.com/HRQtZ9vb and then run the oops through ./scripts/decode_stacktrace.sh which will figure out line numbers, inlining etc. Because I don't even see why it would call __copy_user() in the first place. This is 'finit_module()' that loads the module data from a file, not user space. So I guess it must be the strndup_user() in mod->args = strndup_user(uargs, ~0UL >> 1); but that doesn't look like it should even care about any module layout. Plus I would have expected to see strndup_user() in the call trace, but whatever. End result: that ia64 trace is very hard to read, and _maybe_ running it through the decode script might give more information about what it is that triggers... Ok, I put the decoded console messages on [2]. [2]: https://pastebin.com/dLYMijfS Cheers, Frank
Re: Boot regression in Linux v6.4-rc3
On Sat, May 27, 2023 at 12:01 AM Frank Scheiner wrote: > > If it is of any help, my initial report is available for example via: > > https://marc.info/?l=linux-ia64=168509859125505=2 > > ...the whole thread is currently at: > > https://marc.info/?t=16850986823=1=2 This does make it clear just how great a mailing list archive lore is. Konstantin, is there any particular reason why linux-i...@vger.kernel.org isn't in lore? Is it just a rational hatred of all things itanium? Anyway, the WARN_ON() is likely related, but the bug is clearly an unexpected page fault in __copy_user() when called by load_module(). The ia64 oops output is nasty, presumably because ia64 aggressively inlines things. It would help a lot if you enabled debug info (maybe you already do?) and then run the oops through ./scripts/decode_stacktrace.sh which will figure out line numbers, inlining etc. Because I don't even see why it would call __copy_user() in the first place. This is 'finit_module()' that loads the module data from a file, not user space. So I guess it must be the strndup_user() in mod->args = strndup_user(uargs, ~0UL >> 1); but that doesn't look like it should even care about any module layout. Plus I would have expected to see strndup_user() in the call trace, but whatever. End result: that ia64 trace is very hard to read, and _maybe_ running it through the decode script might give more information about what it is that triggers... Linus
Re: Boot regression in Linux v6.4-rc3
On 27.05.23 00:22, Linus Torvalds wrote: [...] But this is my "monkey see, monkey do" pattern matching reaction, not from any deeper understanding of the problem (I can't even see the report) or really even the code. If it is of any help, my initial report is available for example via: https://marc.info/?l=linux-ia64=168509859125505=2 ...the whole thread is currently at: https://marc.info/?t=16850986823=1=2 Cheers, Frank
Re: Boot regression in Linux v6.4-rc3
Hi,
On 26.05.23 23:01, Song Liu wrote:
Thanks for running the test.
Thanks for staying with me.
I am not very familiar with the code, but I think we shouldn't hit that
WARN_ON_ONCE. Could you please try with the follow patch to see
which section caused this issue?
Thanks,
Song
diff --git i/kernel/module/main.c w/kernel/module/main.c
index 0f9183f1ca9f..caf3d30cd133 100644
--- i/kernel/module/main.c
+++ w/kernel/module/main.c
@@ -1537,8 +1537,11 @@ static void __layout_sections(struct module
*mod, struct load_info *info, bool i
|| is_init != module_init_layout_section(sname))
continue;
- if (WARN_ON_ONCE(type == MOD_INVALID))
+ if (WARN_ON_ONCE(type == MOD_INVALID)) {
+ pr_warn("%s: section %s (sh_flags
%llx) matched to MOD_INVALID\n", __func__,
+ sname, s->sh_flags);
continue;
+ }
s->sh_entsize =
module_get_offset_and_type(mod, type, s, i);
pr_debug("\t%s\n", sname);
I put that as -patch3 on top of 6.4-rc3, the result is on [1].
[1]: https://pastebin.com/KqnWL2pM
I this time put the whole console messages there, just in case some of
the earlier messages could be of any help. To jump to the actual oopses,
search for "Loading, please wait...".
Cheers,
Frank
Re: Boot regression in Linux v6.4-rc3
On Fri, May 26, 2023 at 12:55:14PM +0200, Frank Scheiner wrote:
> Dear all,
>
> there is a boot regression in effect in Linux v6.4-rc3 that affects at
> least:
>
> * rx2620 (w/2 x Montecito and zx1)
> * rx2800-i2 (w/1 x Tukwila)
Jesus, ia64 is even dropped from qemu as of 2.11. We're now around qemu
7.11 to give some perspective. I'm just wondering how to reproduce
testing easily instead of this ping pong back and forth for this
architecture for some oddball architectures.
Through commit 96ec72a3425d1 ("ia64: Mark architecture as orphaned")
it was noted even the old maintainer no longer had access to working
machines and so it was orphan'd.
Not saying that debugging commit ac3b4328392344 ("module: replace
module_layout with module_memory") is going to be impossible, quite
the contrary I think it would be good to root cause it, if possible,
as perhaps it may also be similar to some other future oddball arch
bug later that may come up.
But certainly just trying to see what options we have to test this
architecture.
And what's the status of removal for ia64 anyway?
Luis
Re: Boot regression in Linux v6.4-rc3
On Fri, May 26, 2023 at 3:22 PM Linus Torvalds
wrote:
>
> On Fri, May 26, 2023 at 2:59 PM Luis Chamberlain wrote:
> >
> > Not saying that debugging commit ac3b4328392344 ("module: replace
> > module_layout with module_memory") is going to be impossible, quite
> > the contrary I think it would be good to root cause it, if possible,
> > as perhaps it may also be similar to some other future oddball arch
> > bug later that may come up.
>
> I don't have any context - the mailing lists in question that
> apparently this came in on aren't in lore.
>
> That said, that commit looks odd for the ia64 part.
>
> In particular, this part:
>
> - if (mod->core_layout.size > MAX_LTOFF)
> + struct module_memory *mod_mem;
> +
> + mod_mem = >mem[MOD_DATA];
>
> in apply_relocate_add() (file: arch/ia64/kernel/module.c) seems suspect.
>
> The previous place that used to look at "mod->core_layout.base"
> converted that to "mod->mem[MOD_TEXT].base". As do other changes in
> other architectures.
>
> So that "MOD_DATA" looks *very* wrong. Shouldn't core_layout. be
> translated to use "MOD_TEXT" instead?
MOD_DATA is likely wrong here. But as Frank tested, changing it to MOD_TEXT
didn't fix the issue. I suspect we missed some special cases when we updated
layout_sections().
Thanks,
Song
Re: Boot regression in Linux v6.4-rc3
On Fri, May 26, 2023 at 2:59 PM Luis Chamberlain wrote:
>
> Not saying that debugging commit ac3b4328392344 ("module: replace
> module_layout with module_memory") is going to be impossible, quite
> the contrary I think it would be good to root cause it, if possible,
> as perhaps it may also be similar to some other future oddball arch
> bug later that may come up.
I don't have any context - the mailing lists in question that
apparently this came in on aren't in lore.
That said, that commit looks odd for the ia64 part.
In particular, this part:
- if (mod->core_layout.size > MAX_LTOFF)
+ struct module_memory *mod_mem;
+
+ mod_mem = >mem[MOD_DATA];
in apply_relocate_add() (file: arch/ia64/kernel/module.c) seems suspect.
The previous place that used to look at "mod->core_layout.base"
converted that to "mod->mem[MOD_TEXT].base". As do other changes in
other architectures.
So that "MOD_DATA" looks *very* wrong. Shouldn't core_layout. be
translated to use "MOD_TEXT" instead?
Nothing else in the ia64 parts strike me as odd, but that one looks wrong to me.
But this is my "monkey see, monkey do" pattern matching reaction, not
from any deeper understanding of the problem (I can't even see the
report) or really even the code.
Linus
Re: Boot regression in Linux v6.4-rc3
On Fri, May 26, 2023 at 11:36 AM Frank Scheiner wrote:
>
> Hi Song,
>
> On 26.05.23 18:49, Song Liu wrote:
> > Hi Frank,
> >
> > Thanks for the report.
>
> Sure, thanks for your help in this.
>
> > It seems the error happened during the WARN_ON_ONCE. Could you
> > please try whether something like the following fixes it?
> >
> > diff --git i/kernel/module/main.c w/kernel/module/main.c
> > index 0f9183f1ca9f..ae42dfc1a815 100644
> > --- i/kernel/module/main.c
> > +++ w/kernel/module/main.c
> > @@ -1537,7 +1537,7 @@ static void __layout_sections(struct module
> > *mod, struct load_info *info, bool i
> > || is_init !=
> > module_init_layout_section(sname))
> > continue;
> >
> > - if (WARN_ON_ONCE(type == MOD_INVALID))
> > + if (type == MOD_INVALID)
> > continue;
> >
> > s->sh_entsize =
> > module_get_offset_and_type(mod, type, s, i);
>
> Ok, tried that as -patch1 on top of v6.4-rc3, but didn't help, see [1].
>
> [1]: https://pastebin.com/UK9v30Ae
>
> > If that doesn't work, maybe we need something like this:
> >
> > diff --git i/arch/ia64/kernel/module.c w/arch/ia64/kernel/module.c
> > index 3661135da9d9..4e9a7f0498e2 100644
> > --- i/arch/ia64/kernel/module.c
> > +++ w/arch/ia64/kernel/module.c
> > @@ -815,7 +815,7 @@ apply_relocate_add (Elf64_Shdr *sechdrs, const
> > char *strtab, unsigned int symind
> > uint64_t gp;
> > struct module_memory *mod_mem;
> >
> > - mod_mem = >mem[MOD_DATA];
> > + mod_mem = >mem[MOD_TEXT];
> > if (mod_mem->size > MAX_LTOFF)
> > /*
> > * This takes advantage of fact that
> > SHF_ARCH_SMALL gets allocated
>
> Tried that one as -patch2 on top of v6.4-rc3, but didn't help, see [2].
>
> [2]: https://pastebin.com/gLupBndU
>
> I also tried both patches as -patch1plus2 on top of v6.4-rc3 with a
> similar result, see [3].
>
> [3]: https://pastebin.com/7pXBG5vx
Thanks for running the test.
I am not very familiar with the code, but I think we shouldn't hit that
WARN_ON_ONCE. Could you please try with the follow patch to see
which section caused this issue?
Thanks,
Song
diff --git i/kernel/module/main.c w/kernel/module/main.c
index 0f9183f1ca9f..caf3d30cd133 100644
--- i/kernel/module/main.c
+++ w/kernel/module/main.c
@@ -1537,8 +1537,11 @@ static void __layout_sections(struct module
*mod, struct load_info *info, bool i
|| is_init != module_init_layout_section(sname))
continue;
- if (WARN_ON_ONCE(type == MOD_INVALID))
+ if (WARN_ON_ONCE(type == MOD_INVALID)) {
+ pr_warn("%s: section %s (sh_flags
%llx) matched to MOD_INVALID\n", __func__,
+ sname, s->sh_flags);
continue;
+ }
s->sh_entsize =
module_get_offset_and_type(mod, type, s, i);
pr_debug("\t%s\n", sname);
Re: Boot regression in Linux v6.4-rc3
Hi Song, On 26.05.23 18:49, Song Liu wrote: Hi Frank, Thanks for the report. Sure, thanks for your help in this. It seems the error happened during the WARN_ON_ONCE. Could you please try whether something like the following fixes it? diff --git i/kernel/module/main.c w/kernel/module/main.c index 0f9183f1ca9f..ae42dfc1a815 100644 --- i/kernel/module/main.c +++ w/kernel/module/main.c @@ -1537,7 +1537,7 @@ static void __layout_sections(struct module *mod, struct load_info *info, bool i || is_init != module_init_layout_section(sname)) continue; - if (WARN_ON_ONCE(type == MOD_INVALID)) + if (type == MOD_INVALID) continue; s->sh_entsize = module_get_offset_and_type(mod, type, s, i); Ok, tried that as -patch1 on top of v6.4-rc3, but didn't help, see [1]. [1]: https://pastebin.com/UK9v30Ae If that doesn't work, maybe we need something like this: diff --git i/arch/ia64/kernel/module.c w/arch/ia64/kernel/module.c index 3661135da9d9..4e9a7f0498e2 100644 --- i/arch/ia64/kernel/module.c +++ w/arch/ia64/kernel/module.c @@ -815,7 +815,7 @@ apply_relocate_add (Elf64_Shdr *sechdrs, const char *strtab, unsigned int symind uint64_t gp; struct module_memory *mod_mem; - mod_mem = >mem[MOD_DATA]; + mod_mem = >mem[MOD_TEXT]; if (mod_mem->size > MAX_LTOFF) /* * This takes advantage of fact that SHF_ARCH_SMALL gets allocated Tried that one as -patch2 on top of v6.4-rc3, but didn't help, see [2]. [2]: https://pastebin.com/gLupBndU I also tried both patches as -patch1plus2 on top of v6.4-rc3 with a similar result, see [3]. [3]: https://pastebin.com/7pXBG5vx Cheers, Frank
Re: Boot regression in Linux v6.4-rc3
Hi Frank, Thanks for the report. On Fri, May 26, 2023 at 3:55 AM Frank Scheiner wrote: > > Dear all, > > there is a boot regression in effect in Linux v6.4-rc3 that affects at > least: > > * rx2620 (w/2 x Montecito and zx1) > * rx2800-i2 (w/1 x Tukwila) > > ...(see second part of [1] and following posts for more details, [2] and > [3] for the respective logs), example here: > > ``` > ELILO v3.16 for EFI/IA-64 > .. > Uncompressing Linux... done > Loading file AC100221.initrd.img...done > [0.00] Linux version 6.4.0-rc3 (root@x4270) (ia64-linux-gcc > (GCC) 12.2.0, GNU ld (GNU Binutils) 2.39) #1 SMP Thu May 25 15:52:20 > CEST 2023 > [0.00] efi: EFI v1.1 by HP > [0.00] efi: SALsystab=0x3ee7a000 ACPI 2.0=0x3fe2a000 > ESI=0x3ee7b000 SMBIOS=0x3ee7c000 HCDP=0x3fe28000 > [0.00] PCDP: v3 at 0x3fe28000 > [0.00] earlycon: uart8250 at MMIO 0xf405 (options > '9600n8') > [0.00] printk: bootconsole [uart8250] enabled > [0.00] ACPI: Early table checksum verification disabled > [0.00] ACPI: RSDP 0x3FE2A000 28 (v02 HP) > [0.00] ACPI: XSDT 0x3FE2A02C CC (v01 HP rx2620 > HP ) > [...] > [3.793350] Run /init as init process > Loading, please wait... > Starting systemd-udevd version 252.6-1 > [3.951100] [ cut here ] > [3.951100] WARNING: CPU: 6 PID: 140 at kernel/module/main.c:1547 > __layout_sections+0x370/0x3c0 > [3.949512] Unable to handle kernel paging request at virtual address > 1000 > [3.951100] Modules linked in: > [3.951100] CPU: 6 PID: 140 Comm: (udev-worker) Not tainted 6.4.0-rc3 #1 > [3.956161] (udev-worker)[142]: Oops 11003706212352 [1] > [3.951774] Hardware name: hp server rx2620 , BIOS > 04.29 > 11/30/2007 > [3.951774] > [3.951774] Call Trace: > [3.958339] Unable to handle kernel paging request at virtual address > 1000 > [3.956161] Modules linked in: > [3.951774] [] show_stack.part.0+0x30/0x60 > [3.951774] sp=e00183a67b20 > bsp=e00183a61628 > [3.956161] > [3.956161] > ``` > > [1]: https://lists.debian.org/debian-ia64/2023/05/msg00010.html > > [2]: https://pastebin.com/SAUKbG7Z > > [3]: https://pastebin.com/v1TTB2x3 It seems the error happened during the WARN_ON_ONCE. Could you please try whether something like the following fixes it? diff --git i/kernel/module/main.c w/kernel/module/main.c index 0f9183f1ca9f..ae42dfc1a815 100644 --- i/kernel/module/main.c +++ w/kernel/module/main.c @@ -1537,7 +1537,7 @@ static void __layout_sections(struct module *mod, struct load_info *info, bool i || is_init != module_init_layout_section(sname)) continue; - if (WARN_ON_ONCE(type == MOD_INVALID)) + if (type == MOD_INVALID) continue; s->sh_entsize = module_get_offset_and_type(mod, type, s, i); If that doesn't work, maybe we need something like this: diff --git i/arch/ia64/kernel/module.c w/arch/ia64/kernel/module.c index 3661135da9d9..4e9a7f0498e2 100644 --- i/arch/ia64/kernel/module.c +++ w/arch/ia64/kernel/module.c @@ -815,7 +815,7 @@ apply_relocate_add (Elf64_Shdr *sechdrs, const char *strtab, unsigned int symind uint64_t gp; struct module_memory *mod_mem; - mod_mem = >mem[MOD_DATA]; + mod_mem = >mem[MOD_TEXT]; if (mod_mem->size > MAX_LTOFF) /* * This takes advantage of fact that SHF_ARCH_SMALL gets allocated Song
