Re: kldload zfs spins the system after upgrading from 12.2 to 13-BETA

[Andriy Gapon]

On 20/03/2021 05:01, Yoshihiro Ota wrote:
> On Tue, 9 Mar 2021 11:24:53 +0200
> Andriy Gapon  wrote:
> 
>> On 08/03/2021 05:24, Yoshihiro Ota wrote:
>>> On Sun, 7 Mar 2021 00:09:33 +0200
>>> Andriy Gapon  wrote:
>>>
 On 06/03/2021 20:09, Yoshihiro Ota wrote:
> Hi all,
>
> I'm upgrading fron 12.2-RELEASE to 13-BETA/RC one by one.
>
> After upgrading one in VMWare, 'zfs mount -a' hangs the system.
> I don't have boottime zfs mount on nor don't have zfsroot.
> I just simply ran install world/kernel and mergemaster.

 Please use procstat -kk to capture a kernel stack trace of the hung 
 process.
>>>
>>> Actually, spining was 'kldload zfs'.
>>> Console doesn't response but ping and sshd sessions still work.
>>> procstat output is below.
>>> In addition, this doesn't happen to systems that I've been following 
>>> 13-CURRENT
>>> but rather happen only wiht a system upgraded from 12.2-RELEASE to 13-RC.
>>>
>>>
>>> # procstat -kk 1049
>>>   PIDTID COMMTDNAME  KSTACK 
>>>   
>>>  1049 100215 kldload -   spa_init+0xc6 
>>> zfs_kmod_init+0x1a
>>> zfs_modevent+0x34 module_register_init+0x8c linker_load_module+0xaab 
>>> kern_kldload+0xc1
>>> sys_kldload+0x50 syscall+0x17d g_ctx+0xe280bf29 
>>>
>>
>> If you could use kgdb to find out what source code line spa_init+0xc6
>> corresponds to that may help to see what's going on.
>>
> 
> It look me a while to get kgdb working properly.
> At last, I got the output.
> It looks it is spining on a mutex.
> 
> I have few other machines run the same kernel but they can load zfs.ko.
> It is only vmware VM that spins with 'kldload zfs'.
> 
> vmware# kgdb101 /usr/usr/lib/debug/boot/kernel/zfs.ko.debug
> GNU gdb (GDB) 10.1 [GDB v10.1 for FreeBSD]
> Copyright (C) 2020 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Type "show copying" and "show warranty" for details.
> This GDB was configured as "i386-portbld-freebsd13.0".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> .
> Find the GDB manual and other documentation resources online at:
> .
> 
> For help, type "helpType "apropos word" to search for commands related to 
> "word"...
> Reading symbols from zfs.ko.debug...
> (kgdb) info line *spa_init+0xc6
> Line 2345 of "/usr/src/sys/contrib/openzfs/module/zfs/spa_misc.c"
>starts at address 0x2b0461 
>and ends at 0x2b0467 .
> (kgdb) 
> 
> void
> spa_init(spa_mode_t mode)
> {
> mutex_init(_namespace_lock, NULL, MUTEX_DEFAULT, NULL);
> mutex_init(_spare_lock, NULL, MUTEX_DEFAULT, NULL); // <- line 
> 2345
> 

It would highly unusual (and unexplainable) for a thread to get stuck here.
Are you sure that your source tree matches the binary?
Can you disassemble the function to check instructions at and near 0xc6 offset?


-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: kldload zfs spins the system after upgrading from 12.2 to 13-BETA

[Yoshihiro Ota]

On Tue, 9 Mar 2021 11:24:53 +0200
Andriy Gapon  wrote:

> On 08/03/2021 05:24, Yoshihiro Ota wrote:
> > On Sun, 7 Mar 2021 00:09:33 +0200
> > Andriy Gapon  wrote:
> > 
> >> On 06/03/2021 20:09, Yoshihiro Ota wrote:
> >>> Hi all,
> >>>
> >>> I'm upgrading fron 12.2-RELEASE to 13-BETA/RC one by one.
> >>>
> >>> After upgrading one in VMWare, 'zfs mount -a' hangs the system.
> >>> I don't have boottime zfs mount on nor don't have zfsroot.
> >>> I just simply ran install world/kernel and mergemaster.
> >>
> >> Please use procstat -kk to capture a kernel stack trace of the hung 
> >> process.
> > 
> > Actually, spining was 'kldload zfs'.
> > Console doesn't response but ping and sshd sessions still work.
> > procstat output is below.
> > In addition, this doesn't happen to systems that I've been following 
> > 13-CURRENT
> > but rather happen only wiht a system upgraded from 12.2-RELEASE to 13-RC.
> > 
> > 
> > # procstat -kk 1049
> >   PIDTID COMMTDNAME  KSTACK 
> >   
> >  1049 100215 kldload -   spa_init+0xc6 
> > zfs_kmod_init+0x1a
> > zfs_modevent+0x34 module_register_init+0x8c linker_load_module+0xaab 
> > kern_kldload+0xc1
> > sys_kldload+0x50 syscall+0x17d g_ctx+0xe280bf29 
> > 
> 
> If you could use kgdb to find out what source code line spa_init+0xc6
> corresponds to that may help to see what's going on.
> 

It look me a while to get kgdb working properly.
At last, I got the output.
It looks it is spining on a mutex.

I have few other machines run the same kernel but they can load zfs.ko.
It is only vmware VM that spins with 'kldload zfs'.

vmware# kgdb101 /usr/usr/lib/debug/boot/kernel/zfs.ko.debug
GNU gdb (GDB) 10.1 [GDB v10.1 for FreeBSD]
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "i386-portbld-freebsd13.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.

For help, type "helpType "apropos word" to search for commands related to 
"word"...
Reading symbols from zfs.ko.debug...
(kgdb) info line *spa_init+0xc6
Line 2345 of "/usr/src/sys/contrib/openzfs/module/zfs/spa_misc.c"
   starts at address 0x2b0461 
   and ends at 0x2b0467 .
(kgdb) 

void
spa_init(spa_mode_t mode)
{
mutex_init(_namespace_lock, NULL, MUTEX_DEFAULT, NULL);
mutex_init(_spare_lock, NULL, MUTEX_DEFAULT, NULL); // <- line 2345


Hiro
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: kldload zfs spins the system after upgrading from 12.2 to 13-BETA

[Yoshihiro Ota]

On Tue, 9 Mar 2021 11:24:53 +0200
Andriy Gapon  wrote:

> On 08/03/2021 05:24, Yoshihiro Ota wrote:
> > On Sun, 7 Mar 2021 00:09:33 +0200
> > Andriy Gapon  wrote:
> > 
> >> On 06/03/2021 20:09, Yoshihiro Ota wrote:
> >>> Hi all,
> >>>
> >>> I'm upgrading fron 12.2-RELEASE to 13-BETA/RC one by one.
> >>>
> >>> After upgrading one in VMWare, 'zfs mount -a' hangs the system.
> >>> I don't have boottime zfs mount on nor don't have zfsroot.
> >>> I just simply ran install world/kernel and mergemaster.
> >>
> >> Please use procstat -kk to capture a kernel stack trace of the hung 
> >> process.
> > 
> > Actually, spining was 'kldload zfs'.
> > Console doesn't response but ping and sshd sessions still work.
> > procstat output is below.
> > In addition, this doesn't happen to systems that I've been following 
> > 13-CURRENT
> > but rather happen only wiht a system upgraded from 12.2-RELEASE to 13-RC.
> > 
> > 
> > # procstat -kk 1049
> >   PIDTID COMMTDNAME  KSTACK 
> >   
> >  1049 100215 kldload -   spa_init+0xc6 
> > zfs_kmod_init+0x1a
> > zfs_modevent+0x34 module_register_init+0x8c linker_load_module+0xaab 
> > kern_kldload+0xc1
> > sys_kldload+0x50 syscall+0x17d g_ctx+0xe280bf29 
> > 
> 
> If you could use kgdb to find out what source code line spa_init+0xc6
> corresponds to that may help to see what's going on.

Hi Andriy,

Could you teach me what command can show such address in kgdb?

Thanks,
Hiro
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: kldload zfs spins the system after upgrading from 12.2 to 13-BETA

[Andriy Gapon]

On 08/03/2021 05:24, Yoshihiro Ota wrote:
> On Sun, 7 Mar 2021 00:09:33 +0200
> Andriy Gapon  wrote:
> 
>> On 06/03/2021 20:09, Yoshihiro Ota wrote:
>>> Hi all,
>>>
>>> I'm upgrading fron 12.2-RELEASE to 13-BETA/RC one by one.
>>>
>>> After upgrading one in VMWare, 'zfs mount -a' hangs the system.
>>> I don't have boottime zfs mount on nor don't have zfsroot.
>>> I just simply ran install world/kernel and mergemaster.
>>
>> Please use procstat -kk to capture a kernel stack trace of the hung process.
> 
> Actually, spining was 'kldload zfs'.
> Console doesn't response but ping and sshd sessions still work.
> procstat output is below.
> In addition, this doesn't happen to systems that I've been following 
> 13-CURRENT
> but rather happen only wiht a system upgraded from 12.2-RELEASE to 13-RC.
> 
> 
> # procstat -kk 1049
>   PIDTID COMMTDNAME  KSTACK   
> 
>  1049 100215 kldload -   spa_init+0xc6 
> zfs_kmod_init+0x1a zfs_modevent+0x34 module_register_init+0x8c 
> linker_load_module+0xaab kern_kldload+0xc1 sys_kldload+0x50 syscall+0x17d 
> g_ctx+0xe280bf29 
> 

If you could use kgdb to find out what source code line spa_init+0xc6
corresponds to that may help to see what's going on.

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: kldload zfs spins the system after upgrading from 12.2 to 13-BETA

[Yoshihiro Ota]

On Sun, 7 Mar 2021 00:09:33 +0200
Andriy Gapon  wrote:

> On 06/03/2021 20:09, Yoshihiro Ota wrote:
> > Hi all,
> > 
> > I'm upgrading fron 12.2-RELEASE to 13-BETA/RC one by one.
> > 
> > After upgrading one in VMWare, 'zfs mount -a' hangs the system.
> > I don't have boottime zfs mount on nor don't have zfsroot.
> > I just simply ran install world/kernel and mergemaster.
> 
> Please use procstat -kk to capture a kernel stack trace of the hung process.

Actually, spining was 'kldload zfs'.
Console doesn't response but ping and sshd sessions still work.
procstat output is below.
In addition, this doesn't happen to systems that I've been following 13-CURRENT
but rather happen only wiht a system upgraded from 12.2-RELEASE to 13-RC.


# procstat -kk 1049
  PIDTID COMMTDNAME  KSTACK 
  
 1049 100215 kldload -   spa_init+0xc6 
zfs_kmod_init+0x1a zfs_modevent+0x34 module_register_init+0x8c 
linker_load_module+0xaab kern_kldload+0xc1 sys_kldload+0x50 syscall+0x17d 
g_ctx+0xe280bf29 
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"