== Bionic GA 4.15.0 kernel ==
ubuntu@P8lpar4:~$ sudo su
root@P8lpar4:/home/ubuntu# drmgr -c cpu -r -q 1
Validating CPU DLPAR capability...yes.
CPU 121
root@P8lpar4:/home/ubuntu# drmgr -c cpu -a -q 1
[262984.440091] BUG: arch topology borken
[262984.440110] the DIE domain not a subset of the NODE domain
[262984.440114] BUG: arch topology borken
[262984.440116] the DIE domain not a subset of the NODE domain
[262984.440120] BUG: arch topology borken
[262984.440122] the DIE domain not a subset of the NODE domain
[262984.443241] Unable to handle kernel paging request for data at address
0x50dbee4a
[262984.443261] Faulting instruction address: 0xc00000000017982c
[262984.443281] LE SMP NR_CPUS=2048 NUMA pSeries
m iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
ip_tables
ync_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear
ibmvscs
[262984.443323] CPU: 120 PID: 1467 Comm: kworker/120:2 Not tainted
4.15.0-52-generic
#56-Ubuntu
[262984.443331] Workqueue: events cpuset_hotplug_workfn
[262984.443334] NIP: c00000000017982c LR: c000000000179908 CTR:
0000000000000000
[262984.443341] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: 22824228
XER: 000
00005
[262984.443349] GPR00: c000000000179908 c000000fcec0b7b0 c0000000016eb800
c000000f7c4
[262984.443349] GPR04: 0000000000000001 0000000000000000 0000000000000008
00000000000
[262984.443349] GPR08: 0000000000000018 ffffffffffffffff c000000fe22f5808
00000000000
1c600
[262984.443349] GPR16: 00000000000003c0 c000000001722494 c000000fd249d800
00000000000
00001
[262984.443349] GPR20: c000000ff7068580 0000000000000008 c0000000011d8580
c0000000017
[262984.443349] GPR24: 0000000000000000 ffffffffffffe830 ffffffffffffec30
00000000000
0102f
65d75
[262984.443389] LR [c000000000179908] destroy_sched_domain+0x38/0xc0
[262984.443392] Call Trace:
[262984.443395] [c000000fcec0b7b0] [ffffffffffffec30] 0xffffffffffffec30
(unreliable)
[262984.443400] [c000000fcec0b7f0] [c000000000179908]
destroy_sched_domain+0x38/0xc0
[262984.443404] [c000000fcec0b820] [c000000000179e10]
cpu_attach_domain+0xf0/0x870
[262984.443408] [c000000fcec0b960] [c00000000017b7e4]
build_sched_domains+0x1254/0x12
f0
[262984.443412] [c000000fcec0ba90] [c00000000017c9d0]
partition_sched_domains+0x2d0/0
x410
[262984.443416] [c000000fcec0bb20] [c000000000202b20]
rebuild_sched_domains_locked+0x
60/0x80
[262984.443420] [c000000fcec0bb50] [c000000000205e28]
rebuild_sched_domains+0x38/0x60
[262984.443424] [c000000fcec0bb80] [c000000000205f88]
cpuset_hotplug_workfn+0x138/0xb
60
[262984.443429] [c000000fcec0bc90] [c000000000138798]
process_one_work+0x298/0x5a0
[262984.443433] [c000000fcec0bd20] [c000000000138b38] worker_thread+0x98/0x630
[262984.443436] [c000000fcec0bdc0] [c000000000141728] kthread+0x1a8/0x1b0
[262984.451901] [c000000fcec0be30] [c00000000000b65c]
ret_from_kernel_thread+0x5c/0x8
0
[262984.451904] Instruction dump:
[262984.451906] 7d908026 fbe1fff8 91810008 f8010010 f821ffc1 7c7d1b78 2e240000
7c7f1b
78
[262984.451912] 48000010 7fbee840 7fdff378 419e0074 <ebdf0000> 4192002c
7c0004ac e95f
0010
[262984.451919] ---[ end trace b6da6a7114e365f9 ]---
[262984.454656]
== After the patches are applied 4.15.0 ==
ubuntu@P8lpar4:~$ uname -a
Linux P8lpar4 4.15.0-53-generic #57~lp1833716+build.1-Ubuntu SMP Fri Jun 21
15:18:40 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux
ubuntu@P8lpar4:~$
ubuntu@P8lpar4:~$ sudo su
root@P8lpar4:/home/ubuntu# drmgr -c cpu -r -q 1
Validating CPU DLPAR capability...yes.
CPU 121
root@P8lpar4:/home/ubuntu# drmgr -c cpu -a -q 1
Validating CPU DLPAR capability...yes.
CPU 121
root@P8lpar4:/home/ubuntu#
Ubuntu 18.04.2 LTS P8lpar4 hvc0
P8lpar4 login: [ 25.687520] cloud-init[3439]: Cloud-init v.
19.1-1-gbaa47854-0ubunt
u1~18.04.1 running 'modules:final' at Mon, 24 Jun 2019 16:37:33 +0000. Up 25.30
secon
ds.
** Description changed:
[Impact]
On Bionic GA kernel (4.15.0), hot add of cpu with drmgr causes the kernel to
crash. The patches identified to fix these issues disables changing the NUMA
associations for CPUs and Memory at runtime by default.
[Test]
# drmgr -c cpu -r -q 1
# drmgr -c cpu -a -q 1
+ Test kernel available in ppa:ubuntu-power-triage/lp1833716
+ Please see comment #2 for before and after results with the patches applied.
[Fix]
558f86493df0 powerpc/numa: document topology_updates_enabled, disable by
default
2d4d9b308f8f powerpc/numa: improve control of topology updates
[Regression Potential]
The two patches relate to powerpc/numa and does not impact other
architectures or platform code. Regression potential is low.
[Other Information]
== Comment: #0 - Hari Krishna Bathini <[email protected]> - 2019-05-07
13:18:35 ==
---Problem Description---
On 4.15.0-48-generic kernel, hot adding a cpu with drmgr is crashing the
kernel
with below traces:
---
root@ubuntu:~# drmgr -c cpu -r -q 1
Validating CPU DLPAR capability...yes.
CPU 9
root@ubuntu:~#
root@ubuntu:~#
root@ubuntu:~#
root@ubuntu:~#
root@ubuntu:~#
root@ubuntu:~# drmgr -c cpu -a -q 1
Validating CPU DLPAR capability...yes.
[ 218.555493] BUG: arch topology borken
[ 218.555503] the DIE domain not a subset of the NODE domain
[ 218.555512] BUG: arch topology borken
[ 218.555516] the DIE domain not a subset of the NODE domain
[ 218.555523] BUG: arch topology borken
[ 218.555528] the DIE domain not a subset of the NODE domain
[ 218.555535] BUG: arch topology borken
[ 218.555539] the DIE domain not a subset of the NODE domain
[ 218.555545] BUG: arch topology borken
[ 218.555550] the DIE domain not a subset of the NODE domain
[ 218.555556] BUG: arch topology borken
[ 218.555560] the DIE domain not a subset of the NODE domain
[ 218.555567] BUG: arch topology borken
[ 218.555571] the DIE domain not a subset of the NODE domain
[ 218.555577] BUG: arch topology borken
[ 218.555581] the DIE domain not a subset of the NODE domain
[ 218.555672] Unable to handle kernel paging request for data at address
0x9332ae80f961139f
[ 218.555679] Faulting instruction address: 0xc0000000001768cc
[ 218.555686] Oops: Kernel access of bad area, sig: 11 [#1]
[ 218.555691] LE SMP NR_CPUS=2048 NUMA pSeries
[ 218.555699] Modules linked in: vmx_crypto crct10dif_vpmsum sch_fq_codel
ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10
raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq
libcrc32c raid1 raid0 multipath linear ibmvscsi ibmveth crc32c_vpmsum
[ 218.555745] CPU: 8 PID: 276 Comm: kworker/8:1 Not tainted
4.15.0-48-generic #51-Ubuntu
[ 218.555757] Workqueue: events cpuset_hotplug_workfn
[ 218.555763] NIP: c0000000001768cc LR: c0000000001769a8 CTR:
0000000000000000
[ 218.555770] REGS: c0000001f5f1f530 TRAP: 0380 Not tainted
(4.15.0-48-generic)
[ 218.555776] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: 22824228
XER: 00000004
[ 218.555789] CFAR: c000000000176920 SOFTE: 1
[ 218.555789] GPR00: c0000000001769a8 c0000001f5f1f7b0 c0000000016eb400
c0000001f7bfd200
[ 218.555789] GPR04: 0000000000000001 0000000000000000 0000000000000008
0000000000000010
[ 218.555789] GPR08: 0000000000000018 ffffffffffffffff c0000001f7bfd408
0000000000000000
[ 218.555789] GPR12: 0000000000008000 c000000007a35800 0000000000000007
c0000001f549d900
[ 218.555789] GPR16: 0000000000000040 c000000001722494 c0000001f0f29400
0000000000000001
[ 218.555789] GPR20: c0000001ffb68580 0000000000000008 c0000000011d8580
c00000000171dd78
[ 218.555789] GPR24: 0000000000000000 ffffffffffffe830 ffffffffffffec30
00000000000012af
[ 218.555789] GPR28: 000000000000102f c0000001f7bfd200 9332ae80f961139f
9332ae80f961139f
[ 218.555859] NIP [c0000000001768cc] free_sched_groups.part.2+0x4c/0xf0
[ 218.555866] LR [c0000000001769a8] destroy_sched_domain+0x38/0xc0
[ 218.555871] Call Trace:
[ 218.555875] [c0000001f5f1f7b0] [ffffffffffffec30] 0xffffffffffffec30
(unreliable)
[ 218.555884] [c0000001f5f1f7f0] [c0000000001769a8]
destroy_sched_domain+0x38/0xc0
[ 218.555892] [c0000001f5f1f820] [c000000000176eb0]
cpu_attach_domain+0xf0/0x870
[ 218.555900] [c0000001f5f1f960] [c000000000178884]
build_sched_domains+0x1254/0x12f0
[ 218.555908] [c0000001f5f1fa90] [c000000000179a70]
partition_sched_domains+0x2d0/0x410
[ 218.555916] [c0000001f5f1fb20] [c0000000001ffb60]
rebuild_sched_domains_locked+0x60/0x80
[ 218.555924] [c0000001f5f1fb50] [c000000000202e68]
rebuild_sched_domains+0x38/0x60
[ 218.555932] [c0000001f5f1fb80] [c000000000202fc8]
cpuset_hotplug_workfn+0x138/0xb60
[ 218.555941] [c0000001f5f1fc90] [c000000000135858]
process_one_work+0x298/0x5a0
[ 218.555949] [c0000001f5f1fd20] [c000000000135bf8] worker_thread+0x98/0x630
[ 218.555956] [c0000001f5f1fdc0] [c00000000013e7e8] kthread+0x1a8/0x1b0
[ 218.555964] [c0000001f5f1fe30] [c00000000000b658]
ret_from_kernel_thread+0x5c/0x84
[ 218.555971] Instruction dump:
[ 218.555975] 7d908026 fbe1fff8 91810008 f8010010 f821ffc1 7c7d1b78 2e240000
7c7f1b78
[ 218.555985] 48000010 7fbee840 7fdff378 419e0074 <ebdf0000> 4192002c
7c0004ac e95f0010
[ 218.555997] ---[ end trace 1d7b9b38e50835a4 ]---
---
---uname output---
Linux ubuntu 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:26:19 UTC 2019
ppc64le ppc64le ppc64le GNU/Linux
Machine Type = na
---Debugger---
A debugger is not configured
---Steps to Reproduce---
1. Install a 4.15 kernel (4.15.0-48-generic)
2. Hot remove a core: drmgr -c cpu -r -q 1
3. Hot add a core: drmgr -c cpu -a -q 1
Actual Result:
System crashes after "drmgr -c cpu -a -q 1" command is issued
Expected result:
Hot add succeeds without any crash
== Comment: #20 - SEETEENA THOUFEEK <[email protected]> - 2019-06-20
07:00:39 ==
Please integrate these two patches
1.
https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/commit/?h=next&id=2d4d9b308f8f8dec68f6dbbff18c68ec7c6bd26f
powerpc/numa: improve control of topology updates
When booted with "topology_updates=no", or when "off" is written to
/proc/powerpc/topology_updates, NUMA reassignments are inhibited for
PRRN and VPHN events. However, migration and suspend unconditionally
re-enable reassignments via start_topology_update(). This is
incoherent.
Check the topology_updates_enabled flag in
start/stop_topology_update() so that callers of those APIs need not be
aware of whether reassignments are enabled. This allows the
administrative decision on reassignments to remain in force across
migrations and suspensions.
Signed-off-by: Nathan Lynch <[email protected]>
Signed-off-by: Michael Ellerman <[email protected]>
2.
https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/commit/?h=next&id=558f86493df09f68f79fe056d9028d317a3ce8ab
powerpc/numa: document topology_updates_enabled, disable by default
Changing the NUMA associations for CPUs and memory at runtime is
basically unsupported by the core mm, scheduler etc. We see all manner
of crashes, warnings and instability when the pseries code tries to do
this. Disable this behavior by default, and document the switch a bit.
Signed-off-by: Nathan Lynch <[email protected]>
Signed-off-by: Michael Ellerman <[email protected]>
Thanks in advance for your support.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1833716
Title:
System crashes on hot adding a core with drmgr command
(4.15.0-48-generic)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-power-systems/+bug/1833716/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
