ubuntu_ftrace_smoke_test on all v5.4 kernels on all Azure instances
hangs:
----
114 11:22:52 INFO | Writing results to
/home/azure/autotest/client/results/default
115 11:22:52 DEBUG| Initializing the state engine
116 11:22:52 DEBUG| Persistent state client.steps now set to []
117 11:22:52 DEBUG| Persistent option harness now set to None
118 11:22:52 DEBUG| Persistent option harness_args now set to None
119 11:22:52 DEBUG| Selected harness: standalone
120 11:22:52 INFO | START ---- ---- timestamp=1631791372 localtime=Sep 16
11:22:52
121 11:22:52 DEBUG| Persistent state client._record_indent now set to 1
122 11:22:52 DEBUG| Test has timeout: 900 sec.
123 11:22:52 INFO | START ubuntu_ftrace_smoke_test.ftrace-smoke-test
ubuntu_ftrace_smoke_test.ftrace-smoke-test timestamp=1631791372 timeout=900
localtime=Sep 16 11:22:52
124 11:22:52 DEBUG| Persistent state client._record_indent now set to 2
125 11:22:52 DEBUG| Persistent state client.unexpected_reboot now set to
('ubuntu_ftrace_smoke_test.ftrace-smoke-test',
'ubuntu_ftrace_smoke_test.ftrace-smoke-test')
126 11:22:52 DEBUG| Waiting for pid 2953 for 900 seconds
127 11:22:52 DEBUG| Running
'/home/azure/autotest/client/tests/ubuntu_ftrace_smoke_test/ubuntu_ftrace_smoke_test.sh'
128 11:22:52 DEBUG| [stdout] PASSED (CONFIG_FUNCTION_TRACER=y in
/boot/config-5.4.0-1059-azure)
129 11:22:52 DEBUG| [stdout] PASSED (CONFIG_FUNCTION_GRAPH_TRACER=y in
/boot/config-5.4.0-1059-azure)
130 11:22:52 DEBUG| [stdout] PASSED (CONFIG_STACK_TRACER=y in
/boot/config-5.4.0-1059-azure)
131 11:22:52 DEBUG| [stdout] PASSED (CONFIG_DYNAMIC_FTRACE=y in
/boot/config-5.4.0-1059-azure)
132 11:22:54 DEBUG| [stdout] PASSED all expected /sys/kernel/debug/tracing
files exist
133 11:22:54 DEBUG| [stdout] PASSED (function_graph in
/sys/kernel/debug/tracing/available_tracers)
134 11:22:54 DEBUG| [stdout] PASSED (function in
/sys/kernel/debug/tracing/available_tracers)
135 11:22:54 DEBUG| [stdout] PASSED (nop in
/sys/kernel/debug/tracing/available_tracers)
136 11:22:57 DEBUG| [stdout] PASSED (tracer function can be enabled)
----
Found on Azure clouds:
* focal/linux/5.4.0-85.95
* focal/linux-azure/5.4.0-1059.62
* bionic/linux-azure-5.4/5.4.0-1059.62~18.04.1
* SRU cycles: 2021.08.16, 2021.07.19, 2021.05.31 (not checked earlier)
Not found:
* Other clouds.
* Azure: hirsute/linux/5.11.0-35.37
* Azure: focal/linux-azure-5.11/5.11.0-1016.17~20.04.1
* Azure: focal/linux-azure-5.8/5.8.0-1042.45~20.04.1
* Azure: bionic/linux-azure-4.15/4.15.0-1124.137
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-azure in Ubuntu.
https://bugs.launchpad.net/bugs/1882669
Title:
Focal / Groovy aws and azure instances crash with ftracetest in
ubuntu_kernel_selftests and ubuntu_ftrace_smoke_test
Status in ubuntu-kernel-tests:
New
Status in linux-aws package in Ubuntu:
Confirmed
Status in linux-azure package in Ubuntu:
Confirmed
Status in linux-azure source package in Focal:
Confirmed
Status in linux-azure source package in Groovy:
Fix Released
Bug description:
Issue found on 5.4.0-1013.13
The ftrace in ubuntu_kernel_selftests failed on the second test:
# === Ftrace unit tests ===
# [1] Basic trace file check [PASS]
# [2] Basic test for tracers
(System hang here)
Syslog:
kernel: [ 2234.408225] in mmio_trace_init
kernel: [ 2234.988256] mmiotrace: Disabling non-boot CPUs...
kernel: [ 2235.028451] mmiotrace: Error taking CPU1 down: -16
kernel: [ 2235.052564] mmiotrace: Error taking CPU2 down: -16
kernel: [ 2235.076593] mmiotrace: Error taking CPU3 down: -16
kernel: [ 2235.096576] mmiotrace: Error taking CPU4 down: -16
kernel: [ 2235.116487] mmiotrace: Error taking CPU5 down: -16
kernel: [ 2235.132446] mmiotrace: Error taking CPU6 down: -16
kernel: [ 2235.152488] mmiotrace: Error taking CPU7 down: -16
kernel: [ 2235.172526] mmiotrace: Error taking CPU8 down: -16
kernel: [ 2235.192486] mmiotrace: Error taking CPU9 down: -16
kernel: [ 2235.212455] mmiotrace: Error taking CPU10 down: -16
kernel: [ 2235.233020] mmiotrace: Error taking CPU11 down: -16
kernel: [ 2235.256355] mmiotrace: Error taking CPU12 down: -16
kernel: [ 2235.272473] mmiotrace: Error taking CPU13 down: -16
kernel: [ 2235.292393] mmiotrace: Error taking CPU14 down: -16
kernel: [ 2235.312426] mmiotrace: Error taking CPU15 down: -16
kernel: [ 2235.344407] mmiotrace: Error taking CPU16 down: -16
kernel: [ 2235.380494] mmiotrace: Error taking CPU17 down: -16
kernel: [ 2235.400319] mmiotrace: Error taking CPU18 down: -16
kernel: [ 2235.432631] mmiotrace: Error taking CPU19 down: -16
kernel: [ 2235.456453] mmiotrace: Error taking CPU20 down: -16
kernel: [ 2235.480432] mmiotrace: Error taking CPU21 down: -16
kernel: [ 2235.496501] mmiotrace: Error taking CPU22 down: -16
kernel: [ 2235.516442] mmiotrace: Error taking CPU23 down: -16
kernel: [ 2235.536394] mmiotrace: Error taking CPU24 down: -16
kernel: [ 2235.556489] mmiotrace: Error taking CPU25 down: -16
kernel: [ 2235.573759] smpboot: CPU 26 is now offline
kernel: [ 2235.575071] mmiotrace: CPU26 is down.
kernel: [ 2235.585643] smpboot: CPU 27 is now offline
kernel: [ 2235.586699] mmiotrace: CPU27 is down.
kernel: [ 2235.609679] smpboot: CPU 28 is now offline
kernel: [ 2235.610788] mmiotrace: CPU28 is down.
kernel: [ 2235.633530] smpboot: CPU 29 is now offline
kernel: [ 2235.634511] mmiotrace: CPU29 is down.
kernel: [ 2235.653652] smpboot: CPU 30 is now offline
kernel: [ 2235.655415] mmiotrace: CPU30 is down.
kernel: [ 2235.678126] smpboot: CPU 31 is now offline
kernel: [ 2235.679061] mmiotrace: CPU31 is down.
kernel: [ 2235.679062] mmiotrace: multiple CPUs still online, may miss events.
kernel: [ 2235.679063] mmiotrace: enabled.
kernel: [ 2235.679128] in mmio_trace_reset
kernel: [ 2235.696163] mmiotrace: Re-enabling CPUs...
kernel: [ 2235.732157] mmiotrace: enabled CPU1.
kernel: [ 2235.772207] mmiotrace: enabled CPU2.
kernel: [ 2235.812216] mmiotrace: enabled CPU3.
kernel: [ 2235.844220] mmiotrace: enabled CPU4.
kernel: [ 2235.876211] mmiotrace: enabled CPU5.
kernel: [ 2235.916159] mmiotrace: enabled CPU6.
kernel: [ 2235.956214] mmiotrace: enabled CPU7.
kernel: [ 2235.988145] mmiotrace: enabled CPU8.
kernel: [ 2236.020214] mmiotrace: enabled CPU9.
kernel: [ 2236.060207] mmiotrace: enabled CPU10.
kernel: [ 2236.096208] mmiotrace: enabled CPU11.
kernel: [ 2236.128205] mmiotrace: enabled CPU12.
/usr/sbin/irqbalance: WARNING, didn't collect load info for all cpus,
balancing is broken
kernel: [ 2236.168204] mmiotrace: enabled CPU13.
kernel: [ 2236.200206] mmiotrace: enabled CPU14.
kernel: [ 2236.232207] mmiotrace: enabled CPU15.
kernel: [ 2236.264224] mmiotrace: enabled CPU16.
kernel: [ 2236.300207] mmiotrace: enabled CPU17.
kernel: [ 2236.332210] mmiotrace: enabled CPU18.
kernel: [ 2236.372224] mmiotrace: enabled CPU19.
kernel: [ 2236.404147] mmiotrace: enabled CPU20.
kernel: [ 2236.452153] mmiotrace: enabled CPU21.
kernel: [ 2236.492152] mmiotrace: enabled CPU22.
kernel: [ 2236.524153] mmiotrace: enabled CPU23.
kernel: [ 2236.564211] mmiotrace: enabled CPU24.
kernel: [ 2236.612204] mmiotrace: enabled CPU25.
kernel: [ 2236.644353] smpboot: Booting Node 0 Processor 26 APIC 0x1a
kernel: [ 2236.645769] mmiotrace: enabled CPU26.
kernel: [ 2236.684537] smpboot: Booting Node 0 Processor 27 APIC 0x1b
kernel: [ 2236.685751] mmiotrace: enabled CPU27.
kernel: [ 2236.724284] smpboot: Booting Node 0 Processor 28 APIC 0x1c
kernel: [ 2236.725508] mmiotrace: enabled CPU28.
kernel: [ 2236.772271] smpboot: Booting Node 0 Processor 29 APIC 0x1d
kernel: [ 2236.773532] mmiotrace: enabled CPU29.
kernel: [ 2236.816285] smpboot: Booting Node 0 Processor 30 APIC 0x1e
kernel: [ 2236.817403] mmiotrace: enabled CPU30.
kernel: [ 2236.856265] smpboot: Booting Node 0 Processor 31 APIC 0x1f
kernel: [ 2236.857716] mmiotrace: enabled CPU31.
kernel: [ 2236.868235] mmiotrace: disabled.
(System hang here)
For the ftrace smoke test it will hang here:
PASSED (CONFIG_FUNCTION_TRACER=y in /boot/config-5.4.0-1012-azure)
PASSED (CONFIG_FUNCTION_GRAPH_TRACER=y in /boot/config-5.4.0-1012-azure)
PASSED (CONFIG_STACK_TRACER=y in /boot/config-5.4.0-1012-azure)
PASSED (CONFIG_DYNAMIC_FTRACE=y in /boot/config-5.4.0-1012-azure)
PASSED all expected /sys/kernel/debug/tracing files exist
PASSED (function_graph in /sys/kernel/debug/tracing/available_tracers)
PASSED (function in /sys/kernel/debug/tracing/available_tracers)
PASSED (nop in /sys/kernel/debug/tracing/available_tracers)
PASSED (tracer function can be enabled)
(System hang here)
According to the test result on Focal generic kernel, the next test should be:
(tracer function_graph can be enabled)
This is not a regression as it can be found on 5.4.0-1012.12 as well.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1882669/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : [email protected]
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
