On Thursday, March 12, 2020 6:09 PM Dan Williams <[email protected]> wrote: > > If you force loaded a module with unresolved symbols all bets are off, > lets get "make TESTS=libndctl check" running cleanly before trying to > debug this report.
My experiments with "make TESTS=libndctl check" I have described in the separate thread "nfit_test: issue #2": https://lists.01.org/hyperkitty/list/[email protected]/thread/QDFPRM5C3UQJKPQ6OKA6PO57Z55S67RF/ I have *NOT* force loaded a module with unresolved symbols. I have loaded the 'nfit_test' module manually (using 'insmod') without any errors. I have repeated those steps below for the latest git://git.kernel.org/pub/scm/linux/kernel/git/nvdimm/nvdimm.git kernel from the ' libnvdimm-for-next' branch and the oops occurred again: [ 1559.560169] BUG: kernel NULL pointer dereference, address: 0000000000000018 [ 1559.560177] #PF: supervisor read access in kernel mode [ 1559.560180] #PF: error_code(0x0000) - not-present page [ 1559.560182] PGD 80000003886b7067 P4D 80000003886b7067 PUD 3f53af067 PMD 0 [ 1559.560190] Oops: 0000 [#1] PREEMPT SMP PTI [ 1559.560196] CPU: 0 PID: 79767 Comm: pmempool Tainted: G O 5.6.0-rc1-13504-g7b27a8622f80 #1 [ 1559.560199] Hardware name: System manufacturer System Product Name/RAMPAGE IV EXTREME, BIOS 4701 11/18/2013 [ 1559.560207] RIP: 0010:dev_dax_huge_fault+0x2b3/0x570 [device_dax] The steps to reproduce: $ sudo modprobe -v nfit_test insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/kernel/drivers/char/hw_random/rng-core.ko.xz insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/kernel/drivers/char/tpm/tpm.ko.xz insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/kernel/security/keys/trusted-keys/trusted.ko.xz insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/kernel/security/keys/encrypted-keys/encrypted-keys.ko.xz install /usr/bin/ndctl load-keys ; /sbin/modprobe --ignore-install libnvdimm $CMDLINE_OPTS No TPM handle discovered. failed to open file /etc/ndctl/keys/nvdimm-master.blob: No such file or directory insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/kernel/drivers/nvdimm/libnvdimm.ko.xz insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/kernel/drivers/acpi/nfit/nfit.ko.xz insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/extra/test/nfit_test_iomap.ko.xz insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/extra/test/nfit_test.ko.xz modprobe: ERROR: could not insert 'nfit_test': Unknown symbol in module, or unknown parameter (see dmesg) $ dmesg | tail [ 102.769871] Key type encrypted registered [ 102.799289] nfit_test_iomap: loading out-of-tree module taints kernel. [ 102.804008] nfit_test: Unknown symbol libnvdimm_test (err -2) [ 102.804054] nfit_test: Unknown symbol acpi_nfit_test (err -2) [ 102.804118] nfit_test: Unknown symbol pmem_test (err -2) [ 102.804164] nfit_test: Unknown symbol dax_pmem_core_test (err -2) [ 102.804226] nfit_test: Unknown symbol dax_pmem_compat_test (err -2) [ 102.804273] nfit_test: Unknown symbol device_dax_test (err -2) [ 102.804308] nfit_test: Unknown symbol dax_pmem_test (err -2) Removing the wrong modules: $ sudo rmmod nfit $ sudo rmmod libnvdimm Inserting the right modules manually: $ sudo insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/extra/libnvdimm.ko.xz $ sudo insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/extra/nd_btt.ko.xz $ sudo insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/extra/nd_pmem.ko.xz $ sudo insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/extra/nfit.ko.xz $ sudo insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/extra/dax_pmem_core.ko.xz $ sudo insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/extra/dax_pmem.ko.xz $ sudo insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/extra/device_dax.ko.xz $ sudo insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/extra/dax_pmem_compat.ko.xz $ sudo insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/extra/test/nfit_test.ko.xz The 'nfit_test' module is successfully inserted with *NO ERRORS* now: $ dmesg | tail [ 464.439504] nfit_test: mcsafe_test: disabled, skip. [ 464.500439] nfit_test nfit_test.0: failed to evaluate _FIT [ 464.507964] nfit_test nfit_test.1: Error found in NVDIMM nmem4 flags: save_fail restore_fail flush_fail not_armed [ 464.507990] nfit_test nfit_test.1: Error found in NVDIMM nmem5 flags: map_fail [ 464.508614] nd_pmem namespace6.0: region6 read-only, marking pmem6 read-only [ 464.508729] pmem6: detected capacity change from 0 to 33554432 [ 464.508737] pmem7: detected capacity change from 0 to 4194304 $ lsmod | grep nfit nfit_test 49152 8 dax_pmem_compat 20480 1 nfit_test device_dax 20480 2 nfit_test,dax_pmem_compat dax_pmem 20480 1 nfit_test dax_pmem_core 20480 3 dax_pmem,nfit_test,dax_pmem_compat nfit 73728 1 nfit_test nd_pmem 24576 1 nfit_test libnvdimm 200704 8 dax_pmem,nfit_test,dax_pmem_core,nd_btt,nd_pmem,dax_pmem_compat,nd_blk,nfit nfit_test_iomap 24576 6 nfit_test,dax_pmem_core,device_dax,nd_pmem,libnvdimm,nfit Trying to remove and reinsert the 'nfit_test' module: $ sudo ndctl disable-region all disabled 8 regions $ sudo modprobe -v -r nfit_test rmmod nfit_test rmmod nfit $ sudo modprobe -v nfit_test insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/kernel/drivers/acpi/nfit/nfit.ko.xz insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/extra/test/nfit_test.ko.xz modprobe: ERROR: could not insert 'nfit_test': Unknown symbol in module, or unknown parameter (see dmesg) $ dmesg | tail [ 919.861636] nfit_test: Unknown symbol acpi_nfit_test (err -2) Removing the wrong module: $ sudo rmmod nfit Reinserting manually: $ sudo insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/extra/nfit.ko.xz $ sudo insmod /lib/modules/5.6.0-rc1-13504-g7b27a8622f80/extra/test/nfit_test.ko.xz The 'nfit_test' module is successfully inserted with *NO ERRORS* again. $ lsmod | grep nfit nfit_test 49152 8 nfit 73728 1 nfit_test dax_pmem_compat 20480 1 nfit_test device_dax 20480 2 nfit_test,dax_pmem_compat dax_pmem 20480 1 nfit_test dax_pmem_core 20480 3 dax_pmem,nfit_test,dax_pmem_compat nd_pmem 24576 1 nfit_test libnvdimm 200704 8 dax_pmem,nfit_test,dax_pmem_core,nd_btt,nd_pmem,dax_pmem_compat,nd_blk,nfit nfit_test_iomap 24576 6 nfit_test,dax_pmem_core,device_dax,nd_pmem,libnvdimm,nfit Run PMDK pmempool_sync tests: $ cd ~/pmdk/src/test [~/pmdk/src/test]$ ./RUNTESTS pmempool_sync pmempool_sync/TEST27: SETUP (check/pmem/debug) pmempool_sync/TEST27: PASS [15.603 s] pmempool_sync/TEST27: SETUP (check/pmem/nondebug) pmempool_sync/TEST27: PASS [14.988 s] pmempool_sync/TEST28: SETUP (check/pmem/debug) ../unittest/unittest.sh: line 866: 79767 Killed ../../tools/pmempool/pmempool rm /tmp//test_pmempool_sync28/testset1 pmempool_sync/TEST28 crashed (signal 9). $ dmesg | tail [ 1543.234929] EXT4-fs (pmem0): mounted filesystem with ordered data mode. Opts: (null) [ 1543.234943] ext4 filesystem being mounted at /tmp/test_pmempool_sync27/mnt-pmem supports timestamps until 2038 (0x7fffffff) [ 1558.910001] nfit_test: mcsafe_test: disabled, skip. [ 1558.966718] nfit_test nfit_test.0: failed to evaluate _FIT [ 1558.974772] nfit_test nfit_test.1: Error found in NVDIMM nmem4 flags: save_fail restore_fail flush_fail not_armed [ 1558.974783] nfit_test nfit_test.1: Error found in NVDIMM nmem5 flags: map_fail [ 1558.975247] nd_pmem namespace6.0: region6 read-only, marking pmem6 read-only [ 1558.975376] pmem7: detected capacity change from 0 to 4194304 [ 1558.975380] pmem6: detected capacity change from 0 to 33554432 [ 1559.147912] pmem7: detected capacity change from 0 to 4194304 [ 1559.152775] nd_pmem namespace6.0: region6 read-only, marking pmem6 read-only [ 1559.152852] pmem6: detected capacity change from 0 to 33554432 [ 1559.560169] BUG: kernel NULL pointer dereference, address: 0000000000000018 [ 1559.560177] #PF: supervisor read access in kernel mode [ 1559.560180] #PF: error_code(0x0000) - not-present page [ 1559.560182] PGD 80000003886b7067 P4D 80000003886b7067 PUD 3f53af067 PMD 0 [ 1559.560190] Oops: 0000 [#1] PREEMPT SMP PTI [ 1559.560196] CPU: 0 PID: 79767 Comm: pmempool Tainted: G O 5.6.0-rc1-13504-g7b27a8622f80 #1 [ 1559.560199] Hardware name: System manufacturer System Product Name/RAMPAGE IV EXTREME, BIOS 4701 11/18/2013 [ 1559.560207] RIP: 0010:dev_dax_huge_fault+0x2b3/0x570 [device_dax] [ 1559.560211] Code: 37 48 c1 ee 0c 48 01 f0 48 ba ff ff ff ff ff ff 0f 00 49 c1 ef 0c 48 21 d3 49 01 c7 48 c1 e3 06 48 03 1d 58 bf 86 ea 48 89 da <48> 83 7a 18 00 75 10 49 8b 8c 24 f0 00 00 00 48 89 42 20 48 89 4a [ 1559.560215] RSP: 0000:ffffad85816f7db0 EFLAGS: 00010247 [ 1559.560218] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 1559.560220] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff964e4c0eb000 [ 1559.560223] RBP: ffffad85816f7e28 R08: 0005f6e000000000 R09: 0000000000251901 [ 1559.560226] R10: ffff964effffc000 R11: 0000000000033160 R12: ffff964eea06a100 [ 1559.560228] R13: 0000000000000001 R14: 0000000000000100 R15: 0000000000000001 [ 1559.560232] FS: 00007f6957dfe600(0000) GS:ffff964eefa00000(0000) knlGS:0000000000000000 [ 1559.560235] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1559.560237] CR2: 0000000000000018 CR3: 000000039118e004 CR4: 00000000000606f0 [ 1559.560240] Call Trace: [ 1559.560257] __do_fault+0x38/0x120 [ 1559.560263] __handle_mm_fault+0xfe9/0x1570 [ 1559.560269] ? security_mmap_file+0x7a/0xe0 [ 1559.560277] handle_mm_fault+0xce/0x200 [ 1559.560284] do_user_addr_fault+0x1ef/0x470 [ 1559.560293] page_fault+0x34/0x40 [ 1559.560298] RIP: 0033:0x7f69582d1f5c [ 1559.560302] Code: c3 48 81 fa 00 08 00 00 77 a1 48 83 fa 40 77 16 f3 0f 7f 07 f3 0f 7f 47 10 f3 0f 7f 44 17 f0 f3 0f 7f 44 17 e0 c3 48 8d 4f 40 <f3> 0f 7f 07 48 83 e1 c0 f3 0f 7f 44 17 f0 f3 0f 7f 47 10 f3 0f 7f [ 1559.560304] RSP: 002b:00007fffb0512278 EFLAGS: 00010206 [ 1559.560307] RAX: 00007f6955c00000 RBX: 0000000000000000 RCX: 00007f6955c00040 [ 1559.560310] RDX: 0000000000200000 RSI: 0000000000000000 RDI: 00007f6955c00000 [ 1559.560312] RBP: 0000000001e00000 R08: 000000000000000a R09: 0000000000000000 [ 1559.560315] R10: 0000000000000001 R11: 0000000000000206 R12: 000000000000000a [ 1559.560317] R13: 0000000000000000 R14: 00007f6955c00000 R15: 0000000000000000 [ 1559.560324] Modules linked in: kmem nfit_test(O) nfit(O) nd_blk dax_pmem_compat(O) device_dax(O) dax_pmem(O) dax_pmem_core(O) nd_pmem(O) nd_btt(O) libnvdimm(O) nfit_test_iomap(O) encrypted_keys trusted tpm rng_core uinput rfcomm xt_CHECKSUM xt_MASQUERADE nf_nat_tftp nf_conntrack_tftp tun bridge stp llc nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT xt_tcpudp ip6t_REJECT nf_reject_ipv6 ip6t_rpfilter ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep sunrpc nls_iso8859_1 nls_cp437 vfat fat dm_mirror dm_region_hash dm_log intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi ledtrig_audio btus b [ 1559.560377] snd_hda_intel btrtl crct10dif_pclmul iTCO_wdt btbcm snd_intel_dspcfg iTCO_vendor_support btintel crc32_pclmul ghash_clmulni_intel snd_hda_codec bluetooth aesni_intel snd_hda_core crypto_simd cryptd glue_helper snd_hwdep intel_cstate snd_seq snd_seq_device mousedev snd_pcm input_leds eeepc_wmi intel_uncore asus_wmi joydev battery ecdh_generic intel_rapl_perf sparse_keymap wmi_bmof pcspkr i2c_i801 lpc_ich ecc snd_timer mei_me snd mei soundcore evdev mac_hid ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 rfkill hid_generic usbhid hid sr_mod cdrom sd_mod nouveau i2c_algo_bit drm_kms_helper mxm_wmi syscopyarea sysfillrect sysimgblt ahci fb_sys_fops cec libahci ttm libata crc32c_intel drm xhci_pci scsi_mod e1000e xhci_hcd ehci_pci ehci_hcd agpgart wmi dm_mod fuse [last unloaded: nfit] [ 1559.560435] CR2: 0000000000000018 [ 1559.560439] ---[ end trace b60c468cd995d984 ]--- [ 1559.560446] RIP: 0010:dev_dax_huge_fault+0x2b3/0x570 [device_dax] [ 1559.560450] Code: 37 48 c1 ee 0c 48 01 f0 48 ba ff ff ff ff ff ff 0f 00 49 c1 ef 0c 48 21 d3 49 01 c7 48 c1 e3 06 48 03 1d 58 bf 86 ea 48 89 da <48> 83 7a 18 00 75 10 49 8b 8c 24 f0 00 00 00 48 89 42 20 48 89 4a [ 1559.560453] RSP: 0000:ffffad85816f7db0 EFLAGS: 00010247 [ 1559.560456] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [ 1559.560458] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff964e4c0eb000 [ 1559.560461] RBP: ffffad85816f7e28 R08: 0005f6e000000000 R09: 0000000000251901 [ 1559.560463] R10: ffff964effffc000 R11: 0000000000033160 R12: ffff964eea06a100 [ 1559.560466] R13: 0000000000000001 R14: 0000000000000100 R15: 0000000000000001 [ 1559.560469] FS: 00007f6957dfe600(0000) GS:ffff964eefa00000(0000) knlGS:0000000000000000 [ 1559.560472] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1559.560475] CR2: 0000000000000018 CR3: 000000039118e004 CR4: 00000000000606f0 $ $ ps aux | grep -e 'D+' | grep -v grep root 80616 0.0 0.0 7240 4020 pts/2 D+ 09:44 0:00 ndctl disable-region all root 81004 0.0 0.0 7240 2112 pts/2 D+ 09:47 0:00 ndctl disable-region all ldorau 81291 0.0 0.0 34572 5676 pts/2 D+ 09:50 0:00 ../../tools/pmempool/pmempool create obj --layout pmempool /tmp//test_pmempool_sync32/testset1 $ sudo cat /proc/80616/stack [<0>] __synchronize_srcu.part.0+0x78/0xa0 [<0>] kill_dax+0x22/0x70 [<0>] pmem_release_disk+0x12/0x40 [nd_pmem] [<0>] release_nodes+0x19c/0x1e0 [<0>] device_release_driver_internal+0xf4/0x1c0 [<0>] unbind_store+0xef/0x120 [<0>] kernfs_fop_write+0xce/0x1b0 [<0>] vfs_write+0xb6/0x1a0 [<0>] ksys_write+0x5f/0xe0 [<0>] do_syscall_64+0x4e/0x150 [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9 $ sudo cat /proc/81004/stack [<0>] flush_namespaces+0x15/0x30 [libnvdimm] [<0>] device_for_each_child+0x58/0x90 [<0>] flush_regions_dimms+0x33/0x40 [libnvdimm] [<0>] device_for_each_child+0x58/0x90 [<0>] wait_probe_show+0x3d/0x60 [libnvdimm] [<0>] dev_attr_show+0x19/0x40 [<0>] sysfs_kf_seq_show+0x9b/0xf0 [<0>] seq_read+0xcd/0x440 [<0>] vfs_read+0x9d/0x150 [<0>] ksys_read+0x5f/0xe0 [<0>] do_syscall_64+0x4e/0x150 [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9 $ sudo cat /proc/81291/stack [<0>] flush_namespaces+0x15/0x30 [libnvdimm] [<0>] device_for_each_child+0x58/0x90 [<0>] flush_regions_dimms+0x33/0x40 [libnvdimm] [<0>] device_for_each_child+0x58/0x90 [<0>] wait_probe_show+0x3d/0x60 [libnvdimm] [<0>] dev_attr_show+0x19/0x40 [<0>] sysfs_kf_seq_show+0x9b/0xf0 [<0>] seq_read+0xcd/0x440 [<0>] vfs_read+0x9d/0x150 [<0>] ksys_read+0x5f/0xe0 [<0>] do_syscall_64+0x4e/0x150 [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9 -- Lukasz _______________________________________________ Linux-nvdimm mailing list -- [email protected] To unsubscribe send an email to [email protected]
