On 08/02/15 17:54, sf...@users.sourceforge.net wrote: > OmegaPhil: >> I've gone through this procedure - everything is fine. This is as >> expected as originally I said this problem only arose when the aufs >> volume was mounted at boot via fstab - when you do it later on >> everything is fine. > > Then what will happen if you put 0 - 6 steps just before mounting aufs > from fstab in rc scripts? And what is the refcount just after processing > fstab? Is it incemented unexpectedly? > > >> Would you agree that me continuing the original kernel tracing at an >> earlier stage would be the best way to proceed? > > Assuming the refcount is unrelated to your original EBUSY problem, I am > joining with you and thinking how to identify when the refcount is > incremented. > Isn't it enough to put lsmod or "cat /sys/module/aufs/refcnt" in rc > scripts? > Ah, probably you did it already too. Then when and what command > incremented the refcount unexpectedly? > > Or if it is easy for you to build kernel, inserting printk() just after > every these calls may be helpful. But I don't know how it is better than > your trace-cmd. > > __this_cpu_write(mod->refptr->incs, 1); > __this_cpu_inc(module->refptr->incs); > __this_cpu_inc(module->refptr->decs); > > > J. R. Okajima
Going deeper into boot was where I stopped after doing the rc.local test, when I raised the question here for a better method. I have now managed to get a script to run prior to mountall.sh, which demonstrated nothing held a reference to aufs at that point. I then put the kernel tracing stuff into the script, but it completely failed to record anything obtaining a reference to aufs prior to the X login screen, which is completely useless... To see how things were handled, I forcefully removed the aufs module: ====================================================================== Feb 8 21:03:03 omega1 kernel: [ 972.887546] Disabling lock debugging due to kernel taint Feb 8 21:03:03 omega1 kernel: [ 972.928949] kmem_cache_destroy au_dinfo: Slab cache still has objects Feb 8 21:03:03 omega1 kernel: [ 972.928962] CPU: 1 PID: 22935 Comm: rmmod Tainted: G R C O 3.16.0-4-a md64 #1 Debian 3.16.7-ckt2-1 Feb 8 21:03:03 omega1 kernel: [ 972.928967] Hardware name: Gigabyte Technology Co., Ltd. GA-990FXA-UD5/GA-99 0FXA-UD5, BIOS F12 10/03/2013 Feb 8 21:03:03 omega1 kernel: [ 972.928971] ffffffffa0826ce0 ffffffff81507263 ffff88080b77c980 ffffffff8115 bacd Feb 8 21:03:03 omega1 kernel: [ 972.928980] 0000000000000000 ffffffffa081f1e9 0000000000000000 ffffffffa081 f2c1 Feb 8 21:03:03 omega1 kernel: [ 972.928987] ffffffff810d6a84 00000200811aa73a ffffffffa0826ce0 000000000000 0a00 Feb 8 21:03:03 omega1 kernel: [ 972.928993] Call Trace: Feb 8 21:03:03 omega1 kernel: [ 972.929010] [<ffffffff81507263>] ? dump_stack+0x41/0x51 Feb 8 21:03:03 omega1 kernel: [ 972.929017] [<ffffffff8115bacd>] ? kmem_cache_destroy+0x9d/0xc0 Feb 8 21:03:03 omega1 kernel: [ 972.929029] [<ffffffffa081f1e9>] ? au_cache_fin+0x1e/0x35 [aufs] Feb 8 21:03:03 omega1 kernel: [ 972.929039] [<ffffffffa081f2c1>] ? aufs_exit+0x11/0xd50 [aufs] Feb 8 21:03:03 omega1 kernel: [ 972.929045] [<ffffffff810d6a84>] ? SyS_delete_module+0x154/0x210 Feb 8 21:03:03 omega1 kernel: [ 972.929053] [<ffffffff8150f368>] ? page_fault+0x28/0x30 Feb 8 21:03:03 omega1 kernel: [ 972.929058] [<ffffffff8150d32d>] ? system_call_fast_compare_end+0x10/0x15 Feb 8 21:03:03 omega1 kernel: [ 972.929078] kmem_cache_destroy au_icntnr: Slab cache still has objects Feb 8 21:03:03 omega1 kernel: [ 972.929083] CPU: 1 PID: 22935 Comm: rmmod Tainted: G R C O 3.16.0-4-a md64 #1 Debian 3.16.7-ckt2-1 Feb 8 21:03:03 omega1 kernel: [ 972.929086] Hardware name: Gigabyte Technology Co., Ltd. GA-990FXA-UD5/GA-990FXA-UD5, BIOS F12 10/03/2013 Feb 8 21:03:03 omega1 kernel: [ 972.929089] ffffffffa0826ce0 ffffffff81507263 ffff88080b77ca80 ffffffff8115bacd Feb 8 21:03:03 omega1 kernel: [ 972.929094] 0000000000000008 ffffffffa081f1e9 0000000000000000 ffffffffa081f2c1 Feb 8 21:03:03 omega1 kernel: [ 972.929100] ffffffff810d6a84 00000200811aa73a ffffffffa0826ce0 0000000000000a00 Feb 8 21:03:03 omega1 kernel: [ 972.929105] Call Trace: Feb 8 21:03:03 omega1 kernel: [ 972.929111] [<ffffffff81507263>] ? dump_stack+0x41/0x51 Feb 8 21:03:03 omega1 kernel: [ 972.929117] [<ffffffff8115bacd>] ? kmem_cache_destroy+0x9d/0xc0 Feb 8 21:03:03 omega1 kernel: [ 972.929126] [<ffffffffa081f1e9>] ? au_cache_fin+0x1e/0x35 [aufs] Feb 8 21:03:03 omega1 kernel: [ 972.929134] [<ffffffffa081f2c1>] ? aufs_exit+0x11/0xd50 [aufs] Feb 8 21:03:03 omega1 kernel: [ 972.929140] [<ffffffff810d6a84>] ? SyS_delete_module+0x154/0x210 Feb 8 21:03:03 omega1 kernel: [ 972.929147] [<ffffffff8150f368>] ? page_fault+0x28/0x30 Feb 8 21:03:03 omega1 kernel: [ 972.929152] [<ffffffff8150d32d>] ? system_call_fast_compare_end+0x10/0x15 Feb 8 21:03:03 omega1 kernel: [ 972.929190] kmem_cache_destroy au_vdir: Slab cache still has objects Feb 8 21:03:03 omega1 kernel: [ 972.929195] CPU: 1 PID: 22935 Comm: rmmod Tainted: G R C O 3.16.0-4-amd64 #1 Debian 3.16.7-ckt2-1 Feb 8 21:03:03 omega1 kernel: [ 972.929198] Hardware name: Gigabyte Technology Co., Ltd. GA-990FXA-UD5/GA-990FXA-UD5, BIOS F12 10/03/2013 Feb 8 21:03:03 omega1 kernel: [ 972.929200] ffffffffa0826ce0 ffffffff81507263 ffff88080a948b80 ffffffff8115bacd Feb 8 21:03:03 omega1 kernel: [ 972.929206] 0000000000000018 ffffffffa081f1e9 0000000000000000 ffffffffa081f2c1 Feb 8 21:03:03 omega1 kernel: [ 972.929211] ffffffff810d6a84 00000200811aa73a ffffffffa0826ce0 0000000000000a00 Feb 8 21:03:03 omega1 kernel: [ 972.929216] Call Trace: Feb 8 21:03:03 omega1 kernel: [ 972.929222] [<ffffffff81507263>] ? dump_stack+0x41/0x51 Feb 8 21:03:03 omega1 kernel: [ 972.929228] [<ffffffff8115bacd>] ? kmem_cache_destroy+0x9d/0xc0 Feb 8 21:03:03 omega1 kernel: [ 972.929237] [<ffffffffa081f1e9>] ? au_cache_fin+0x1e/0x35 [aufs] Feb 8 21:03:03 omega1 kernel: [ 972.929245] [<ffffffffa081f2c1>] ? aufs_exit+0x11/0xd50 [aufs] Feb 8 21:03:03 omega1 kernel: [ 972.929251] [<ffffffff810d6a84>] ? SyS_delete_module+0x154/0x210 Feb 8 21:03:03 omega1 kernel: [ 972.929257] [<ffffffff8150f368>] ? page_fault+0x28/0x30 Feb 8 21:03:03 omega1 kernel: [ 972.929262] [<ffffffff8150d32d>] ? system_call_fast_compare_end+0x10/0x15 Feb 8 21:03:03 omega1 kernel: [ 972.929338] ------------[ cut here ]------------ Feb 8 21:03:03 omega1 kernel: [ 972.929345] WARNING: CPU: 1 PID: 22935 at /build/linux-CMiYW9/linux-3.16.7-ckt2/fs/aufs/dynop.c:378 SyS_delete_module+0x154/0x210() Feb 8 21:03:03 omega1 kernel: [ 972.929347] Modules linked in: xt_DSCP xt_owner xt_tcpudp iptable_mangle iptable_filter ip_tables x_tables cfg80211 rfkill cpufreq_conservative cpufreq_stats cpufreq_userspace cpufreq_powersave pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) binfmt_misc nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc aufs(C-) fuse it87 hwmon_vid loop algif_skcipher af_alg dm_crypt snd_hda_codec_hdmi joydev mxm_wmi hid_pl ff_memless snd_usb_audio snd_usbmidi_lib snd_rawmidi snd_seq_device kvm_amd kvm evdev crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd serio_raw pcspkr edac_mce_amd edac_core snd_hda_codec_realtek fam15h_power k10temp snd_hda_codec_generic sp5100_tco i2c_piix4 snd_hda_intel radeon snd_hda_controller snd_hda_codec snd_hwdep snd_pcm snd_timer ttm snd drm_kms_helper drm i2c_algo_bit soundcore tpm_infineon i2c_core tpm_tis tpm acpi_cpufreq shpchp processor wmi thermal_sys button ext4 crc16 mbcache jbd2 btrfs xor raid6_pq dm_mod md_mod sr_mod cdrom hid_generic usbhid hid usb_storage sg sd_mod crc_t10dif crct10dif_generic ohci_pci crct10dif_pclmul crct10dif_common ahci crc32c_intel libahci firewire_ohci ohci_hcd mpt2sas ehci_pci raid_class xhci_hcd libata ehci_hcd scsi_transport_sas firewire_core r8169 crc_itu_t mii usbcore scsi_mod usb_common Feb 8 21:03:03 omega1 kernel: [ 972.929483] CPU: 1 PID: 22935 Comm: rmmod Tainted: G R C O 3.16.0-4-amd64 #1 Debian 3.16.7-ckt2-1 Feb 8 21:03:03 omega1 kernel: [ 972.929486] Hardware name: Gigabyte Technology Co., Ltd. GA-990FXA-UD5/GA-990FXA-UD5, BIOS F12 10/03/2013 Feb 8 21:03:03 omega1 kernel: [ 972.929489] 0000000000000009 ffffffff81507263 0000000000000000 ffffffff81065847 Feb 8 21:03:03 omega1 kernel: [ 972.929494] 0000000000000000 ffffffffa0826ce0 0000000000000a00 00007fff52abd90e Feb 8 21:03:03 omega1 kernel: [ 972.929499] 00007fff52abc5d8 ffffffff810d6a84 00000200811aa73a ffffffffa0826ce0 Feb 8 21:03:03 omega1 kernel: [ 972.929504] Call Trace: Feb 8 21:03:03 omega1 kernel: [ 972.929509] [<ffffffff81507263>] ? dump_stack+0x41/0x51 Feb 8 21:03:03 omega1 kernel: [ 972.929516] [<ffffffff81065847>] ? warn_slowpath_common+0x77/0x90 Feb 8 21:03:03 omega1 kernel: [ 972.929523] [<ffffffff810d6a84>] ? SyS_delete_module+0x154/0x210 Feb 8 21:03:03 omega1 kernel: [ 972.929530] [<ffffffff8150f368>] ? page_fault+0x28/0x30 Feb 8 21:03:03 omega1 kernel: [ 972.929535] [<ffffffff8150d32d>] ? system_call_fast_compare_end+0x10/0x15 Feb 8 21:03:03 omega1 kernel: [ 972.929539] ---[ end trace cb82ca5d4f3daf0b ]--- ====================================================================== But aside from demonstrating something is definitely using it in kernel memory, I can't see anything else of use. As a bonus the system refused to shutdown normally at that point. When I next reach this (Tuesday) I will start to slowly fiddle with the init.d scripts to move my test script later and later to see when things start to fail. I can now make my own kernel etc but that is a last resort, and even though I might be forced to, I don't really want to change code outside of the aufs module.
signature.asc
Description: OpenPGP digital signature
------------------------------------------------------------------------------ Dive into the World of Parallel Programming. The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
