Hi, I was testing my aufs-related fixes to dockerd, and came upon this kernel oops: # dmesg .... [35180.792125] ------------[ cut here ]------------ [35180.792127] kernel BUG at /build/linux-3btXxq/linux-4.15.0/fs/aufs/dynop.c:207! [35180.795621] invalid opcode: 0000 [#1] SMP PTI [35180.797690] Modules linked in: veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc aufs overlay nls_iso8859_1 kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel input_leds joydev serio_raw sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear aesni_intel aes_x86_64 virtio_blk crypto_simd virtio_net cryptd glue_helper psmouse virtio_scsi floppy [35180.832904] CPU: 2 PID: 18881 Comm: dockerd Not tainted 4.15.0-50-generic #54-Ubuntu [35180.836106] Hardware name: DigitalOcean Droplet, BIOS 20171212 12/12/2017 [35180.838893] RIP: 0010:dy_bug+0x9/0x10 [aufs] [35180.840793] RSP: 0018:ffffb1a984017a20 EFLAGS: 00010246 [35180.842931] RAX: ffffffff9dc4f060 RBX: ffff9b5d8f890700 RCX: 0000000000000000 [35180.845755] RDX: 0000000000000000 RSI: ffffffff9dc4f060 RDI: ffff9b5d8f890720 [35180.849012] RBP: ffffb1a984017a20 R08: ffff9b5e9fd27120 R09: ffff9b5d9b814060 [35180.853563] R10: ffffffff9cff2100 R11: 0000000000000c40 R12: ffff9b5d9b814060 [35180.858243] R13: ffff9b5d9b814000 R14: ffff9b5cb40d7c58 R15: ffff9b5d9c295e00 [35180.865344] FS: A 00007f8e7bff7700(0000) GS:ffff9b5e9fd00000(0000) knlGS:0000000000000000 [35180.869508] CS: A 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35180.872336] CR2: 000000c4200f8160 CR3: 0000000183c94006 CR4: 00000000007606e0 [35180.875245] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35180.878249] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [35180.881038] PKRU: 55555554 [35180.882316] Call Trace: [35180.883546] A au_dy_iaop+0x1c3/0x320 [aufs] [35180.885077] A ? au_iinfo_init+0x6e/0xd0 [aufs] [35180.886772] A au_new_inode+0x4bb/0x730 [aufs] [35180.888616] A epilog+0xfd/0x190 [aufs] [35180.890320] A add_simple+0x2b1/0x420 [aufs] [35180.893184] A au_aopen_or_create+0x46/0x60 [aufs] [35180.895149] A aufs_atomic_open+0x1c8/0x3b0 [aufs] [35180.896990] A ? aufs_permission+0x190/0x2d0 [aufs] [35180.899162] A ? __inode_permission+0x5b/0x160 [35180.900976] A path_openat+0xaff/0x1770 [35180.902637] A ? path_openat+0xaff/0x1770 [35180.904326] A ? au_h_iptr+0x16/0x40 [aufs] [35180.906018] A do_filp_open+0x9b/0x110 [35180.907605] A ? __check_object_size+0xaf/0x1b0 [35180.909913] A ? __alloc_fd+0xb2/0x170 [35180.912472] A do_sys_open+0x1bb/0x2c0 [35180.915163] A ? do_sys_open+0x1bb/0x2c0 [35180.917923] A SyS_openat+0x14/0x20 [35180.920395] A do_syscall_64+0x73/0x130 [35180.925637] A entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [35180.928896] RIP: 0033:0x4b98ca [35180.930976] RSP: 002b:000000c422c76d10 EFLAGS: 00000202 ORIG_RAX: 0000000000000101 [35180.935378] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00000000004b98ca [35180.939158] RDX: 0000000000080040 RSI: 000000c4252efa40 RDI: ffffffffffffff9c [35180.942999] RBP: 000000c422c76d90 R08: 0000000000000000 R09: 0000000000000000 [35180.945841] R10: 00000000000001ed R11: 0000000000000202 R12: ffffffffffffffff [35180.948867] R13: 000000000000003d R14: 000000000000003c R15: 0000000000000049 [35180.951815] Code: 01 00 48 85 c0 74 07 48 89 87 c8 00 00 00 5d c3 66 90 0f 1f 44 00 00 55 48 89 e5 e8 a2 c5 93 dc 5d c3 0f 1f 44 00 00 55 48 89 e5 <0f> 0b 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 fa 48 63 47 f0 48 [35180.960552] RIP: dy_bug+0x9/0x10 [aufs] RSP: ffffb1a984017a20 [35180.962961] ---[ end trace 9c8a22b601f49ad0 ]--- This is on Ubuntu 18.04 with their latest stable kernel (based on 4.15), and here's the version of aufs kmod: # modinfo aufs filename: A A A /lib/modules/4.15.0-50-generic/kernel/fs/aufs/aufs.ko alias: A A A A A fs-aufs version: A A A A 4.15-20180219 description: A A aufs -- Advanced multi layered unification filesystem author: A A A A Junjiro R. Okajima <[1]aufs-users@lists.sourceforge.net> license: A A A A GPL srcversion: A A 9C7732CB810A119C03C59CE depends: A A A A retpoline: A A A Y intree: A A A A Y name: A A A A A aufs vermagic: A A A 4.15.0-50-generic SMP mod_unload signat: A A A A PKCS#7 signer: A A A A sig_key: A A A A sig_hashalgo: A md4 parm: A A A A A brs:use <sysfs>/fs/aufs/si_*/brN (int) parm: A A A A A allow_userns:allow unprivileged to mount under userns (bool) Kernel sources for this can be found by doing git clone git://[2]git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/bio nic git reset --hardA Ubuntu-4.15.0-50.54 I'm not sure I can recreate this without docker, but what the test does is pretty simple: it prepares and mounts an aufs (with a parent) when creates a few directories and empty files in it, then unmounts it and removes everything. All this is massively parallel, but on different aufs mounts. Note that I was running this test for days, and only got this oops once, so I'm pretty sure it is very hard to reproduce. Unfortunately I had to reboot the node since this happened. If I will reproduce it again, please let me know what other information might be helpful. Docker is written in Go, and from its stack dump I see a goroutine (thread) got stuck in open() syscall while trying to create a file: goroutine 363684 [syscall, 70 minutes]: syscall.Syscall6(0x101, 0xffffffffffffff9c, 0xc4252efa40, 0x80040, 0x1ed, 0x0, 0x0, 0x6e, 0xc4252ef9d0, 0x6e) A A A A /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5 syscall.openat(0xffffffffffffff9c, 0xc4252ef9d0, 0x6e, 0x80040, 0x1ed, 0xc4252ef960, 0x6f, 0x6f) A A A A /usr/local/go/src/syscall/zsyscall_linux_amd64.go:38 +0xae syscall.Open(0xc4252ef9d0, 0x6e, 0x80040, 0x1ed, 0xc4252ef8f0, 0x6f, 0xc4252ef9d0) A A A A /usr/local/go/src/syscall/syscall_linux.go:51 +0x50 os.openFileNolog(0xc4252ef9d0, 0x6e, 0x40, 0x1ed, 0x2, 0xc4252ef9d0, 0x6e) A A A A /usr/local/go/src/os/file_unix.go:168 +0x7a os.OpenFile(0xc4252ef9d0, 0x6e, 0x40, 0xc4000001ed, 0x6e, 0x0, 0x0) A A A A /usr/local/go/src/os/file.go:269 +0x5f [3]github.com/docker/docker/daemon/initlayer.Setup(0xc4230d3ce0, 0x5e, 0x0, 0x0, 0x12491ed, 0x1) A A A A /go/src/[4]github.com/docker/docker/daemon/initlayer/setup_unix.go:50 +0x94e [5]github.com/docker/docker/daemon.(*Daemon).setupInitLayer(0xc4203d600 0, 0xc4230d3ce0, 0x5e, 0x0, 0xc4200beb48) A A A A /go/src/[6]github.com/docker/docker/daemon/daemon.go:1033 +0x57 [7]github.com/docker/docker/daemon.(*Daemon).(github.com/docker/docker/ daemon.setupInitLayer)-fm(0xc4230d3ce0, 0x5e, 0x45, 0x0) A A A A /go/src/[8]github.com/docker/docker/daemon/daemon_unix.go:934 +0x3e [9]github.com/docker/docker/layer.(*layerStore).initMount(0xc4203b5310, 0xc421892980, 0x40, 0xc420582600, 0x40, 0x0, 0x0, 0xc4226b3bb0, 0x0, 0x0, ...) A A A A /go/src/[10]github.com/docker/docker/layer/layer_store.go:662 +0x1de [11]github.com/docker/docker/layer.(*layerStore).CreateRWLayer(0xc4203b 5310, 0xc4218928c0, 0x40, 0xc420040a00, 0x47, 0xc4233f2ec0, 0x0, 0x0, 0x0, 0x0) A A A A /go/src/[12]github.com/docker/docker/layer/layer_store.go:516 +0x537 [13]github.com/docker/docker/daemon.(*Daemon).setRWLayer(0xc4203d6000, 0xc423c50fc0, 0xc422cbfc00, 0x0) A A A A /go/src/[14]github.com/docker/docker/daemon/create.go:251 +0x11f [15]github.com/docker/docker/daemon.(*Daemon).create(0xc4203d6000, 0x0, 0x0, 0xc422bc3e00, 0xc422cbfc00, 0xc4258e3420, 0x0, 0x0, 0x0, 0x0, ...) A A A A /go/src/[16]github.com/docker/docker/daemon/create.go:133 +0x236 [17]github.com/docker/docker/daemon.(*Daemon).containerCreate(0xc4203d6 000, 0x0, 0x0, 0xc422bc3e00, 0xc422cbfc00, 0xc4258e3420, 0x0, 0x0, 0x0, 0x0, ...) A A A A /go/src/[18]github.com/docker/docker/daemon/create.go:60 +0x1a4 [19]github.com/docker/docker/daemon.(*Daemon).ContainerCreate(0xc4203d6 000, 0x0, 0x0, 0xc422bc3e00, 0xc422cbfc00, 0xc4258e3420, 0x0, 0x0, 0x0, 0x0, ...) A A A A /go/src/[20]github.com/docker/docker/daemon/create.go:33 +0x9d Regards, A Kir.
References 1. mailto:aufs-users@lists.sourceforge.net 2. http://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/bionic 3. http://github.com/docker/docker/daemon/initlayer.Setup(0xc4230d3ce0 4. http://github.com/docker/docker/daemon/initlayer/setup_unix.go:50 5. http://github.com/docker/docker/daemon.(*Daemon).setupInitLayer(0xc4203d6000 6. http://github.com/docker/docker/daemon/daemon.go:1033 7. http://github.com/docker/docker/daemon.(*Daemon).(github.com/docker/docker/daemon.setupInitLayer)-fm(0xc4230d3ce0 8. http://github.com/docker/docker/daemon/daemon_unix.go:934 9. http://github.com/docker/docker/layer.(*layerStore).initMount(0xc4203b5310 10. http://github.com/docker/docker/layer/layer_store.go:662 11. http://github.com/docker/docker/layer.(*layerStore).CreateRWLayer(0xc4203b5310 12. http://github.com/docker/docker/layer/layer_store.go:516 13. http://github.com/docker/docker/daemon.(*Daemon).setRWLayer(0xc4203d6000 14. http://github.com/docker/docker/daemon/create.go:251 15. http://github.com/docker/docker/daemon.(*Daemon).create(0xc4203d6000 16. http://github.com/docker/docker/daemon/create.go:133 17. http://github.com/docker/docker/daemon.(*Daemon).containerCreate(0xc4203d6000 18. http://github.com/docker/docker/daemon/create.go:60 19. http://github.com/docker/docker/daemon.(*Daemon).ContainerCreate(0xc4203d6000 20. http://github.com/docker/docker/daemon/create.go:33
