We are running a system with a customized 3.4 kernel on embedded powerpc, which
3 FS overlapped on /.
Incorrectly we performed 2 logrotate with crond every minute, and with some
logrotate upgrade (to 3.9.1), it sometimes do unlink and fsync on the same
logrotate.status.tmp file, which caused dead lock.
I checked the codes, it seems the 2 call paths will lock the i_mutex and some
unionfs locks in wrong order, thus causes the deadlock.
It is a wrong usage in App, meanwhile it also looks a hole in unionfs.
I tried the kernel with both 2.5.11 and a vanilla 2.6, same thing. Also don't
think it is about the kernel we use, checked a little with vanilla kernel too.
Don't know about other kernels like 4.x.
Liwei
PS the call trace when deadlock happens:
logrotate D 0ff2087c 0 29296 29294 0x00000000
Call Trace:
[e65c1c80] [1a1fc3a0] 0x1a1fc3a0 (unreliable)
[e65c1d40] [c0007f20] __switch_to+0xa0/0xd0
[e65c1d50] [c073d6a8] __schedule+0x3b8/0x8b0
[e65c1e60] [c073e0d4] schedule_preempt_disabled+0x24/0x40
[e65c1e70] [c073c39c] __mutex_lock_slowpath+0x10c/0x1e0
[e65c1eb0] [c073c4d4] mutex_lock+0x64/0x80 <--LOCKING inode i_mutex
[e65c1ec0] [c019ebec] generic_file_fsync+0x5c/0xc0
[e65c1ee0] [c029ead8] unionfs_fsync+0xc8/0x280 <--LOCKed some unionfs locks
[e65c1f30] [c01a89f0] sys_fsync+0x30/0x70
[e65c1f40] [c0010d64] ret_from_syscall+0x0/0x3c
--- Exception: c01 at 0xff2087c
LR = 0x10002228
logrotate D 0ff188b4 0 29297 29295 0x00000000
Call Trace:
[e674dc10] [c00860a4] load_balance+0xd4/0x760 (unreliable)
[e674dcd0] [c0007f20] __switch_to+0xa0/0xd0
[e674dce0] [c073d6a8] __schedule+0x3b8/0x8b0
[e674ddf0] [c073e0d4] schedule_preempt_disabled+0x24/0x40
[e674de00] [c073c39c] __mutex_lock_slowpath+0x10c/0x1e0
[e674de40] [c073c4d4] mutex_lock+0x64/0x80 <--LOCKING a unionfs lock
[e674de50] [c02a8908] unionfs_unlink+0x98/0x4c0
[e674de90] [c018328c] vfs_unlink+0xdc/0x160 <--LOCKED inode
[e674deb0] [c01834c0] do_unlinkat+0x1b0/0x1c0
[e674df40] [c0010d64] ret_from_syscall+0x0/0x3c
--- Exception: c01 at 0xff188b4
LR = 0x10001f80
_______________________________________________
unionfs mailing list: http://unionfs.filesystems.org/
[email protected]
http://www.fsl.cs.sunysb.edu/mailman/listinfo/unionfs
