Hi, With aufs 20081013, debian kernel 2.6.25 recompiled with MAX_LOCKDEP_SUBCLASSES increased to 16, aufs exported by NFS, I get a potential recursive locking alert, and later a deadlock running several instances of LTP test renameat01 in parallel on different NFS clients.
Here is the lockdep alert:
[ 103.602744] =============================================
[ 103.602874] [ INFO: possible recursive locking detected ]
[ 103.602953] 2.6.25-elric #10
[ 103.603022] ---------------------------------------------
[ 103.603094] nfsd/2597 is trying to acquire lock:
[ 103.603161] (&rw->rwsem){----}, at: [<ffffffff883dd63c>]
aufs_read_lock+0x9b/0xd7 [aufs]
[ 103.603415]
[ 103.603416] but task is already holding lock:
[ 103.603535] (&rw->rwsem){----}, at: [<ffffffff8840d86c>]
aufs_fh_to_dentry+0x106/0x48a [aufs]
[ 103.603786]
[ 103.603787] other info that might help us debug this:
[ 103.603909] 2 locks held by nfsd/2597:
[ 103.603972] #0: (hash_sem){..--}, at: [<ffffffff882e9801>]
nfsd+0x189/0x2b4 [nfsd]
[ 103.604250] #1: (&rw->rwsem){----}, at: [<ffffffff8840d86c>]
aufs_fh_to_dentry+0x106/0x48a [aufs]
[ 103.604536]
[ 103.604536] stack backtrace:
[ 103.604652] Pid: 2597, comm: nfsd Not tainted 2.6.25-elric #10
[ 103.604721]
[ 103.604721] Call Trace:
[ 103.604843] [<ffffffff80253b7b>] __lock_acquire+0x8e0/0xc7a
[ 103.604914] [<ffffffff80246081>] __kernel_text_address+0x1a/0x26
[ 103.604987] [<ffffffff80254362>] lock_acquire+0x88/0xb2
[ 103.605067] [<ffffffff883dd63c>] :aufs:aufs_read_lock+0x9b/0xd7
[ 103.605141] [<ffffffff8042f7a0>] down_read+0x32/0x3b
[ 103.605219] [<ffffffff883dd63c>] :aufs:aufs_read_lock+0x9b/0xd7
[ 103.605292] [<ffffffff8025177b>] save_trace+0x37/0x91
[ 103.605378] [<ffffffff883fe9cd>] :aufs:aufs_permission+0x2f1/0x931
[ 103.605454] [<ffffffff802b676c>] lookup_mnt+0x19/0x4c
[ 103.605598] [<ffffffff802b3195>] dput+0x37/0x10f
[ 103.605672] [<ffffffff802a90f9>] __follow_mount+0x26/0x7b
[ 103.605743] [<ffffffff802a8efd>] permission+0xb5/0x10a
[ 103.605813] [<ffffffff802aac76>] __link_path_walk+0x141/0xda8
[ 103.605884] [<ffffffff802b238b>] __d_path+0xaa/0x142
[ 103.605955] [<ffffffff802ab93a>] path_walk+0x5d/0xb9
[ 103.606025] [<ffffffff802abc20>] do_path_lookup+0x165/0x1de
[ 103.606113] [<ffffffff8840d22e>] :aufs:decode_by_path+0x4b3/0x579
[ 103.606189] [<ffffffff802b43e9>] ifind_fast+0x1a/0x92
[ 103.606259] [<ffffffff80430685>] _spin_unlock+0x17/0x20
[ 103.606330] [<ffffffff802b4458>] ifind_fast+0x89/0x92
[ 103.606416] [<ffffffff8840da6d>] :aufs:aufs_fh_to_dentry+0x307/0x48a
[ 103.606509] [<ffffffff8829d46e>] :sunrpc:sunrpc_cache_lookup+0x30/0x132
[ 103.606581] [<ffffffff8043063b>] _read_unlock+0x17/0x20
[ 103.606581] [<ffffffff8829c3d7>] :sunrpc:cache_check+0x66/0x435
[ 103.606581] [<ffffffff88289593>] :exportfs:exportfs_decode_fh+0x30/0x1e1
[ 103.606581] [<ffffffff882ebc32>] :nfsd:nfsd_acceptable+0x0/0xd3
[ 103.606581] [<ffffffff8029dc3d>] kfree+0x20d/0x224
[ 103.606581] [<ffffffff80252e07>] trace_hardirqs_on+0xf5/0x118
[ 103.606581] [<ffffffff80242023>] set_current_groups+0x15e/0x16a
[ 103.606581] [<ffffffff882f1e96>] :nfsd:nfsd_setuser+0x11f/0x20d
[ 103.606581] [<ffffffff882eb7bb>] :nfsd:nfsd_setuser_and_check_port+0x52/0x57
[ 103.606581] [<ffffffff882ebf52>] :nfsd:fh_verify+0x24d/0x48f
[ 103.606581] [<ffffffff8029d142>] __kmalloc+0x142/0x176
[ 103.606581] [<ffffffff882f3ea3>] :nfsd:nfsd3_proc_getattr+0x7d/0xb5
[ 103.606581] [<ffffffff882e9249>] :nfsd:nfsd_dispatch+0xde/0x1b6
[ 103.606581] [<ffffffff88296854>] :sunrpc:svc_process+0x3eb/0x707
[ 103.606581] [<ffffffff882e9678>] :nfsd:nfsd+0x0/0x2b4
[ 103.606581] [<ffffffff882e9818>] :nfsd:nfsd+0x1a0/0x2b4
[ 103.606581] [<ffffffff8020d028>] child_rip+0xa/0x12
[ 103.606581] [<ffffffff8020c5bc>] restore_args+0x0/0x30
[ 103.606581] [<ffffffff882e9678>] :nfsd:nfsd+0x0/0x2b4
[ 103.606581] [<ffffffff8020d01e>] child_rip+0x0/0x12
[ 103.606581]
Here is the result of SysRq w after deadlock (snipped sched debug info):
[ 4170.506054] SysRq : Show Blocked State
[ 4170.506054] task PC stack pid father
[ 4170.506054] nfsd D 0000000000000000 0 2595 2
[ 4170.506054] ffff81007c9cbb10 0000000000000046 0000000000000002
0000000100000000
[ 4170.506054] ffffffff80650100 ffffffff80650100 ffffffff80650100
ffffffff8064d1c0
[ 4170.506054] ffffffff80650100 ffff81007c87ec30 ffff81007c87e9c0
ffff81007d9b0100
[ 4170.506054] Call Trace:
[ 4170.506054] [<ffffffff8042fffe>] __down_read+0x15/0x9e
[ 4170.506054] [<ffffffff8043006f>] __down_read+0x86/0x9e
[ 4170.506054] [<ffffffff883f2ee1>] :aufs:di_read_lock+0x93/0x180
[ 4170.506054] [<ffffffff883fd0b3>] :aufs:au_do_pin+0x16a/0x47b
[ 4170.506054] [<ffffffff8024b10f>] down_write_trylock+0xe/0x38
[ 4170.506054] [<ffffffff883fd17a>] :aufs:au_do_pin+0x231/0x47b
[ 4170.506054] [<ffffffff8024b10f>] down_write_trylock+0xe/0x38
[ 4170.506054] [<ffffffff88403667>] :aufs:au_ren_pin3+0x6a/0x105
[ 4170.506054] [<ffffffff88403cc0>] :aufs:au_ren_lock+0x4a5/0x870
[ 4170.506054] [<ffffffff884060ee>] :aufs:aufs_rename+0x694/0x1396
[ 4170.506054] [<ffffffff802b2b38>] __d_lookup+0xa1/0x163
[ 4170.506054] [<ffffffff802a9f47>] vfs_rename+0x296/0x41a
[ 4170.506054] [<ffffffff882ee6c1>] :nfsd:nfsd_rename+0x28c/0x31e
[ 4170.506054] [<ffffffff882f46b6>] :nfsd:nfsd3_proc_rename+0x125/0x135
[ 4170.506054] [<ffffffff882e9249>] :nfsd:nfsd_dispatch+0xde/0x1b6
[ 4170.506054] [<ffffffff88296854>] :sunrpc:svc_process+0x3eb/0x707
[ 4170.506054] [<ffffffff882e9678>] :nfsd:nfsd+0x0/0x2b4
[ 4170.506054] [<ffffffff882e9818>] :nfsd:nfsd+0x1a0/0x2b4
[ 4170.506054] [<ffffffff8020d028>] child_rip+0xa/0x12
[ 4170.506054] [<ffffffff8020c5bc>] restore_args+0x0/0x30
[ 4170.506054] [<ffffffff882e9678>] :nfsd:nfsd+0x0/0x2b4
[ 4170.506054] [<ffffffff8020d01e>] child_rip+0x0/0x12
[ 4170.506054]
[ 4170.506054] nfsd D 0000000000000000 0 2599 2
[ 4170.506054] ffff81007d1d1bd0 0000000000000002 0000000000000002
0000000100000000
[ 4170.506054] ffffffff80650100 ffffffff80650100 ffffffff80650100
ffffffff8064d1c0
[ 4170.506054] ffffffff80650100 ffff81007cc22930 ffff81007cc226c0
ffff81007d9b0100
[ 4170.506054] Call Trace:
[ 4170.506054] [<ffffffff8042fffe>] __down_read+0x15/0x9e
[ 4170.506054] [<ffffffff802a918e>] lock_rename+0x40/0xe9
[ 4170.506054] [<ffffffff8042f223>] mutex_lock_nested+0x172/0x285
[ 4170.506054] [<ffffffff802a918e>] lock_rename+0x40/0xe9
[ 4170.506054] [<ffffffff88403a12>] :aufs:au_ren_lock+0x1f7/0x870
[ 4170.506054] [<ffffffff884060ee>] :aufs:aufs_rename+0x694/0x1396
[ 4170.506054] [<ffffffff802b2b38>] __d_lookup+0xa1/0x163
[ 4170.506054] [<ffffffff802a9f47>] vfs_rename+0x296/0x41a
[ 4170.506054] [<ffffffff882ee6c1>] :nfsd:nfsd_rename+0x28c/0x31e
[ 4170.506054] [<ffffffff882f46b6>] :nfsd:nfsd3_proc_rename+0x125/0x135
[ 4170.506054] [<ffffffff882e9249>] :nfsd:nfsd_dispatch+0xde/0x1b6
[ 4170.506054] [<ffffffff88296854>] :sunrpc:svc_process+0x3eb/0x707
[ 4170.506054] [<ffffffff882e9678>] :nfsd:nfsd+0x0/0x2b4
[ 4170.506054] [<ffffffff882e9818>] :nfsd:nfsd+0x1a0/0x2b4
[ 4170.506054] [<ffffffff8020d028>] child_rip+0xa/0x12
[ 4170.506054] [<ffffffff8020c5bc>] restore_args+0x0/0x30
[ 4170.506054] [<ffffffff882e9678>] :nfsd:nfsd+0x0/0x2b4
[ 4170.506054] [<ffffffff8020d01e>] child_rip+0x0/0x12
[ 4170.506054]
[ 4170.506054] nfsd D 0000000000000000 0 2602 2
[ 4170.506054] ffff81007cab7bd0 0000000000000002 ffff81007cab7b98
0000000100000000
[ 4170.506054] ffffffff80650100 ffffffff80650100 ffffffff80650100
ffffffff8064d1c0
[ 4170.506054] ffffffff80650100 ffff81007e940770 ffff81007e940500
ffff81007f3a8240
[ 4170.506054] Call Trace:
[ 4170.506054] [<ffffffff8042fffe>] __down_read+0x15/0x9e
[ 4170.506054] [<ffffffff802a918e>] lock_rename+0x40/0xe9
[ 4170.506054] [<ffffffff8042f223>] mutex_lock_nested+0x172/0x285
[ 4170.506054] [<ffffffff802a918e>] lock_rename+0x40/0xe9
[ 4170.506054] [<ffffffff88403a12>] :aufs:au_ren_lock+0x1f7/0x870
[ 4170.506054] [<ffffffff884060ee>] :aufs:aufs_rename+0x694/0x1396
[ 4170.506054] [<ffffffff802b2b38>] __d_lookup+0xa1/0x163
[ 4170.506054] [<ffffffff802a9f47>] vfs_rename+0x296/0x41a
[ 4170.506054] [<ffffffff882ee6c1>] :nfsd:nfsd_rename+0x28c/0x31e
[ 4170.506054] [<ffffffff882f46b6>] :nfsd:nfsd3_proc_rename+0x125/0x135
[ 4170.506054] [<ffffffff882e9249>] :nfsd:nfsd_dispatch+0xde/0x1b6
[ 4170.506054] [<ffffffff88296854>] :sunrpc:svc_process+0x3eb/0x707
[ 4170.506054] [<ffffffff882e9678>] :nfsd:nfsd+0x0/0x2b4
[ 4170.506054] [<ffffffff882e9818>] :nfsd:nfsd+0x1a0/0x2b4
[ 4170.506054] [<ffffffff8020d028>] child_rip+0xa/0x12
[ 4170.506054] [<ffffffff8020c5bc>] restore_args+0x0/0x30
[ 4170.506054] [<ffffffff882e9678>] :nfsd:nfsd+0x0/0x2b4
[ 4170.506054] [<ffffffff8020d01e>] child_rip+0x0/0x12
[ 4170.506054]
[ 4170.506054] rpc.mountd D 0000000000000000 0 2606 1
[ 4170.506054] ffff81007dd07a88 0000000000000046 0000000000000002
0000000100000000
[ 4170.506054] ffffffff80650100 ffffffff80650100 ffffffff80650100
ffffffff8064d1c0
[ 4170.506054] ffffffff80650100 ffff81007d532670 ffff81007d532400
ffff81003796a080
[ 4170.506054] Call Trace:
[ 4170.506054] [<ffffffff8042ff59>] __down_write_nested+0x15/0x9e
[ 4170.506054] [<ffffffff8042ffca>] __down_write_nested+0x86/0x9e
[ 4170.506054] [<ffffffff883f2e31>] :aufs:di_write_lock+0x87/0xa4
[ 4170.506054] [<ffffffff883dd659>] :aufs:aufs_read_lock+0xb8/0xd7
[ 4170.506054] [<ffffffff8840c39c>] :aufs:au_nfsd_do_read_lock+0x12b/0x21d
[ 4170.506054] [<ffffffff883f24b8>] :aufs:aufs_d_revalidate+0xe3/0x447
[ 4170.506054] [<ffffffff802b2b38>] __d_lookup+0xa1/0x163
[ 4170.506054] [<ffffffff802b2be8>] __d_lookup+0x151/0x163
[ 4170.506054] [<ffffffff802b3195>] dput+0x37/0x10f
[ 4170.506054] [<ffffffff802a93e0>] do_lookup+0x15a/0x1b1
[ 4170.506054] [<ffffffff802aae95>] __link_path_walk+0x360/0xda8
[ 4170.506054] [<ffffffff802ab93a>] path_walk+0x5d/0xb9
[ 4170.506054] [<ffffffff802abc20>] do_path_lookup+0x165/0x1de
[ 4170.506054] [<ffffffff802aa92e>] getname+0x140/0x1a3
[ 4170.506054] [<ffffffff802ac467>] __user_walk_fd+0x37/0x4c
[ 4170.506054] [<ffffffff802a5604>] vfs_stat_fd+0x1b/0x4a
[ 4170.506054] [<ffffffff802a568f>] sys_newstat+0x19/0x31
[ 4170.506054] [<ffffffff804301a4>] trace_hardirqs_on_thunk+0x35/0x3a
[ 4170.506054] [<ffffffff8020be8a>] system_call_after_swapgs+0x8a/0x8f
[ 4170.506054]
Thanks!
Louis
--
Dr Louis Rilling Kerlabs
Skype: louis.rilling Batiment Germanium
Phone: (+33|0) 6 80 89 08 23 80 avenue des Buttes de Coesmes
http://www.kerlabs.com/ 35700 Rennes
signature.asc
Description: Digital signature
------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/
