Re: [Nfs-ganesha-devel] ABBA deadlock in 2.5 (likely in 2.6 as well)
Sounds good, will give it a try with nasty "racer". Thank you Dan. Regards, Malahal. On Mon, Oct 23, 2017 at 8:27 PM, Daniel Gryniewiczwrote: > Maybe something like this: > > https://paste.fedoraproject.org/paste/CptGkmoRutBKYjno5FiSjg/ > > Daniel > > > On 10/23/2017 10:13 AM, Malahal Naineni wrote: > >> Let us say we have X/Y path in the file system. We have attr_lock and >> content_lock on each object. The locks we are interested here are >> attr_lock on X (hear after referred to as AX) and content_lock on X >> (hear after CX). Similarly we have AY and CY for object named Y. >> >> 1. Thread 50 (lookup called for X ) takes AX and waits for CX (attr_lock >> followed by content_lock is the expected order) >> >> 2. Thread 251 (readdirplus on X) takes CX, AY and then waits for CY for >> processing object Y >> >> 3. Thread 132 (readdirplus on Y) takes CY, and then waits for AX (this >> is due to lookup of parent) >> >> Classic philosopher's problem: 1 waits for 2, 2 waits for 3 and then 3 >> waits for 1. The lock ordering for attr_lock and content_lock for an >> object is attr_lock followed by content_lock. We can assume that parent >> locks should be acquired before the child locks, but DOTDOT appears as a >> child in readdirplus/readdir. If we can handle parent differently, we >> might be OK. Any help would be appreciated. >> >> Regards, Malahal. >> >> (gdb) thread 50 >> [Switching to thread 50 (Thread 0x3fff6cffe850 (LWP 37851))] >> #0 0x3fff8a50089c in .__pthread_rwlock_wrlock () from >> /lib64/libpthread.so.0 >> (gdb) bt >> #0 0x3fff8a50089c in .__pthread_rwlock_wrlock () from >> /lib64/libpthread.so.0 >> #1 0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffccc0541f0, >> need_acl=false, invalidate=true) >> at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FS >> AL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187 >> #2 0x101ae1b0 in mdcache_getattrs (obj_hdl=0x3ffccc054228, >> attrs_out=0x3fff6cffcfb8) >> at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FS >> AL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1228 >> #3 0x100b9fdc in nfs_SetPostOpAttr (obj=0x3ffccc054228, >> Fattr=0x3ffe64050dc8, attrs=0x0) >> at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/Pr >> otocols/NFS/nfs_proto_tools.c:91 >> #4 0x100c6ba8 in nfs3_lookup (arg=0x3ffa9ff04780, >> req=0x3ffa9ff03f78, res=0x3ffe64050d50) >> at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/Pr >> otocols/NFS/nfs3_lookup.c:131 >> #5 0x10065220 in nfs_rpc_execute (reqdata=0x3ffa9ff03f50) >> at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/Ma >> inNFSD/nfs_worker_thread.c:1290 >> #6 0x10065c9c in worker_run (ctx=0x10013c1d3f0) >> at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/Ma >> inNFSD/nfs_worker_thread.c:1562 >> #7 0x101670f4 in fridgethr_start_routine (arg=0x10013c1d3f0) >> at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/su >> pport/fridgethr.c:550 >> #8 0x3fff8a4fc2bc in .start_thread () from /lib64/libpthread.so.0 >> #9 0x3fff8a31b304 in .__clone () from /lib64/libc.so.6 >> (gdb) frame 1 >> #1 0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffccc0541f0, >> need_acl=false, invalidate=true) >> at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FS >> AL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187 >> 1187PTHREAD_RWLOCK_wrlock(>content_lock); >> >> (gdb) p entry >> $1 = (mdcache_entry_t *) 0x3ffccc0541f0 >> (gdb) p entry->content_lock >> $2 = {__data = {__lock = 0, __nr_readers = 1, __readers_wakeup = 2408, >> __writer_wakeup = 4494, __nr_readers_queued = 0, >> __nr_writers_queued = 6, __writer = 0, __shared = 0, __pad1 = 0, >> __pad2 = 0, __flags = 0}, >>__size = "\000\000\000\000\000\000\000\001\000\000\th\000\000\021\216 >> \000\000\000\000\000\000\000\006", '\000' , __align = >> 1} >> (gdb) thread 251 >> [Switching to thread 251 (Thread 0x3fff2e7fe850 (LWP 37976))] >> #0 0x3fff8a50089c in .__pthread_rwlock_wrlock () from >> /lib64/libpthread.so.0 >> (gdb) bt >> #0 0x3fff8a50089c in .__pthread_rwlock_wrlock () from >> /lib64/libpthread.so.0 >> #1 0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffc30041bd0, >> need_acl=false, invalidate=true) >> at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FS >> AL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187 >> #2 0x101ae1b0 in mdcache_getattrs (obj_hdl=0x3ffc30041c08, >> attrs_out=0x3fff2e7fcb58) >> at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FS >> AL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1228 >> #3 0x101c0f50 in mdcache_readdir_chunked >> (directory=0x3ffccc0541f0, whence=1298220731, >> dir_state=0x3fff2e7fcee8, cb=@0x1024b040: 0x1003f524 >> , attrmask=122830, eod_met=0x3fff2e7fd01c) >> at
Re: [Nfs-ganesha-devel] ABBA deadlock in 2.5 (likely in 2.6 as well)
> Maybe something like this: > > https://paste.fedoraproject.org/paste/CptGkmoRutBKYjno5FiSjg/ That looks like it should fix the problem. Frank > On 10/23/2017 10:13 AM, Malahal Naineni wrote: > > Let us say we have X/Y path in the file system. We have attr_lock and > > content_lock on each object. The locks we are interested here are > > attr_lock on X (hear after referred to as AX) and content_lock on X > > (hear after CX). Similarly we have AY and CY for object named Y. > > > > 1. Thread 50 (lookup called for X ) takes AX and waits for CX (attr_lock > > followed by content_lock is the expected order) > > > > 2. Thread 251 (readdirplus on X) takes CX, AY and then waits for CY for > > processing object Y > > > > 3. Thread 132 (readdirplus on Y) takes CY, and then waits for AX (this > > is due to lookup of parent) > > > > Classic philosopher's problem: 1 waits for 2, 2 waits for 3 and then 3 > > waits for 1. The lock ordering for attr_lock and content_lock for an > > object is attr_lock followed by content_lock. We can assume that > > parent locks should be acquired before the child locks, but DOTDOT > > appears as a child in readdirplus/readdir. If we can handle parent > > differently, we might be OK. Any help would be appreciated. > > > > Regards, Malahal. > > > > (gdb) thread 50 > > [Switching to thread 50 (Thread 0x3fff6cffe850 (LWP 37851))] > > #0 0x3fff8a50089c in .__pthread_rwlock_wrlock () from > > /lib64/libpthread.so.0 > > (gdb) bt > > #0 0x3fff8a50089c in .__pthread_rwlock_wrlock () from > > /lib64/libpthread.so.0 > > #1 0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffccc0541f0, > > need_acl=false, invalidate=true) > > at > > /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1- > Source/FSAL/Stackab > > le_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187 > > #2 0x101ae1b0 in mdcache_getattrs (obj_hdl=0x3ffccc054228, > > attrs_out=0x3fff6cffcfb8) > > at > > /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1- > Source/FSAL/Stackab > > le_FSALs/FSAL_MDCACHE/mdcache_handle.c:1228 > > #3 0x100b9fdc in nfs_SetPostOpAttr (obj=0x3ffccc054228, > > Fattr=0x3ffe64050dc8, attrs=0x0) > > at > > /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1- > Source/Protocols/NF > > S/nfs_proto_tools.c:91 > > #4 0x100c6ba8 in nfs3_lookup (arg=0x3ffa9ff04780, > > req=0x3ffa9ff03f78, res=0x3ffe64050d50) > > at > > /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1- > Source/Protocols/NF > > S/nfs3_lookup.c:131 > > #5 0x10065220 in nfs_rpc_execute (reqdata=0x3ffa9ff03f50) > > at > > /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1- > Source/MainNFSD/nfs > > _worker_thread.c:1290 > > #6 0x10065c9c in worker_run (ctx=0x10013c1d3f0) > > at > > /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1- > Source/MainNFSD/nfs > > _worker_thread.c:1562 > > #7 0x101670f4 in fridgethr_start_routine (arg=0x10013c1d3f0) > > at > > /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/support/frid > > gethr.c:550 > > #8 0x3fff8a4fc2bc in .start_thread () from /lib64/libpthread.so.0 > > #9 0x3fff8a31b304 in .__clone () from /lib64/libc.so.6 > > (gdb) frame 1 > > #1 0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffccc0541f0, > > need_acl=false, invalidate=true) > > at > > /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1- > Source/FSAL/Stackab > > le_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187 > > 1187PTHREAD_RWLOCK_wrlock(>content_lock); > > (gdb) p entry > > $1 = (mdcache_entry_t *) 0x3ffccc0541f0 > > (gdb) p entry->content_lock > > $2 = {__data = {__lock = 0, __nr_readers = 1, __readers_wakeup = 2408, > > __writer_wakeup = 4494, __nr_readers_queued = 0, > > __nr_writers_queued = 6, __writer = 0, __shared = 0, __pad1 = 0, > > __pad2 = 0, __flags = 0}, > >__size = > > > "\000\000\000\000\000\000\000\001\000\000\th\000\000\021\216\000\000\0 > > 00\000\000\000\000\006", '\000' , __align = 1} > > (gdb) thread 251 > > [Switching to thread 251 (Thread 0x3fff2e7fe850 (LWP 37976))] > > #0 0x3fff8a50089c in .__pthread_rwlock_wrlock () from > > /lib64/libpthread.so.0 > > (gdb) bt > > #0 0x3fff8a50089c in .__pthread_rwlock_wrlock () from > > /lib64/libpthread.so.0 > > #1 0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffc30041bd0, > > need_acl=false, invalidate=true) > > at > > /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1- > Source/FSAL/Stackab > > le_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187 > > #2 0x101ae1b0 in mdcache_getattrs (obj_hdl=0x3ffc30041c08, > > attrs_out=0x3fff2e7fcb58) > > at > > /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1- > Source/FSAL/Stackab > > le_FSALs/FSAL_MDCACHE/mdcache_handle.c:1228 > > #3 0x101c0f50 in mdcache_readdir_chunked > > (directory=0x3ffccc0541f0, whence=1298220731, > > dir_state=0x3fff2e7fcee8, cb=@0x1024b040: 0x1003f524 > > , attrmask=122830, eod_met=0x3fff2e7fd01c) > > at > >
Re: [Nfs-ganesha-devel] ABBA deadlock in 2.5 (likely in 2.6 as well)
Maybe something like this: https://paste.fedoraproject.org/paste/CptGkmoRutBKYjno5FiSjg/ Daniel On 10/23/2017 10:13 AM, Malahal Naineni wrote: Let us say we have X/Y path in the file system. We have attr_lock and content_lock on each object. The locks we are interested here are attr_lock on X (hear after referred to as AX) and content_lock on X (hear after CX). Similarly we have AY and CY for object named Y. 1. Thread 50 (lookup called for X ) takes AX and waits for CX (attr_lock followed by content_lock is the expected order) 2. Thread 251 (readdirplus on X) takes CX, AY and then waits for CY for processing object Y 3. Thread 132 (readdirplus on Y) takes CY, and then waits for AX (this is due to lookup of parent) Classic philosopher's problem: 1 waits for 2, 2 waits for 3 and then 3 waits for 1. The lock ordering for attr_lock and content_lock for an object is attr_lock followed by content_lock. We can assume that parent locks should be acquired before the child locks, but DOTDOT appears as a child in readdirplus/readdir. If we can handle parent differently, we might be OK. Any help would be appreciated. Regards, Malahal. (gdb) thread 50 [Switching to thread 50 (Thread 0x3fff6cffe850 (LWP 37851))] #0 0x3fff8a50089c in .__pthread_rwlock_wrlock () from /lib64/libpthread.so.0 (gdb) bt #0 0x3fff8a50089c in .__pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #1 0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffccc0541f0, need_acl=false, invalidate=true) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187 #2 0x101ae1b0 in mdcache_getattrs (obj_hdl=0x3ffccc054228, attrs_out=0x3fff6cffcfb8) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1228 #3 0x100b9fdc in nfs_SetPostOpAttr (obj=0x3ffccc054228, Fattr=0x3ffe64050dc8, attrs=0x0) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/Protocols/NFS/nfs_proto_tools.c:91 #4 0x100c6ba8 in nfs3_lookup (arg=0x3ffa9ff04780, req=0x3ffa9ff03f78, res=0x3ffe64050d50) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/Protocols/NFS/nfs3_lookup.c:131 #5 0x10065220 in nfs_rpc_execute (reqdata=0x3ffa9ff03f50) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/MainNFSD/nfs_worker_thread.c:1290 #6 0x10065c9c in worker_run (ctx=0x10013c1d3f0) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/MainNFSD/nfs_worker_thread.c:1562 #7 0x101670f4 in fridgethr_start_routine (arg=0x10013c1d3f0) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/support/fridgethr.c:550 #8 0x3fff8a4fc2bc in .start_thread () from /lib64/libpthread.so.0 #9 0x3fff8a31b304 in .__clone () from /lib64/libc.so.6 (gdb) frame 1 #1 0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffccc0541f0, need_acl=false, invalidate=true) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187 1187PTHREAD_RWLOCK_wrlock(>content_lock); (gdb) p entry $1 = (mdcache_entry_t *) 0x3ffccc0541f0 (gdb) p entry->content_lock $2 = {__data = {__lock = 0, __nr_readers = 1, __readers_wakeup = 2408, __writer_wakeup = 4494, __nr_readers_queued = 0, __nr_writers_queued = 6, __writer = 0, __shared = 0, __pad1 = 0, __pad2 = 0, __flags = 0}, __size = "\000\000\000\000\000\000\000\001\000\000\th\000\000\021\216\000\000\000\000\000\000\000\006", '\000' , __align = 1} (gdb) thread 251 [Switching to thread 251 (Thread 0x3fff2e7fe850 (LWP 37976))] #0 0x3fff8a50089c in .__pthread_rwlock_wrlock () from /lib64/libpthread.so.0 (gdb) bt #0 0x3fff8a50089c in .__pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #1 0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffc30041bd0, need_acl=false, invalidate=true) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187 #2 0x101ae1b0 in mdcache_getattrs (obj_hdl=0x3ffc30041c08, attrs_out=0x3fff2e7fcb58) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1228 #3 0x101c0f50 in mdcache_readdir_chunked (directory=0x3ffccc0541f0, whence=1298220731, dir_state=0x3fff2e7fcee8, cb=@0x1024b040: 0x1003f524 , attrmask=122830, eod_met=0x3fff2e7fd01c) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:3047 #4 0x101ab998 in mdcache_readdir (dir_hdl=0x3ffccc054228, whence=0x3fff2e7fcfa0, dir_state=0x3fff2e7fcee8, cb=@0x1024b040: 0x1003f524 , attrmask=122830, eod_met=0x3fff2e7fd01c) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:637 #5 0x10040090 in fsal_readdir
[Nfs-ganesha-devel] ABBA deadlock in 2.5 (likely in 2.6 as well)
Let us say we have X/Y path in the file system. We have attr_lock and content_lock on each object. The locks we are interested here are attr_lock on X (hear after referred to as AX) and content_lock on X (hear after CX). Similarly we have AY and CY for object named Y. 1. Thread 50 (lookup called for X ) takes AX and waits for CX (attr_lock followed by content_lock is the expected order) 2. Thread 251 (readdirplus on X) takes CX, AY and then waits for CY for processing object Y 3. Thread 132 (readdirplus on Y) takes CY, and then waits for AX (this is due to lookup of parent) Classic philosopher's problem: 1 waits for 2, 2 waits for 3 and then 3 waits for 1. The lock ordering for attr_lock and content_lock for an object is attr_lock followed by content_lock. We can assume that parent locks should be acquired before the child locks, but DOTDOT appears as a child in readdirplus/readdir. If we can handle parent differently, we might be OK. Any help would be appreciated. Regards, Malahal. (gdb) thread 50 [Switching to thread 50 (Thread 0x3fff6cffe850 (LWP 37851))] #0 0x3fff8a50089c in .__pthread_rwlock_wrlock () from /lib64/libpthread.so.0 (gdb) bt #0 0x3fff8a50089c in .__pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #1 0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffccc0541f0, need_acl=false, invalidate=true) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187 #2 0x101ae1b0 in mdcache_getattrs (obj_hdl=0x3ffccc054228, attrs_out=0x3fff6cffcfb8) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1228 #3 0x100b9fdc in nfs_SetPostOpAttr (obj=0x3ffccc054228, Fattr=0x3ffe64050dc8, attrs=0x0) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/Protocols/NFS/nfs_proto_tools.c:91 #4 0x100c6ba8 in nfs3_lookup (arg=0x3ffa9ff04780, req=0x3ffa9ff03f78, res=0x3ffe64050d50) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/Protocols/NFS/nfs3_lookup.c:131 #5 0x10065220 in nfs_rpc_execute (reqdata=0x3ffa9ff03f50) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/MainNFSD/nfs_worker_thread.c:1290 #6 0x10065c9c in worker_run (ctx=0x10013c1d3f0) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/MainNFSD/nfs_worker_thread.c:1562 #7 0x101670f4 in fridgethr_start_routine (arg=0x10013c1d3f0) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/support/fridgethr.c:550 #8 0x3fff8a4fc2bc in .start_thread () from /lib64/libpthread.so.0 #9 0x3fff8a31b304 in .__clone () from /lib64/libc.so.6 (gdb) frame 1 #1 0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffccc0541f0, need_acl=false, invalidate=true) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187 1187 PTHREAD_RWLOCK_wrlock(>content_lock); (gdb) p entry $1 = (mdcache_entry_t *) 0x3ffccc0541f0 (gdb) p entry->content_lock $2 = {__data = {__lock = 0, __nr_readers = 1, __readers_wakeup = 2408, __writer_wakeup = 4494, __nr_readers_queued = 0, __nr_writers_queued = 6, __writer = 0, __shared = 0, __pad1 = 0, __pad2 = 0, __flags = 0}, __size = "\000\000\000\000\000\000\000\001\000\000\th\000\000\021\216\000\000\000\000\000\000\000\006", '\000' , __align = 1} (gdb) thread 251 [Switching to thread 251 (Thread 0x3fff2e7fe850 (LWP 37976))] #0 0x3fff8a50089c in .__pthread_rwlock_wrlock () from /lib64/libpthread.so.0 (gdb) bt #0 0x3fff8a50089c in .__pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #1 0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffc30041bd0, need_acl=false, invalidate=true) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187 #2 0x101ae1b0 in mdcache_getattrs (obj_hdl=0x3ffc30041c08, attrs_out=0x3fff2e7fcb58) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1228 #3 0x101c0f50 in mdcache_readdir_chunked (directory=0x3ffccc0541f0, whence=1298220731, dir_state=0x3fff2e7fcee8, cb=@0x1024b040: 0x1003f524 , attrmask=122830, eod_met=0x3fff2e7fd01c) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:3047 #4 0x101ab998 in mdcache_readdir (dir_hdl=0x3ffccc054228, whence=0x3fff2e7fcfa0, dir_state=0x3fff2e7fcee8, cb=@0x1024b040: 0x1003f524 , attrmask=122830, eod_met=0x3fff2e7fd01c) at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:637 #5 0x10040090 in fsal_readdir (directory=0x3ffccc054228, cookie=1298220731, nbfound=0x3fff2e7fd018, eod_met=0x3fff2e7fd01c, attrmask=122830, cb=@0x10250bb0: 0x100cabf4 , opaque=0x3fff2e7fd038) at