Re: [Nfs-ganesha-devel] ABBA deadlock in 2.5 (likely in 2.6 as well)

2017-10-23 Thread Malahal Naineni
Sounds good, will give it a try with nasty "racer". Thank you Dan.

Regards, Malahal.

On Mon, Oct 23, 2017 at 8:27 PM, Daniel Gryniewicz  wrote:

> Maybe something like this:
>
> https://paste.fedoraproject.org/paste/CptGkmoRutBKYjno5FiSjg/
>
> Daniel
>
>
> On 10/23/2017 10:13 AM, Malahal Naineni wrote:
>
>> Let us say we have X/Y path in the file system. We have attr_lock and
>> content_lock on each object. The locks we are interested here are
>> attr_lock on X (hear after referred to as AX) and content_lock on X
>> (hear after CX).  Similarly we have AY and CY for object named Y.
>>
>> 1. Thread 50 (lookup called for X ) takes AX and waits for CX (attr_lock
>> followed by content_lock is the expected order)
>>
>> 2. Thread 251 (readdirplus on X) takes CX, AY and then waits for CY for
>> processing object Y
>>
>> 3. Thread 132 (readdirplus on Y) takes CY, and then waits for AX (this
>> is due to lookup of parent)
>>
>> Classic philosopher's problem: 1 waits for 2, 2 waits for 3 and then 3
>> waits for 1. The lock ordering for attr_lock and content_lock for an
>> object is attr_lock followed by content_lock. We can assume that parent
>> locks should be acquired before the child locks, but DOTDOT appears as a
>> child in readdirplus/readdir. If we can handle parent differently, we
>> might be OK. Any help would be appreciated.
>>
>> Regards, Malahal.
>>
>> (gdb) thread 50
>> [Switching to thread 50 (Thread 0x3fff6cffe850 (LWP 37851))]
>> #0  0x3fff8a50089c in .__pthread_rwlock_wrlock () from
>> /lib64/libpthread.so.0
>> (gdb) bt
>> #0  0x3fff8a50089c in .__pthread_rwlock_wrlock () from
>> /lib64/libpthread.so.0
>> #1  0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffccc0541f0,
>> need_acl=false, invalidate=true)
>>  at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FS
>> AL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187
>> #2  0x101ae1b0 in mdcache_getattrs (obj_hdl=0x3ffccc054228,
>> attrs_out=0x3fff6cffcfb8)
>>  at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FS
>> AL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1228
>> #3  0x100b9fdc in nfs_SetPostOpAttr (obj=0x3ffccc054228,
>> Fattr=0x3ffe64050dc8, attrs=0x0)
>>  at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/Pr
>> otocols/NFS/nfs_proto_tools.c:91
>> #4  0x100c6ba8 in nfs3_lookup (arg=0x3ffa9ff04780,
>> req=0x3ffa9ff03f78, res=0x3ffe64050d50)
>>  at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/Pr
>> otocols/NFS/nfs3_lookup.c:131
>> #5  0x10065220 in nfs_rpc_execute (reqdata=0x3ffa9ff03f50)
>>  at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/Ma
>> inNFSD/nfs_worker_thread.c:1290
>> #6  0x10065c9c in worker_run (ctx=0x10013c1d3f0)
>>  at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/Ma
>> inNFSD/nfs_worker_thread.c:1562
>> #7  0x101670f4 in fridgethr_start_routine (arg=0x10013c1d3f0)
>>  at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/su
>> pport/fridgethr.c:550
>> #8  0x3fff8a4fc2bc in .start_thread () from /lib64/libpthread.so.0
>> #9  0x3fff8a31b304 in .__clone () from /lib64/libc.so.6
>> (gdb) frame 1
>> #1  0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffccc0541f0,
>> need_acl=false, invalidate=true)
>>  at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FS
>> AL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187
>> 1187PTHREAD_RWLOCK_wrlock(>content_lock);
>>
>> (gdb) p entry
>> $1 = (mdcache_entry_t *) 0x3ffccc0541f0
>> (gdb) p entry->content_lock
>> $2 = {__data = {__lock = 0, __nr_readers = 1, __readers_wakeup = 2408,
>> __writer_wakeup = 4494, __nr_readers_queued = 0,
>>  __nr_writers_queued = 6, __writer = 0, __shared = 0, __pad1 = 0,
>> __pad2 = 0, __flags = 0},
>>__size = "\000\000\000\000\000\000\000\001\000\000\th\000\000\021\216
>> \000\000\000\000\000\000\000\006", '\000' , __align =
>> 1}
>> (gdb) thread 251
>> [Switching to thread 251 (Thread 0x3fff2e7fe850 (LWP 37976))]
>> #0  0x3fff8a50089c in .__pthread_rwlock_wrlock () from
>> /lib64/libpthread.so.0
>> (gdb) bt
>> #0  0x3fff8a50089c in .__pthread_rwlock_wrlock () from
>> /lib64/libpthread.so.0
>> #1  0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffc30041bd0,
>> need_acl=false, invalidate=true)
>>  at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FS
>> AL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187
>> #2  0x101ae1b0 in mdcache_getattrs (obj_hdl=0x3ffc30041c08,
>> attrs_out=0x3fff2e7fcb58)
>>  at /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FS
>> AL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1228
>> #3  0x101c0f50 in mdcache_readdir_chunked
>> (directory=0x3ffccc0541f0, whence=1298220731,
>>  dir_state=0x3fff2e7fcee8, cb=@0x1024b040: 0x1003f524
>> , attrmask=122830, eod_met=0x3fff2e7fd01c)
>>  at 

Re: [Nfs-ganesha-devel] ABBA deadlock in 2.5 (likely in 2.6 as well)

2017-10-23 Thread Frank Filz
 
> Maybe something like this:
> 
> https://paste.fedoraproject.org/paste/CptGkmoRutBKYjno5FiSjg/

That looks like it should fix the problem.

Frank

> On 10/23/2017 10:13 AM, Malahal Naineni wrote:
> > Let us say we have X/Y path in the file system. We have attr_lock and
> > content_lock on each object. The locks we are interested here are
> > attr_lock on X (hear after referred to as AX) and content_lock on X
> > (hear after CX).  Similarly we have AY and CY for object named Y.
> >
> > 1. Thread 50 (lookup called for X ) takes AX and waits for CX (attr_lock
> > followed by content_lock is the expected order)
> >
> > 2. Thread 251 (readdirplus on X) takes CX, AY and then waits for CY for
> > processing object Y
> >
> > 3. Thread 132 (readdirplus on Y) takes CY, and then waits for AX (this
> > is due to lookup of parent)
> >
> > Classic philosopher's problem: 1 waits for 2, 2 waits for 3 and then 3
> > waits for 1. The lock ordering for attr_lock and content_lock for an
> > object is attr_lock followed by content_lock. We can assume that
> > parent locks should be acquired before the child locks, but DOTDOT
> > appears as a child in readdirplus/readdir. If we can handle parent
> > differently, we might be OK. Any help would be appreciated.
> >
> > Regards, Malahal.
> >
> > (gdb) thread 50
> > [Switching to thread 50 (Thread 0x3fff6cffe850 (LWP 37851))]
> > #0  0x3fff8a50089c in .__pthread_rwlock_wrlock () from
> > /lib64/libpthread.so.0
> > (gdb) bt
> > #0  0x3fff8a50089c in .__pthread_rwlock_wrlock () from
> > /lib64/libpthread.so.0
> > #1  0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffccc0541f0,
> > need_acl=false, invalidate=true)
> >  at
> > /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-
> Source/FSAL/Stackab
> > le_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187
> > #2  0x101ae1b0 in mdcache_getattrs (obj_hdl=0x3ffccc054228,
> > attrs_out=0x3fff6cffcfb8)
> >  at
> > /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-
> Source/FSAL/Stackab
> > le_FSALs/FSAL_MDCACHE/mdcache_handle.c:1228
> > #3  0x100b9fdc in nfs_SetPostOpAttr (obj=0x3ffccc054228,
> > Fattr=0x3ffe64050dc8, attrs=0x0)
> >  at
> > /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-
> Source/Protocols/NF
> > S/nfs_proto_tools.c:91
> > #4  0x100c6ba8 in nfs3_lookup (arg=0x3ffa9ff04780,
> > req=0x3ffa9ff03f78, res=0x3ffe64050d50)
> >  at
> > /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-
> Source/Protocols/NF
> > S/nfs3_lookup.c:131
> > #5  0x10065220 in nfs_rpc_execute (reqdata=0x3ffa9ff03f50)
> >  at
> > /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-
> Source/MainNFSD/nfs
> > _worker_thread.c:1290
> > #6  0x10065c9c in worker_run (ctx=0x10013c1d3f0)
> >  at
> > /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-
> Source/MainNFSD/nfs
> > _worker_thread.c:1562
> > #7  0x101670f4 in fridgethr_start_routine (arg=0x10013c1d3f0)
> >  at
> > /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/support/frid
> > gethr.c:550
> > #8  0x3fff8a4fc2bc in .start_thread () from /lib64/libpthread.so.0
> > #9  0x3fff8a31b304 in .__clone () from /lib64/libc.so.6
> > (gdb) frame 1
> > #1  0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffccc0541f0,
> > need_acl=false, invalidate=true)
> >  at
> > /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-
> Source/FSAL/Stackab
> > le_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187
> > 1187PTHREAD_RWLOCK_wrlock(>content_lock);
> > (gdb) p entry
> > $1 = (mdcache_entry_t *) 0x3ffccc0541f0
> > (gdb) p entry->content_lock
> > $2 = {__data = {__lock = 0, __nr_readers = 1, __readers_wakeup = 2408,
> > __writer_wakeup = 4494, __nr_readers_queued = 0,
> >  __nr_writers_queued = 6, __writer = 0, __shared = 0, __pad1 = 0,
> > __pad2 = 0, __flags = 0},
> >__size =
> >
> "\000\000\000\000\000\000\000\001\000\000\th\000\000\021\216\000\000\0
> > 00\000\000\000\000\006", '\000' , __align = 1}
> > (gdb) thread 251
> > [Switching to thread 251 (Thread 0x3fff2e7fe850 (LWP 37976))]
> > #0  0x3fff8a50089c in .__pthread_rwlock_wrlock () from
> > /lib64/libpthread.so.0
> > (gdb) bt
> > #0  0x3fff8a50089c in .__pthread_rwlock_wrlock () from
> > /lib64/libpthread.so.0
> > #1  0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffc30041bd0,
> > need_acl=false, invalidate=true)
> >  at
> > /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-
> Source/FSAL/Stackab
> > le_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187
> > #2  0x101ae1b0 in mdcache_getattrs (obj_hdl=0x3ffc30041c08,
> > attrs_out=0x3fff2e7fcb58)
> >  at
> > /usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-
> Source/FSAL/Stackab
> > le_FSALs/FSAL_MDCACHE/mdcache_handle.c:1228
> > #3  0x101c0f50 in mdcache_readdir_chunked
> > (directory=0x3ffccc0541f0, whence=1298220731,
> >  dir_state=0x3fff2e7fcee8, cb=@0x1024b040: 0x1003f524
> > , attrmask=122830, eod_met=0x3fff2e7fd01c)
> >  at
> > 

Re: [Nfs-ganesha-devel] ABBA deadlock in 2.5 (likely in 2.6 as well)

2017-10-23 Thread Daniel Gryniewicz

Maybe something like this:

https://paste.fedoraproject.org/paste/CptGkmoRutBKYjno5FiSjg/

Daniel

On 10/23/2017 10:13 AM, Malahal Naineni wrote:

Let us say we have X/Y path in the file system. We have attr_lock and
content_lock on each object. The locks we are interested here are
attr_lock on X (hear after referred to as AX) and content_lock on X
(hear after CX).  Similarly we have AY and CY for object named Y.

1. Thread 50 (lookup called for X ) takes AX and waits for CX (attr_lock
followed by content_lock is the expected order)

2. Thread 251 (readdirplus on X) takes CX, AY and then waits for CY for
processing object Y

3. Thread 132 (readdirplus on Y) takes CY, and then waits for AX (this
is due to lookup of parent)

Classic philosopher's problem: 1 waits for 2, 2 waits for 3 and then 3
waits for 1. The lock ordering for attr_lock and content_lock for an
object is attr_lock followed by content_lock. We can assume that parent
locks should be acquired before the child locks, but DOTDOT appears as a
child in readdirplus/readdir. If we can handle parent differently, we
might be OK. Any help would be appreciated.

Regards, Malahal.

(gdb) thread 50
[Switching to thread 50 (Thread 0x3fff6cffe850 (LWP 37851))]
#0  0x3fff8a50089c in .__pthread_rwlock_wrlock () from 
/lib64/libpthread.so.0

(gdb) bt
#0  0x3fff8a50089c in .__pthread_rwlock_wrlock () from 
/lib64/libpthread.so.0
#1  0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffccc0541f0, 
need_acl=false, invalidate=true)
 at 
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187
#2  0x101ae1b0 in mdcache_getattrs (obj_hdl=0x3ffccc054228, 
attrs_out=0x3fff6cffcfb8)
 at 
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1228
#3  0x100b9fdc in nfs_SetPostOpAttr (obj=0x3ffccc054228, 
Fattr=0x3ffe64050dc8, attrs=0x0)
 at 
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/Protocols/NFS/nfs_proto_tools.c:91
#4  0x100c6ba8 in nfs3_lookup (arg=0x3ffa9ff04780, 
req=0x3ffa9ff03f78, res=0x3ffe64050d50)
 at 
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/Protocols/NFS/nfs3_lookup.c:131

#5  0x10065220 in nfs_rpc_execute (reqdata=0x3ffa9ff03f50)
 at 
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/MainNFSD/nfs_worker_thread.c:1290

#6  0x10065c9c in worker_run (ctx=0x10013c1d3f0)
 at 
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/MainNFSD/nfs_worker_thread.c:1562

#7  0x101670f4 in fridgethr_start_routine (arg=0x10013c1d3f0)
 at 
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/support/fridgethr.c:550

#8  0x3fff8a4fc2bc in .start_thread () from /lib64/libpthread.so.0
#9  0x3fff8a31b304 in .__clone () from /lib64/libc.so.6
(gdb) frame 1
#1  0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffccc0541f0, 
need_acl=false, invalidate=true)
 at 
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187

1187PTHREAD_RWLOCK_wrlock(>content_lock);
(gdb) p entry
$1 = (mdcache_entry_t *) 0x3ffccc0541f0
(gdb) p entry->content_lock
$2 = {__data = {__lock = 0, __nr_readers = 1, __readers_wakeup = 2408, 
__writer_wakeup = 4494, __nr_readers_queued = 0,
 __nr_writers_queued = 6, __writer = 0, __shared = 0, __pad1 = 0, 
__pad2 = 0, __flags = 0},
   __size = 
"\000\000\000\000\000\000\000\001\000\000\th\000\000\021\216\000\000\000\000\000\000\000\006", 
'\000' , __align = 1}

(gdb) thread 251
[Switching to thread 251 (Thread 0x3fff2e7fe850 (LWP 37976))]
#0  0x3fff8a50089c in .__pthread_rwlock_wrlock () from 
/lib64/libpthread.so.0

(gdb) bt
#0  0x3fff8a50089c in .__pthread_rwlock_wrlock () from 
/lib64/libpthread.so.0
#1  0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffc30041bd0, 
need_acl=false, invalidate=true)
 at 
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187
#2  0x101ae1b0 in mdcache_getattrs (obj_hdl=0x3ffc30041c08, 
attrs_out=0x3fff2e7fcb58)
 at 
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1228
#3  0x101c0f50 in mdcache_readdir_chunked 
(directory=0x3ffccc0541f0, whence=1298220731,
 dir_state=0x3fff2e7fcee8, cb=@0x1024b040: 0x1003f524 
, attrmask=122830, eod_met=0x3fff2e7fd01c)
 at 
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:3047
#4  0x101ab998 in mdcache_readdir (dir_hdl=0x3ffccc054228, 
whence=0x3fff2e7fcfa0, dir_state=0x3fff2e7fcee8,
 cb=@0x1024b040: 0x1003f524 , attrmask=122830, 
eod_met=0x3fff2e7fd01c)
 at 
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:637
#5  0x10040090 in fsal_readdir 

[Nfs-ganesha-devel] ABBA deadlock in 2.5 (likely in 2.6 as well)

2017-10-23 Thread Malahal Naineni
Let us say we have X/Y path in the file system. We have attr_lock and
content_lock on each object. The locks we are interested here are
attr_lock on X (hear after referred to as AX) and content_lock on X
(hear after CX).  Similarly we have AY and CY for object named Y.

1. Thread 50 (lookup called for X ) takes AX and waits for CX (attr_lock
   followed by content_lock is the expected order)

2. Thread 251 (readdirplus on X) takes CX, AY and then waits for CY for
   processing object Y

3. Thread 132 (readdirplus on Y) takes CY, and then waits for AX (this
   is due to lookup of parent)

Classic philosopher's problem: 1 waits for 2, 2 waits for 3 and then 3
waits for 1. The lock ordering for attr_lock and content_lock for an
object is attr_lock followed by content_lock. We can assume that parent
locks should be acquired before the child locks, but DOTDOT appears as a
child in readdirplus/readdir. If we can handle parent differently, we
might be OK. Any help would be appreciated.

Regards, Malahal.

(gdb) thread 50
[Switching to thread 50 (Thread 0x3fff6cffe850 (LWP 37851))]
#0  0x3fff8a50089c in .__pthread_rwlock_wrlock () from
/lib64/libpthread.so.0
(gdb) bt
#0  0x3fff8a50089c in .__pthread_rwlock_wrlock () from
/lib64/libpthread.so.0
#1  0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffccc0541f0,
need_acl=false, invalidate=true)
at
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187
#2  0x101ae1b0 in mdcache_getattrs (obj_hdl=0x3ffccc054228,
attrs_out=0x3fff6cffcfb8)
at
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1228
#3  0x100b9fdc in nfs_SetPostOpAttr (obj=0x3ffccc054228,
Fattr=0x3ffe64050dc8, attrs=0x0)
at
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/Protocols/NFS/nfs_proto_tools.c:91
#4  0x100c6ba8 in nfs3_lookup (arg=0x3ffa9ff04780,
req=0x3ffa9ff03f78, res=0x3ffe64050d50)
at
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/Protocols/NFS/nfs3_lookup.c:131
#5  0x10065220 in nfs_rpc_execute (reqdata=0x3ffa9ff03f50)
at
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/MainNFSD/nfs_worker_thread.c:1290
#6  0x10065c9c in worker_run (ctx=0x10013c1d3f0)
at
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/MainNFSD/nfs_worker_thread.c:1562
#7  0x101670f4 in fridgethr_start_routine (arg=0x10013c1d3f0)
at
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/support/fridgethr.c:550
#8  0x3fff8a4fc2bc in .start_thread () from /lib64/libpthread.so.0
#9  0x3fff8a31b304 in .__clone () from /lib64/libc.so.6
(gdb) frame 1
#1  0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffccc0541f0,
need_acl=false, invalidate=true)
at
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187
1187 PTHREAD_RWLOCK_wrlock(>content_lock);
(gdb) p entry
$1 = (mdcache_entry_t *) 0x3ffccc0541f0
(gdb) p entry->content_lock
$2 = {__data = {__lock = 0, __nr_readers = 1, __readers_wakeup = 2408,
__writer_wakeup = 4494, __nr_readers_queued = 0,
__nr_writers_queued = 6, __writer = 0, __shared = 0, __pad1 = 0, __pad2
= 0, __flags = 0},
  __size =
"\000\000\000\000\000\000\000\001\000\000\th\000\000\021\216\000\000\000\000\000\000\000\006",
'\000' , __align = 1}
(gdb) thread 251
[Switching to thread 251 (Thread 0x3fff2e7fe850 (LWP 37976))]
#0  0x3fff8a50089c in .__pthread_rwlock_wrlock () from
/lib64/libpthread.so.0
(gdb) bt
#0  0x3fff8a50089c in .__pthread_rwlock_wrlock () from
/lib64/libpthread.so.0
#1  0x101adaa4 in mdcache_refresh_attrs (entry=0x3ffc30041bd0,
need_acl=false, invalidate=true)
at
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1187
#2  0x101ae1b0 in mdcache_getattrs (obj_hdl=0x3ffc30041c08,
attrs_out=0x3fff2e7fcb58)
at
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1228
#3  0x101c0f50 in mdcache_readdir_chunked
(directory=0x3ffccc0541f0, whence=1298220731,
dir_state=0x3fff2e7fcee8, cb=@0x1024b040: 0x1003f524 ,
attrmask=122830, eod_met=0x3fff2e7fd01c)
at
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:3047
#4  0x101ab998 in mdcache_readdir (dir_hdl=0x3ffccc054228,
whence=0x3fff2e7fcfa0, dir_state=0x3fff2e7fcee8,
cb=@0x1024b040: 0x1003f524 , attrmask=122830,
eod_met=0x3fff2e7fd01c)
at
/usr/src/debug/nfs-ganesha-2.5.3-ibm008.20M6-0.1.1-Source/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:637
#5  0x10040090 in fsal_readdir (directory=0x3ffccc054228,
cookie=1298220731, nbfound=0x3fff2e7fd018,
eod_met=0x3fff2e7fd01c, attrmask=122830, cb=@0x10250bb0: 0x100cabf4
,
opaque=0x3fff2e7fd038) at