Hi Sven,
Which NSD server should I run top on, the fs manager? If so the CPU load
is about 155%. I'm working on perf top but not off to a great start...
# perf top
PerfTop: 1095 irqs/sec kernel:61.9% exact: 0.0% [1000Hz
cycles], (all, 28 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Segmentation fault
-Aaron
On 3/24/17 1:04 PM, Sven Oehme wrote:
while this is happening run top and see if there is very high cpu
utilization at this time on the NSD Server.
if there is , run perf top (you might need to install perf command) and
see if the top cpu contender is a spinlock . if so send a screenshot of
perf top as i may know what that is and how to fix.
sven
On Fri, Mar 24, 2017 at 9:43 AM Aaron Knister <[email protected]
<mailto:[email protected]>> wrote:
Since yesterday morning we've noticed some deadlocks on one of our
filesystems that seem to be triggered by writing to it. The waiters on
the clients look like this:
0x19450B0 ( 6730) waiting 2063.294589599 seconds, SyncHandlerThread:
on ThCond 0x1802585CB10 (0xFFFFC9002585CB10) (InodeFlushCondVar), reason
'waiting for the flush flag to commit metadata'
0x7FFFDA65E200 ( 22850) waiting 0.000246257 seconds,
AllocReduceHelperThread: on ThCond 0x7FFFDAC7FE28 (0x7FFFDAC7FE28)
(MsgRecordCondvar), reason 'RPC wait' for allocMsgTypeRelinquishRegion
on node 10.1.52.33 <c0n3271>
0x197EE70 ( 6776) waiting 0.000198354 seconds,
FileBlockWriteFetchHandlerThread: on ThCond 0x7FFFF00CD598
(0x7FFFF00CD598) (MsgRecordCondvar), reason 'RPC wait' for
allocMsgTypeRequestRegion on node 10.1.52.33 <c0n3271>
(10.1.52.33/c0n3271 <http://10.1.52.33/c0n3271> is the fs manager
for the filesystem in question)
there's a single process running on this node writing to the filesystem
in question (well, trying to write, it's been blocked doing nothing for
half an hour now). There are ~10 other client nodes in this situation
right now. We had many more last night before the problem seemed to
disappear in the early hours of the morning and now its back.
Waiters on the fs manager look like this. While the individual waiter is
short it's a near constant stream:
0x7FFF60003540 ( 8269) waiting 0.001151588 seconds, Msg handler
allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0)
(AllocManagerMutex)
0x7FFF601C8860 ( 20606) waiting 0.001115712 seconds, Msg handler
allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0
(0xFFFFC9002163A2E0) (AllocManagerMutex)
0x7FFF91C10080 ( 14723) waiting 0.000959649 seconds, Msg handler
allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0)
(AllocManagerMutex)
0x7FFFB03C2910 ( 12636) waiting 0.000769611 seconds, Msg handler
allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0)
(AllocManagerMutex)
0x7FFF8C092850 ( 18215) waiting 0.000682275 seconds, Msg handler
allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0
(0xFFFFC9002163A2E0) (AllocManagerMutex)
0x7FFF9423F730 ( 12652) waiting 0.000641915 seconds, Msg handler
allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0)
(AllocManagerMutex)
0x7FFF9422D770 ( 12625) waiting 0.000494256 seconds, Msg handler
allocMsgTypeRequestRegion: on ThMutex 0x1802163A2E0 (0xFFFFC9002163A2E0)
(AllocManagerMutex)
0x7FFF9423E310 ( 12651) waiting 0.000437760 seconds, Msg handler
allocMsgTypeRelinquishRegion: on ThMutex 0x1802163A2E0
(0xFFFFC9002163A2E0) (AllocManagerMutex)
I don't know if this data point is useful but both yesterday and today
the metadata NSDs for this filesystem have had a constant aggregate
stream of 25MB/s 4kop/s reads during each episode (very low latency
though so I don't believe the storage is a bottleneck here). Writes are
only a few hundred ops and didn't strike me as odd.
I have a PMR open for this but I'm curious if folks have seen this in
the wild and what it might mean.
-Aaron
--
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776 <tel:(301)%20286-2776>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
--
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
