It absolutely does, thanks Olaf!
The tasks running on these nodes are running on 63 other nodes and
generating ~60K iop/s of metadata writes and I *think* about the same in
reads. Do you think that could be contributing to the higher waiter
times? I'm not sure quite what the job is up to. It's seemingly doing
very little data movement, the cpu %used is very low but the load is
rather high.
-Aaron
On 10/15/16 11:23 AM, Olaf Weiser wrote:
from your file system configuration .. mmfs <dev> -L you'll find the
size of the LOG
since release 4.x ..you can change it, but you need to re-mount the FS
on every client , to make the change effective ...
when a clients initiate writes/changes to GPFS it needs to update its
changes to the log - if this narrows a certain filling degree, GPFS
triggers so called logWrapThreads to write content to disk and so free
space
with your given numbers ... double digit [ms] waiter times .. you fs
get's probably slowed down.. and there's something suspect with the
storage, because LOG-IOs are rather small and should not take that long
to give you an example from a healthy environment... the IO times are so
small, that you usually don't see waiters for this..
I/O start time RW Buf type disk:sectorNum nSec time ms tag1
tag2 Disk UID typ NSD node context thread
--------------- -- ----------- ----------------- ----- -------
--------- --------- ------------------ --- --------------- ---------
----------
06:23:32.358851 W logData 2:524306424 8 0.439
0 0 C0A70D08:57CF40D1 cli 192.167.20.17 LogData
SGExceptionLogBufferFullThread
06:23:33.576367 W logData 1:524257280 8 0.646
0 0 C0A70D08:57CF40D0 cli 192.167.20.16 LogData
SGExceptionLogBufferFullThread
06:23:32.358851 W logData 2:524306424 8 0.439
0 0 C0A70D08:57CF40D1 cli 192.167.20.17 LogData
SGExceptionLogBufferFullThread
06:23:33.576367 W logData 1:524257280 8 0.646
0 0 C0A70D08:57CF40D0 cli 192.167.20.16 LogData
SGExceptionLogBufferFullThread
06:23:32.212426 W iallocSeg 1:524490048 64 0.733
2 245 C0A70D08:57CF40D0 cli 192.167.20.16 Logwrap
LogWrapHelperThread
06:23:32.212412 W logWrap 2:524552192 8 0.755
0 179200 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap
LogWrapHelperThread
06:23:32.212432 W logWrap 2:525162760 8 0.737
0 125473 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap
LogWrapHelperThread
06:23:32.212416 W iallocSeg 2:524488384 64 0.763
2 347 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap
LogWrapHelperThread
06:23:32.212414 W logWrap 2:525266944 8 2.160
0 177664 C0A70D08:57CF40D1 cli 192.167.20.17 Logwrap
LogWrapHelperThread
hope this helps ..
Mit freundlichen Grüßen / Kind regards
Olaf Weiser
EMEA Storage Competence Center Mainz, German / IBM Systems, Storage
Platform,
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland
IBM Allee 1
71139 Ehningen
Phone: +49-170-579-44-66
E-Mail: olaf.wei...@de.ibm.com
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter
Geschäftsführung: Martina Koederitz (Vorsitzende), Susanne Peter,
Norbert Janzen, Dr. Christian Keller, Ivo Koerner, Markus Koerner
Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht
Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940
From: Aaron Knister <aaron.s.knis...@nasa.gov>
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>
Date: 10/15/2016 07:23 AM
Subject: [gpfsug-discuss] SGExceptionLogBufferFullThread waiter
Sent by: gpfsug-discuss-boun...@spectrumscale.org
------------------------------------------------------------------------
I've got a node that's got some curious waiters on it (see below). Could
someone explain what the "SGExceptionLogBufferFullThread" waiter means?
Thanks!
-Aaron
=== mmdiag: waiters ===
0x7FFFF040D600 waiting 0.038822715 seconds,
SGExceptionLogBufferFullThread: on ThCond 0x7FFFDBB07628
(0x7FFFDBB07628) (parallelWaitCond), reason 'wait for parallel write'
for NSD I/O completion on node 10.1.53.5 <c0n20>
0x7FFFE83F3D60 waiting 0.039629116 seconds, CleanBufferThread: on ThCond
0x17B1488 (0x17B1488) (MsgRecordCondvar), reason 'RPC wait' for NSD I/O
completion on node 10.1.53.7 <c0n22>
0x7FFFE8373A90 waiting 0.038921480 seconds, CleanBufferThread: on ThCond
0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar), reason
'force wait on force active buffer write'
0x42CD9B0 waiting 0.028227004 seconds, CleanBufferThread: on ThCond
0x7FFFCD2B4E30 (0x7FFFCD2B4E30) (LogFileBufferDescriptorCondvar), reason
'force wait for buffer write to complete'
0x7FFFE0F0EAD0 waiting 0.027864343 seconds, CleanBufferThread: on ThCond
0x7FFFDC0EEA88 (0x7FFFDC0EEA88) (MsgRecordCondvar), reason 'RPC wait'
for NSD I/O completion on node 10.1.53.7 <c0n22>
0x1575560 waiting 0.028045975 seconds, RemoveHandlerThread: on ThCond
0x18020CE4E08 (0xFFFFC90020CE4E08) (LkObjCondvar), reason 'waiting for
LX lock'
0x1570560 waiting 0.038724949 seconds, CreateHandlerThread: on ThCond
0x18020CE50A0 (0xFFFFC90020CE50A0) (LkObjCondvar), reason 'waiting for
LX lock'
0x1563D60 waiting 0.073919918 seconds, RemoveHandlerThread: on ThCond
0x180235F6440 (0xFFFFC900235F6440) (LkObjCondvar), reason 'waiting for
LX lock'
0x1561560 waiting 0.054854513 seconds, RemoveHandlerThread: on ThCond
0x1802292D200 (0xFFFFC9002292D200) (LkObjCondvar), reason 'waiting for
LX lock'
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
