Just a few thoughts: We've often increased the seqDiscardThreshold in SAS on AIX GPFS implementations to much larger values ( a lot of large file sequential I/O). With a pagepool of 64G, maybe set to 4 or 8GB?
Sequential writes from the application when spread across mutliple LUNs on the storage+Scale often do not resemble sequential writes from the storage POV. If overrunning a disk device representation in the OS, the device queue depth is one parameter that might be expanded to offer some relief. --- Glen Corneau Senior, Power Partner Technical Specialist (PTS-P) IBM Technology, North America Email: [email protected] Cell: 512-420-7988 ________________________________ From: gpfsug-discuss <[email protected]> on behalf of Michal Hruška <[email protected]> Sent: Thursday, February 8, 2024 08:59 To: [email protected] <[email protected]> Subject: [EXTERNAL] Re: [gpfsug-discuss] sequential I/O write - performance @Aaron Yes, I can confirm that 2MB blocks are transfered over. @ Jan-Frode We tried to change multiple parameters, but if you know the best combination for sequential IO, please let me know. #mmlsconfig autoload no dmapiFileHandleSize 32 minReleaseLevel ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/PjiDSg!2e-hQ74-QNlQ0KX70gHlDsWODTjDLcgSEK8lhkIKcCimb6m97aqUeYtL-5luaJl8omKubeXu1XqHpskHhQfv3H_4lomxcIKH$> Report Suspicious ZjQcmQRYFpfptBannerEnd @Aaron Yes, I can confirm that 2MB blocks are transfered over. @ Jan-Frode We tried to change multiple parameters, but if you know the best combination for sequential IO, please let me know. #mmlsconfig autoload no dmapiFileHandleSize 32 minReleaseLevel 5.1.9.0 tscCmdAllowRemoteConnections no ccrEnabled yes cipherList AUTHONLY sdrNotifyAuthEnabled yes pagepool 64G maxblocksize 16384K maxMBpS 40000 maxReceiverThreads 32 nsdMaxWorkerThreads 512 nsdMinWorkerThreads 8 nsdMultiQueue 256 nsdSmallThreadRatio 0 nsdThreadsPerQueue 3 prefetchAggressiveness 2 adminMode central /dev/fs0 @Uwe Using iohist we found out that gpfs is overloading one dm-device (it took about 500ms to finish IOs). We replaced the „problematic“ dm-device (as we have enough drives to play with) for new one but the overloading issue just jumped to another dm-device. We believe that this behaviour is caused by the gpfs but we are unable to locate the root cause of it. Best, Michal
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at gpfsug.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss_gpfsug.org
