Hi, >>I was wondering if there are any good performance sizing guides for a spectrum scale shared nothing architecture (FPO)? >> I don't have any production experience using spectrum scale in a "shared nothing configuration " and was hoping for bandwidth / throughput sizing guidance.
Please ensure that all the recommended FPO settings (e.g. allowWriteAffinity=yes in the FPO storage pool, readReplicaPolicy=local, restripeOnDiskFailure=yes) are set properly. Please find the FPO Best practices/tunings, in the links below: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Big%20Data%20Best%20practices https://www.ibm.com/developerworks/community/wikis/form/anonymous/api/wiki/fa32927c-e904-49cc-a4cc-870bcc8e307c/page/ab5c2792-feef-4a3a-a21b-d22c6f5d728a/attachment/80d5c300-7b39-4d6e-9596-84934fcc4638/media/Deploying_a_big_data_solution_using_IBM_Spectrum_Scale_v1.7.5.pdf >> For example, each node might consist of 24x storage drives (locally attached JBOD, no RAID array). >> Given a particular node configuration I want to be in a position to calculate the maximum bandwidth / throughput. With FPO, GPFS metadata (-m) and data replication (-r) needs to be enabled. The Write-affinity-Depth (WAD) setting defines the policy for directing writes. It indicates that the node writing the data directs the write to disks on its own node for the first copy and to the disks on other nodes for the second and third copies (if specified). readReplicaPolicy=local will enable the policy to read replicas from local disks. At the minimum, ensure that the networking used for GPFS is sized properly and has bandwidth 2X or 3X that of the local disk speeds to ensure FPO write bandwidth is not being constrained by GPFS replication over the network. For example, if 24 x Drives in RAID-0 results in ~4.8 GB/s (assuming ~200MB/s per drive) and GPFS metadata/data replication is set to 3 (-m 3 -r 3) then for optimal FPO write bandwidth, we need to ensure the network-interconnect between the FPO nodes is non-blocking/high-speed and can sustain ~14.4 GB/s ( data_replication_factor * local_storage_bandwidth). One possibility, is minimum of 2 x EDR Infiniband (configure GPFS verbsRdma/verbsPorts) or bonded 40GigE between the FPO nodes (for GPFS daemon-to-daemon communication). Application reads requiring FPO reads from remote GPFS node would as well benefit from high-speed network-interconnect between the FPO nodes. Regards, -Kums From: Evan Koutsandreou <evan.koutsandr...@adventone.com> To: "gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org> Date: 08/20/2017 11:06 PM Subject: [gpfsug-discuss] Shared nothing (FPO) throughout / bandwidth sizing Sent by: gpfsug-discuss-boun...@spectrumscale.org Hi - I was wondering if there are any good performance sizing guides for a spectrum scale shared nothing architecture (FPO)? For example, each node might consist of 24x storage drives (locally attached JBOD, no RAID array). I don't have any production experience using spectrum scale in a "shared nothing configuration " and was hoping for bandwidth / throughput sizing guidance. Given a particular node configuration I want to be in a position to calculate the maximum bandwidth / throughput. Thank you _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss