I did some compares with gpfsperf ... betweend IB and 100GbE .. but we used the 100GbE with ROCE .. so my results might not be representative for you ..
(don't wonder about edited hostnames .. its from a real customer environment..)
so with real data workload.. it is nearly the same... ~ 6 ms
Infiniband
[root@nnnn ~]# mmdsh -N nnnn,nnnn,nnnn,nnnnn \
"tsqosperf read seq /gpfs/sase1/testdev/\$(hostname) -n 100g -r 4m -th 8 -dio" | grep "rate was"
nnnnn Data rate was 5057593.05 Kbytes/sec, Op Rate was 1205.82 Ops/sec, Avg Latency was 6.557 milliseconds, thread utilization 0.988, bytesTransferred 107374182400
nnnnn Data rate was 5046234.10 Kbytes/sec, Op Rate was 1203.12 Ops/sec, Avg Latency was 6.576 milliseconds, thread utilization 0.989, bytesTransferred 107374182400
nnnnn Data rate was 4988625.75 Kbytes/sec, Op Rate was 1189.38 Ops/sec, Avg Latency was 6.557 milliseconds, thread utilization 0.975, bytesTransferred 107374182400
nnnnn Data rate was 4136019.23 Kbytes/sec, Op Rate was 986.10 Ops/sec, Avg Latency was 7.995 milliseconds, thread utilization 0.985, bytesTransferred 10737418240
100GbE RoCE
[root@bb1gssio1 ~]# mmdsh -N c09n1,c09n2,c09n3,c09n4 \
"tsqosperf read seq /gpfs/gpfs0/\$(hostname) -n 100g -r 4m -th 8 -dio" | grep "rate was"
C09n1 Data rate was 5350528.27 Kbytes/sec, Op Rate was 1275.67 Ops/sec, Avg Latency was 6.242 milliseconds, thread utilization 0.995, bytesTransferred 107374182400
C09n2 Data rate was 4964347.14 Kbytes/sec, Op Rate was 1183.59 Ops/sec, Avg Latency was 6.743 milliseconds, thread utilization 0.998, bytesTransferred 107374182400
C09n3 Data rate was 4857516.69 Kbytes/sec, Op Rate was 1158.12 Ops/sec, Avg Latency was 6.893 milliseconds, thread utilization 0.998, bytesTransferred 107374182400
C09n4 Data rate was 4829485.95 Kbytes/sec, Op Rate was 1151.44 Ops/sec, Avg Latency was 6.929 milliseconds, thread utilization 0.997, bytesTransferred 107374182400
with Mellanox ib_read_lat .. the picture looks different.... ~ 2,5 usec versus 4,6 usec ...
Infiniband
[root@nnnnnn ~]# ib_read_lat nnnnnnnnn
---------------------------------------------------------------------------------------
RDMA_Read Latency Test
Dual-port : OFF Device : mlx4_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 1
Mtu : 2048[B]
Link type : IB
Outstand reads : 16
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x68 QPN 0xd751 PSN 0xadfaf8 OUT 0x10 RKey 0x1001411b VAddr 0x007fca31970000
remote address: LID 0x58 QPN 0x09d6 PSN 0x6a6c59 OUT 0x10 RKey 0x081392 VAddr 0x003fff879d0000
---------------------------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec] t_avg[usec] t_stdev[usec] 99% percentile[usec] 99.9% percentile[usec]
2 1000 2.40 18.12 2.46 2.48 0.28 2.57 18.12
---------------------------------------------------------------------------------------
ROCE
---------------------------------------------------------------------------------------
root@dc18n2:~# ib_read_lat 10.10.4.1
---------------------------------------------------------------------------------------
RDMA_Read Latency Test
Dual-port : OFF Device : mlx5_1
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 1
Mtu : 4096[B]
Link type : Ethernet
GID index : 3
Outstand reads : 16
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x0e7e PSN 0x51b972 OUT 0x10 RKey 0x089b6f VAddr 0x007fe80cd11000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:19:02:18
remote address: LID 0000 QPN 0x0d02 PSN 0xdc9761 OUT 0x10 RKey 0x008142 VAddr 0x003fffb24f0000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:10:04:01
---------------------------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec] t_avg[usec] t_stdev[usec] 99% percentile[usec] 99.9% percentile[usec]
2 1000 4.55 7.12 4.61 4.62 0.05 4.82 7.12
---------------------------------------------------------------------------------------
From: Douglas Duckworth <dod2...@med.cornell.edu>
To: <gpfsug-discuss@spectrumscale.org>
Date: 03/06/2018 08:00 PM
Subject: [gpfsug-discuss] 100G RoCEE and Spectrum Scale Performance
Sent by: gpfsug-discuss-boun...@spectrumscale.org
Hi
We are currently running Spectrum Scale over FDR Infiniband. We plan on upgrading to EDR since I have not really encountered documentation saying to abandon the lower-latency advantage found in Infiniband. Our workloads generally benefit from lower latency.
It looks like, ignoring GPFS, EDR still has higher throughput and lower latency when compared to 100G RoCEE.
http://sc16.supercomputing.org/sc-archive/tech_poster/poster_files/post149s2-file3.pdf
However, I wanted to get feedback on how GPFS performs with 100G Ethernet instead of FDR.
Thanks very much!
Doug
Thanks,
Douglas Duckworth, MSc, LFCS
HPC System Administrator
Scientific Computing Unit
Physiology and Biophysics
Weill Cornell Medicine
E: d...@med.cornell.edu
O: 212-746-6305
F: 212-746-8690_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
[root@bb1gssio1 ~]# mmdsh -N c09n1,c09n2,c09n3,c09n4 \
"tsqosperf read seq /gpfs/gpfs0/\$(hostname) -n 100g -r 4m -th 8 -dio" | grep "rate was"
C09n1 Data rate was 5350528.27 Kbytes/sec, Op Rate was 1275.67 Ops/sec, Avg Latency was 6.242 milliseconds, thread utilization 0.995, bytesTransferred 107374182400
C09n2 Data rate was 4964347.14 Kbytes/sec, Op Rate was 1183.59 Ops/sec, Avg Latency was 6.743 milliseconds, thread utilization 0.998, bytesTransferred 107374182400
C09n3 Data rate was 4857516.69 Kbytes/sec, Op Rate was 1158.12 Ops/sec, Avg Latency was 6.893 milliseconds, thread utilization 0.998, bytesTransferred 107374182400
C09n4 Data rate was 4829485.95 Kbytes/sec, Op Rate was 1151.44 Ops/sec, Avg Latency was 6.929 milliseconds, thread utilization 0.997, bytesTransferred 107374182400
with Mellanox ib_read_lat .. the picture looks different.... ~ 2,5 usec versus 4,6 usec ...
Infiniband
[root@nnnnnn ~]# ib_read_lat nnnnnnnnn
---------------------------------------------------------------------------------------
RDMA_Read Latency Test
Dual-port : OFF Device : mlx4_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 1
Mtu : 2048[B]
Link type : IB
Outstand reads : 16
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x68 QPN 0xd751 PSN 0xadfaf8 OUT 0x10 RKey 0x1001411b VAddr 0x007fca31970000
remote address: LID 0x58 QPN 0x09d6 PSN 0x6a6c59 OUT 0x10 RKey 0x081392 VAddr 0x003fff879d0000
---------------------------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec] t_avg[usec] t_stdev[usec] 99% percentile[usec] 99.9% percentile[usec]
2 1000 2.40 18.12 2.46 2.48 0.28 2.57 18.12
---------------------------------------------------------------------------------------
ROCE
---------------------------------------------------------------------------------------
root@dc18n2:~# ib_read_lat 10.10.4.1
---------------------------------------------------------------------------------------
RDMA_Read Latency Test
Dual-port : OFF Device : mlx5_1
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
TX depth : 1
Mtu : 4096[B]
Link type : Ethernet
GID index : 3
Outstand reads : 16
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x0e7e PSN 0x51b972 OUT 0x10 RKey 0x089b6f VAddr 0x007fe80cd11000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:172:19:02:18
remote address: LID 0000 QPN 0x0d02 PSN 0xdc9761 OUT 0x10 RKey 0x008142 VAddr 0x003fffb24f0000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:10:04:01
---------------------------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec] t_avg[usec] t_stdev[usec] 99% percentile[usec] 99.9% percentile[usec]
2 1000 4.55 7.12 4.61 4.62 0.05 4.82 7.12
---------------------------------------------------------------------------------------
From: Douglas Duckworth <dod2...@med.cornell.edu>
To: <gpfsug-discuss@spectrumscale.org>
Date: 03/06/2018 08:00 PM
Subject: [gpfsug-discuss] 100G RoCEE and Spectrum Scale Performance
Sent by: gpfsug-discuss-boun...@spectrumscale.org
Hi
We are currently running Spectrum Scale over FDR Infiniband. We plan on upgrading to EDR since I have not really encountered documentation saying to abandon the lower-latency advantage found in Infiniband. Our workloads generally benefit from lower latency.
It looks like, ignoring GPFS, EDR still has higher throughput and lower latency when compared to 100G RoCEE.
http://sc16.supercomputing.org/sc-archive/tech_poster/poster_files/post149s2-file3.pdf
However, I wanted to get feedback on how GPFS performs with 100G Ethernet instead of FDR.
Thanks very much!
Doug
Thanks,
Douglas Duckworth, MSc, LFCS
HPC System Administrator
Scientific Computing Unit
Physiology and Biophysics
Weill Cornell Medicine
E: d...@med.cornell.edu
O: 212-746-6305
F: 212-746-8690_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
