Some other things you need to take into account:
QDR Infiniband is 40Gbps, not 40GB/s. That is a factor of 8
difference. That is also a theoretical maximum throughput, there
is some overhead. In reality, you will never see 40Gbps.
My system tested out at 6Gbps - 8Gbps using NFS over IPoIB, with
DDR (20Gbps) nodes and a QDR (40Gbps) storage server. IPoIB drops
the theoretical max rates to 18Gbps and 36Gbps respectively.
If you are getting 185MB/s, you are seeing 1.48Gbps.
Keep your B's and b's straight. Did you play with your frame size
at all?
Ian
On Thu, Sep 14, 2017 at 7:10 AM, Jim Klimov <jimkli...@cos.ru
<mailto:jimkli...@cos.ru>> wrote:
On September 14, 2017 2:26:13 PM GMT+02:00, Dirk Willems
<dirk.will...@exitas.be <mailto:dirk.will...@exitas.be>> wrote:
>Hello,
>
>
>I'm trying to understand something let me explain.
>
>
>Oracle always told to me that if you create a etherstub
switch it has
>infiniband speed 40GB/s.
>
>But I have a customer running on Solaris (Yeah I know but let me
>explain) who is copy from 1 NGZ to another NGZ on the same
GZ over Lan
>(I know told him to to use etherstub).
>
>The copy witch is performed for a Oracle database with sql
command, the
>
>DBA witch have 5 streams say it's waiting on the disk, the
disk are 50
>-
>60 % busy the speed is 30 mb/s.
>
>
>So I did some test just to see and understand if it's the
database or
>the system, but with doing my tests I get very confused ???
>
>
>On another Solaris at my work copy over etherstub switch =>
copy speed
>is 185MB/s expected much more of infiniband speed ???
>
>
>root@test1:/export/home/Admin# scp test10G
>Admin@192.168.1.2:/export/
<mailto:Admin@192.168.1.2:/export/>home/Admin/
>Password:
>test10G 100%
>|****************************************************************|
>10240
>MB 00:59
>
>
>root@test2:~# dlstat -i 2
>
> LINK IPKTS RBYTES OPKTS OBYTES
> net1 25.76K 185.14M 10.08K 2.62M
> net1 27.04K 187.16M 11.23K 3.22M
> net1 26.97K 186.37M 11.24K 3.23M
> net1 26.63K 187.67M 10.82K 2.99M
> net1 27.94K 186.65M 12.17K 3.75M
> net1 27.45K 187.46M 11.70K 3.47M
> net1 26.01K 181.95M 10.63K 2.99M
> net1 27.95K 188.19M 12.14K 3.69M
> net1 27.91K 188.36M 12.08K 3.64M
>
>The disks are all separate luns with all separated pools =>
disk are 20
>
>- 30% busy
>
>
>On my OmniOSce at my lab over etherstub
>
>
>root@GNUHealth:~# scp test10G witte@192.168.20.3:/export/
<mailto:witte@192.168.20.3:/export/>home/witte/
>Password:
>test10G 76% 7853MB 116.4MB/s
>
>
>=> copy is 116.4 MB/s => expected much more from infiniband
speed is
>just the same as Lan ???
>
>
>Is not that my disk can not follow 17% busy there sleeping ...
>
> extended device statistics
> r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w
%b device
> 0,0 248,4 0,0 2,1 0,0 1,3 0,0 5,3 0 102 c1
> 0,0 37,5 0,0 0,7 0,0 0,2 0,0 4,7 0 17
c1t0d0 =>
>rpool
> 0,0 38,5 0,0 0,7 0,0 0,2 0,0 4,9 0 17
c1t1d0 =>
>rpool
> 0,0 40,5 0,0 0,1 0,0 0,2 0,0 5,6 0 17
c1t2d0 =>
>data pool
> 0,0 43,5 0,0 0,2 0,0 0,2 0,0 5,4 0 17
c1t3d0 =>
>data pool
> 0,0 44,5 0,0 0,2 0,0 0,2 0,0 5,5 0 18
c1t4d0 =>
>data pool
> 0,0 44,0 0,0 0,2 0,0 0,2 0,0 5,4 0 17
c1t5d0 =>
>data pool
> 0,0 76,0 0,0 1,5 7,4 0,4 97,2 4,9 14 18
rpool
> 0,0 172,4 0,0 0,6 2,0 0,9 11,4 5,5 12 20
DATA
>
>
>
>root@NGINX:/root# dlstat show-link NGINX1 -i 2
>
> LINK TYPE ID INDEX PKTS BYTES
> NGINX1 rx bcast -- 0 0
> NGINX1 rx sw -- 0 0
> NGINX1 tx bcast -- 0 0
> NGINX1 tx sw -- 9.26K 692.00K
> NGINX1 rx local -- 26.00K 216.32M
> NGINX1 rx bcast -- 0 0
> NGINX1 rx sw -- 0 0
> NGINX1 tx bcast -- 0 0
> NGINX1 tx sw -- 7.01K 531.38K
> NGINX1 rx local -- 30.65K 253.73M
> NGINX1 rx bcast -- 0 0
> NGINX1 rx sw -- 0 0
> NGINX1 tx bcast -- 0 0
> NGINX1 tx sw -- 8.95K 669.32K
> NGINX1 rx local -- 29.10K 241.15M
>
>
>On the other NGZ I receive 250MB/s ????
>
>
>- So my question is how comes that the speed is equal to Lan
100MB/s on
>
>OmniOSce but i receive 250MB/s ?
>
>- Why is etherstub so slow if infiniband speed is 40GB/s ???
>
>
>I'm very confused right now ...
>
>
>And want to know for sure how to understand and see this in
the right
>way, because this customer will be the first customer from
my who gonna
>
>switch complety over to OmniOSce on production and because this
>customer
>is one or the biggest company's in Belgium I really don't
want to mess
>up !!!
>
>
>So any help and clarification will be highly appreciate !!!
>
>
>Thank you very much.
>
>
>Kind Regards,
>
>
>Dirk
I am not sure where the infiniband claim comes from, but
copying data disk to disk, you involve the slow layers like
disk, skewed by faster layers like cache of already-read data
and delayed writes :)
If you have a wide pipe that you may fill, it doesn't mean
you do have the means to fill it with a few disks.
To estimate the speeds, try pure UDP streams from process to
process (no disk), large-packet floodping, etc.
I believe etherstub is not constrained artificially, and
defaults to jumbo frames. Going to LAN and back can in fact
use external hardware (IIRC there may be a system option to
disable that, not sure) and so is constrained by that.
Jim
--
Typos courtesy of K-9 Mail on my Android
_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
<mailto:OmniOS-discuss@lists.omniti.com>
http://lists.omniti.com/mailman/listinfo/omnios-discuss
<http://lists.omniti.com/mailman/listinfo/omnios-discuss>
--
Ian Kaufman
Research Systems Administrator
UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu
_______________________________________________
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
<mailto:OmniOS-discuss@lists.omniti.com>
http://lists.omniti.com/mailman/listinfo/omnios-discuss
<http://lists.omniti.com/mailman/listinfo/omnios-discuss>
