On September 14, 2017 2:26:13 PM GMT+02:00, Dirk Willems <dirk.will...@exitas.be> wrote: >Hello, > > >I'm trying to understand something let me explain. > > >Oracle always told to me that if you create a etherstub switch it has >infiniband speed 40GB/s. > >But I have a customer running on Solaris (Yeah I know but let me >explain) who is copy from 1 NGZ to another NGZ on the same GZ over Lan >(I know told him to to use etherstub). > >The copy witch is performed for a Oracle database with sql command, the > >DBA witch have 5 streams say it's waiting on the disk, the disk are 50 >- >60 % busy the speed is 30 mb/s. > > >So I did some test just to see and understand if it's the database or >the system, but with doing my tests I get very confused ??? > > >On another Solaris at my work copy over etherstub switch => copy speed >is 185MB/s expected much more of infiniband speed ??? > > >root@test1:/export/home/Admin# scp test10G >Admin@192.168.1.2:/export/home/Admin/ >Password: >test10G 100% >|****************************************************************| >10240 >MB 00:59 > > >root@test2:~# dlstat -i 2 > > LINK IPKTS RBYTES OPKTS OBYTES > net1 25.76K 185.14M 10.08K 2.62M > net1 27.04K 187.16M 11.23K 3.22M > net1 26.97K 186.37M 11.24K 3.23M > net1 26.63K 187.67M 10.82K 2.99M > net1 27.94K 186.65M 12.17K 3.75M > net1 27.45K 187.46M 11.70K 3.47M > net1 26.01K 181.95M 10.63K 2.99M > net1 27.95K 188.19M 12.14K 3.69M > net1 27.91K 188.36M 12.08K 3.64M > >The disks are all separate luns with all separated pools => disk are 20 > >- 30% busy > > >On my OmniOSce at my lab over etherstub > > >root@GNUHealth:~# scp test10G witte@192.168.20.3:/export/home/witte/ >Password: >test10G 76% 7853MB 116.4MB/s > > >=> copy is 116.4 MB/s => expected much more from infiniband speed is >just the same as Lan ??? > > >Is not that my disk can not follow 17% busy there sleeping ... > > extended device statistics > r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device > 0,0 248,4 0,0 2,1 0,0 1,3 0,0 5,3 0 102 c1 > 0,0 37,5 0,0 0,7 0,0 0,2 0,0 4,7 0 17 c1t0d0 => >rpool > 0,0 38,5 0,0 0,7 0,0 0,2 0,0 4,9 0 17 c1t1d0 => >rpool > 0,0 40,5 0,0 0,1 0,0 0,2 0,0 5,6 0 17 c1t2d0 => >data pool > 0,0 43,5 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t3d0 => >data pool > 0,0 44,5 0,0 0,2 0,0 0,2 0,0 5,5 0 18 c1t4d0 => >data pool > 0,0 44,0 0,0 0,2 0,0 0,2 0,0 5,4 0 17 c1t5d0 => >data pool > 0,0 76,0 0,0 1,5 7,4 0,4 97,2 4,9 14 18 rpool > 0,0 172,4 0,0 0,6 2,0 0,9 11,4 5,5 12 20 DATA > > > >root@NGINX:/root# dlstat show-link NGINX1 -i 2 > > LINK TYPE ID INDEX PKTS BYTES > NGINX1 rx bcast -- 0 0 > NGINX1 rx sw -- 0 0 > NGINX1 tx bcast -- 0 0 > NGINX1 tx sw -- 9.26K 692.00K > NGINX1 rx local -- 26.00K 216.32M > NGINX1 rx bcast -- 0 0 > NGINX1 rx sw -- 0 0 > NGINX1 tx bcast -- 0 0 > NGINX1 tx sw -- 7.01K 531.38K > NGINX1 rx local -- 30.65K 253.73M > NGINX1 rx bcast -- 0 0 > NGINX1 rx sw -- 0 0 > NGINX1 tx bcast -- 0 0 > NGINX1 tx sw -- 8.95K 669.32K > NGINX1 rx local -- 29.10K 241.15M > > >On the other NGZ I receive 250MB/s ???? > > >- So my question is how comes that the speed is equal to Lan 100MB/s on > >OmniOSce but i receive 250MB/s ? > >- Why is etherstub so slow if infiniband speed is 40GB/s ??? > > >I'm very confused right now ... > > >And want to know for sure how to understand and see this in the right >way, because this customer will be the first customer from my who gonna > >switch complety over to OmniOSce on production and because this >customer >is one or the biggest company's in Belgium I really don't want to mess >up !!! > > >So any help and clarification will be highly appreciate !!! > > >Thank you very much. > > >Kind Regards, > > >Dirk
I am not sure where the infiniband claim comes from, but copying data disk to disk, you involve the slow layers like disk, skewed by faster layers like cache of already-read data and delayed writes :) If you have a wide pipe that you may fill, it doesn't mean you do have the means to fill it with a few disks. To estimate the speeds, try pure UDP streams from process to process (no disk), large-packet floodping, etc. I believe etherstub is not constrained artificially, and defaults to jumbo frames. Going to LAN and back can in fact use external hardware (IIRC there may be a system option to disable that, not sure) and so is constrained by that. Jim -- Typos courtesy of K-9 Mail on my Android _______________________________________________ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
