Hi Wido, What is the full topology? Are you using a north-south or east-west? So far I've seen the east-west are slightly slower. What are the fabric modes you have configured? How is everything connected? Also you have no information on the OS - if I remember correctly there was a lot of improvements in the latest kernels...
And what about the bandwith? The values you present don't seem awfully high, and the deviation seems low. On Thu, Nov 6, 2014 at 1:18 PM, Wido den Hollander <[email protected]> wrote: > Hello, > > While working at a customer I've ran into a 10GbE latency which seems > high to me. > > I have access to a couple of Ceph cluster and I ran a simple ping test: > > $ ping -s 8192 -c 100 -n <ip> > > Two results I got: > > rtt min/avg/max/mdev = 0.080/0.131/0.235/0.039 ms > rtt min/avg/max/mdev = 0.128/0.168/0.226/0.023 ms > > Both these environment are running with Intel 82599ES 10Gbit cards in > LACP. One with Extreme Networks switches, the other with Arista. > > Now, on a environment with Cisco Nexus 3000 and Nexus 7000 switches I'm > seeing: > > rtt min/avg/max/mdev = 0.160/0.244/0.298/0.029 ms > > As you can see, the Cisco Nexus network has high latency compared to the > other setup. > > You would say the switches are to blame, but we also tried with a direct > TwinAx connection, but that didn't help. > > This setup also uses the Intel 82599ES cards, so the cards don't seem to > be the problem. > > The MTU is set to 9000 on all these networks and cards. > > I was wondering, others with a Ceph cluster running on 10GbE, could you > perform a simple network latency test like this? I'd like to compare the > results. > > -- > Wido den Hollander > 42on B.V. > Ceph trainer and consultant > > Phone: +31 (0)20 700 9902 > Skype: contact42on > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
