Actually the 1 us “penalty” should be reduced — my measurements involved a proxy moving data between two separate TCP streams, so that penalty is for both tx and rx. I am not sure how much of that penalty is RX and how much is TX, but instinct leads me to suspect its mostly on the RX side.
On Tue, May 3, 2016 at 1:29 PM, Garrett D'Amore <garr...@damore.org> wrote: > Reading the driver for e1000g, it appears that this hardware (or the > driver at any rate) only supports a single receive ring. This means that > its impossible to spread interrupt loading, because you don’t have multiple > rings. (Put another way — in Windows friendly parlance, it appears that > this hardware does not support RSS.) > > Intel hardware using igb does support RSS, but the e1000g seems a bit > more budget minded. > > However, we’re talking about 1Gbps hardware. A single CPU should have no > problem keeping up with this. Even the context switches to service threads > run on separate CPUs are not going to have much impact. (In my testing, > running on the “wrong” CPU generally added on the order of about a > microsecond of latency to the frames where it occurred. I doubt you’re > using tiny frames — if your average frame size is 500 bytes, then you’re > going to peak at 250K pps (1Gbps limit), or spend about 4 us per packet. > An extra 1 us would definitely be measurable, but it would not have > anything like the impact you show above. The results are far too > devastating to be the result of lack of RSS. > > Now, having said that, when you say through put, what are you measuring? > Is it packets per second out of the interface? Or into the interface? Or > something somewhere else? Are these “vms” using kvm or are they rather > native or lx branded zones? (If the former, I wouldn’t be at all surprised > if you are simply running into other resource limitations, as kvm has a > much higher per-zone impact. It also would not surprise me in the least if > it is shown that there are bottlenecks or hot locks inside the kvm code. > > The other thing is that because kvm guests run with a whole machine > emulation, as they fight for available CPU their NIC performance is going > to drop noticeably. This means also that a busy process in one vm can have > a negative bandwidth impact on other vms, even if that first vm is *not* > sending or receiving any data. (Note that this is due to the hardware > being emulated in software. Native zones and LX shouldn’t suffer from this > at all, since they access real hardware and any kernel side code run on > their behalf runs at higher priority than user space code.) > > One thing I’d look at during this is to determine what your system is > doing. Richard already proposed looking at intrstat. You could also use > powertop. Personally, my gut instinct (not backed by any data, note) is > that you maybe suffering from lock contention . So I’d do a little poking > with lockstat to look for hot locks. (As I indicated, I would not be at > all surprised to learn that kvm has some very hot locks in it. But as I > said, you could be fighting purely for CPU in the kvm instances, and that > won’t necessarily show up as a contended lock.) > > - Garrett > > On Tue, May 3, 2016 at 10:16 AM, Stefan <ste...@squentz.de> wrote: > >> Dear List, >> >> on our machines with about 40 vservers we observe low network >> throughput. It seems to scale inversely with the number of vms on the >> respective node: >> >> # vms bw (Mbps) >> ----- --------- >> 37 60 >> 24 94 >> 18 174 >> 12 608 >> 5 933 >> 2 934 >> 1 935 >> >> The measurements were obtained using iperf from a different physical >> machine. It has been suggested that the slowdown may be due to >> interrupt 51 being clamped to CPU 12: >> >> # echo '::interrupts -d' | mdb -k >> IRQ Vect IPL Bus Trg Type CPU Share APIC/INT# Driver Name(s) >> : >> 49 0x40 5 PCI Edg MSI 8 1 - mpt#0 >> 50 0x60 6 PCI Edg MSI 11 1 - e1000g#0 >> 51 0x61 6 PCI Edg MSI 12 1 - e1000g#1 >> 160 0xa0 0 Edg IPI all 0 - poke_cpu >> : >> >> If this is the cause of the problem we would like to deliver the >> interrupts to all of the cpus. How do we achieve this with smartos? >> >> Kind Regards, >> Stefan >> > > ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
