Re: [kvm-devel] [PATCH 0/2] Batch writes to MMIO
Le mercredi 23 avril 2008 à 17:05 +0300, Avi Kivity a écrit : Laurent Vivier wrote: These two patches allow to batch writes to MMIO. When kernel has to send MMIO writes to userspace, it stores them in memory until it has to pass the hand to userspace for another reason. This avoids to have too many context switches on operations that can wait. Did you obtain any measurable performance benefit? Well, the problem is how to measure it. Really, I don't know. But when I add traces I saw MMIO writes are batched: by group of 170 (this is the max in a page) at the beginning, and by group of 10 with XP when we move a window. So all comments are welcome... WARNING: this breaks compatibility with old userspace part. So it's just an RFC :) Yes... to have some comments how to manage this :) Laurent -- - [EMAIL PROTECTED] --- The best way to predict the future is to invent it. - Alan Kay - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/2] Batch writes to MMIO
Laurent Vivier wrote: Le mercredi 23 avril 2008 à 17:05 +0300, Avi Kivity a écrit : Laurent Vivier wrote: These two patches allow to batch writes to MMIO. When kernel has to send MMIO writes to userspace, it stores them in memory until it has to pass the hand to userspace for another reason. This avoids to have too many context switches on operations that can wait. Did you obtain any measurable performance benefit? Well, the problem is how to measure it. Really, I don't know. But when I add traces I saw MMIO writes are batched: by group of 170 (this is the max in a page) at the beginning, That's the lovely animation. You can time XP startup, or look at system time in 'top' once it stabilizes. Also compare total host_state_reloads to boot in kvm_stat (each costs 5 usec, after discount). and by group of 10 with XP when we move a window. That's probably programming the cirrus blitter, also a candidate for batching, but probably not measurable. Maybe one of those Windows graphics benchmarks which draw tons of rectangles very fast and cause epilepsy seizures. I guess most interesting is e1000, if we need to do more than one mmio to start transmitting. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/2] Batch writes to MMIO
Anthony Liguori wrote: The ne2k is pretty mmio heavy. You should be able to observe a boost with something like iperf (guest=host) I would think if this is a real savings. If we're just improving ne2k, the complexity isn't worth it. We have two better nics which are widely supported in guests. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/2] Batch writes to MMIO
Le mercredi 23 avril 2008 à 10:10 -0500, Anthony Liguori a écrit : [...] The ne2k is pretty mmio heavy. You should be able to observe a boost with something like iperf (guest=host) I would think if this is a real savings. I like your advices :-D I use iperf with e1000 emulation and a slightly modified patch (to detect MMIO write in a loop), server is on the host, client on the guest, with default values. RESULT WITHOUT BATCHING: [ 4] 0.0-10.0 sec235 MBytes197 Mbits/sec [ 5] 0.0-10.0 sec194 MBytes163 Mbits/sec [ 4] 0.0-10.0 sec185 MBytes155 Mbits/sec [ 5] 0.0-10.0 sec227 MBytes190 Mbits/sec [ 4] 0.0-10.0 sec196 MBytes164 Mbits/sec [ 5] 0.0-10.0 sec194 MBytes163 Mbits/sec [ 4] 0.0-10.0 sec184 MBytes154 Mbits/sec RESULT WITH BATCHING: Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) [ 4] 0.0-10.0 sec357 MBytes299 Mbits/sec [ 5] 0.0-10.1 sec418 MBytes347 Mbits/sec [ 4] 0.0-10.0 sec408 MBytes342 Mbits/sec [ 5] 0.0-10.0 sec422 MBytes353 Mbits/sec [ 4] 0.0-10.1 sec436 MBytes362 Mbits/sec [ 5] 0.0-10.0 sec416 MBytes348 Mbits/sec [ 4] 0.0-10.0 sec431 MBytes361 Mbits/sec Well, it's nice ? Laurent -- - [EMAIL PROTECTED] --- The best way to predict the future is to invent it. - Alan Kay - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/2] Batch writes to MMIO
Laurent Vivier wrote: Le mercredi 23 avril 2008 à 10:10 -0500, Anthony Liguori a écrit : [...] The ne2k is pretty mmio heavy. You should be able to observe a boost with something like iperf (guest=host) I would think if this is a real savings. I like your advices :-D I use iperf with e1000 emulation and a slightly modified patch (to detect MMIO write in a loop), server is on the host, client on the guest, with default values. RESULT WITHOUT BATCHING: [ 4] 0.0-10.0 sec235 MBytes197 Mbits/sec [ 5] 0.0-10.0 sec194 MBytes163 Mbits/sec [ 4] 0.0-10.0 sec185 MBytes155 Mbits/sec [ 5] 0.0-10.0 sec227 MBytes190 Mbits/sec [ 4] 0.0-10.0 sec196 MBytes164 Mbits/sec [ 5] 0.0-10.0 sec194 MBytes163 Mbits/sec [ 4] 0.0-10.0 sec184 MBytes154 Mbits/sec RESULT WITH BATCHING: Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) [ 4] 0.0-10.0 sec357 MBytes299 Mbits/sec [ 5] 0.0-10.1 sec418 MBytes347 Mbits/sec [ 4] 0.0-10.0 sec408 MBytes342 Mbits/sec [ 5] 0.0-10.0 sec422 MBytes353 Mbits/sec [ 4] 0.0-10.1 sec436 MBytes362 Mbits/sec [ 5] 0.0-10.0 sec416 MBytes348 Mbits/sec [ 4] 0.0-10.0 sec431 MBytes361 Mbits/sec Well, it's nice ? It's too good to be true. I think we're seeing two bugs cancel each other out, resulting in a performance gain. Linux doesn't know how to queue outgoing packets, so it bangs on the mmio that starts the transmit after every packet. mmio batching doesn't know that this mmio register is critical for latency, so it queues it up. The result is that you you get not just mmio batching, but also packet batching! Which dramatically improves performace at the expense of latency. Sorry (if it's true :) -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/2] Batch writes to MMIO
Le mercredi 23 avril 2008 à 19:25 +0300, Avi Kivity a écrit : Laurent Vivier wrote: Le mercredi 23 avril 2008 à 10:10 -0500, Anthony Liguori a écrit : [...] The ne2k is pretty mmio heavy. You should be able to observe a boost with something like iperf (guest=host) I would think if this is a real savings. I like your advices :-D I use iperf with e1000 emulation and a slightly modified patch (to detect MMIO write in a loop), server is on the host, client on the guest, with default values. RESULT WITHOUT BATCHING: [ 4] 0.0-10.0 sec235 MBytes197 Mbits/sec [ 5] 0.0-10.0 sec194 MBytes163 Mbits/sec [ 4] 0.0-10.0 sec185 MBytes155 Mbits/sec [ 5] 0.0-10.0 sec227 MBytes190 Mbits/sec [ 4] 0.0-10.0 sec196 MBytes164 Mbits/sec [ 5] 0.0-10.0 sec194 MBytes163 Mbits/sec [ 4] 0.0-10.0 sec184 MBytes154 Mbits/sec RESULT WITH BATCHING: Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) [ 4] 0.0-10.0 sec357 MBytes299 Mbits/sec [ 5] 0.0-10.1 sec418 MBytes347 Mbits/sec [ 4] 0.0-10.0 sec408 MBytes342 Mbits/sec [ 5] 0.0-10.0 sec422 MBytes353 Mbits/sec [ 4] 0.0-10.1 sec436 MBytes362 Mbits/sec [ 5] 0.0-10.0 sec416 MBytes348 Mbits/sec [ 4] 0.0-10.0 sec431 MBytes361 Mbits/sec Well, it's nice ? It's too good to be true. I think we're seeing two bugs cancel each other out, resulting in a performance gain. Linux doesn't know how to queue outgoing packets, so it bangs on the mmio that starts the transmit after every packet. mmio batching doesn't know that this mmio register is critical for latency, so it queues it up. The result is that you you get not just mmio batching, but also packet batching! Which dramatically improves performace at the expense of latency. How can I check that ? How can I measure latency ? Perhaps I can swap server and client between guest and host ? Sorry (if it's true :) Thank you for your help, Laurent -- - [EMAIL PROTECTED] --- The best way to predict the future is to invent it. - Alan Kay - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/2] Batch writes to MMIO
Laurent Vivier wrote: Le mercredi 23 avril 2008 à 19:25 +0300, Avi Kivity a écrit : Laurent Vivier wrote: Le mercredi 23 avril 2008 à 10:10 -0500, Anthony Liguori a écrit : [...] The ne2k is pretty mmio heavy. You should be able to observe a boost with something like iperf (guest=host) I would think if this is a real savings. I like your advices :-D I use iperf with e1000 emulation and a slightly modified patch (to detect MMIO write in a loop), server is on the host, client on the guest, with default values. RESULT WITHOUT BATCHING: [ 4] 0.0-10.0 sec235 MBytes197 Mbits/sec [ 5] 0.0-10.0 sec194 MBytes163 Mbits/sec [ 4] 0.0-10.0 sec185 MBytes155 Mbits/sec [ 5] 0.0-10.0 sec227 MBytes190 Mbits/sec [ 4] 0.0-10.0 sec196 MBytes164 Mbits/sec [ 5] 0.0-10.0 sec194 MBytes163 Mbits/sec [ 4] 0.0-10.0 sec184 MBytes154 Mbits/sec RESULT WITH BATCHING: Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) [ 4] 0.0-10.0 sec357 MBytes299 Mbits/sec [ 5] 0.0-10.1 sec418 MBytes347 Mbits/sec [ 4] 0.0-10.0 sec408 MBytes342 Mbits/sec [ 5] 0.0-10.0 sec422 MBytes353 Mbits/sec [ 4] 0.0-10.1 sec436 MBytes362 Mbits/sec [ 5] 0.0-10.0 sec416 MBytes348 Mbits/sec [ 4] 0.0-10.0 sec431 MBytes361 Mbits/sec Well, it's nice ? It's too good to be true. I think we're seeing two bugs cancel each other out, resulting in a performance gain. Linux doesn't know how to queue outgoing packets, so it bangs on the mmio that starts the transmit after every packet. mmio batching doesn't know that this mmio register is critical for latency, so it queues it up. The result is that you you get not just mmio batching, but also packet batching! Which dramatically improves performace at the expense of latency. How can I check that ? How can I measure latency ? ping (from guest to host) Regards, Anthony Liguori Perhaps I can swap server and client between guest and host ? Sorry (if it's true :) Thank you for your help, Laurent - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/2] Batch writes to MMIO
Anthony Liguori wrote: How can I check that ? How can I measure latency ? ping (from guest to host) The guest will halt anyway, flushing its mmio queue. Perhaps a ping while a background process spins, consuming all cpu. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/2] Batch writes to MMIO
Anthony Liguori wrote: Avi Kivity wrote: Anthony Liguori wrote: The ne2k is pretty mmio heavy. You should be able to observe a boost with something like iperf (guest=host) I would think if this is a real savings. If we're just improving ne2k, the complexity isn't worth it. We have two better nics which are widely supported in guests. Sure, but the ne2k should be pretty close to an optimal synthetic benchmark for batching. If we don't see a big improvement with it, we're probably not going to see a big improvement with it for anything. I disagree. If there are other inefficiencies in ne2k, they will mask any performance improvement from batching. So whatever result we get will be inconclusive. I don't expect any massive performance improvement except for splash screens. I'll be happy to settle for a few percent with e1000 or scsi or ide. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH 0/2] Batch writes to MMIO
Le mercredi 23 avril 2008 à 11:48 -0500, Anthony Liguori a écrit : Laurent Vivier wrote: Le mercredi 23 avril 2008 à 19:25 +0300, Avi Kivity a écrit : Laurent Vivier wrote: Le mercredi 23 avril 2008 à 10:10 -0500, Anthony Liguori a écrit : [...] The ne2k is pretty mmio heavy. You should be able to observe a boost with something like iperf (guest=host) I would think if this is a real savings. I like your advices :-D I use iperf with e1000 emulation and a slightly modified patch (to detect MMIO write in a loop), server is on the host, client on the guest, with default values. RESULT WITHOUT BATCHING: [ 4] 0.0-10.0 sec235 MBytes197 Mbits/sec [ 5] 0.0-10.0 sec194 MBytes163 Mbits/sec [ 4] 0.0-10.0 sec185 MBytes155 Mbits/sec [ 5] 0.0-10.0 sec227 MBytes190 Mbits/sec [ 4] 0.0-10.0 sec196 MBytes164 Mbits/sec [ 5] 0.0-10.0 sec194 MBytes163 Mbits/sec [ 4] 0.0-10.0 sec184 MBytes154 Mbits/sec RESULT WITH BATCHING: Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) [ 4] 0.0-10.0 sec357 MBytes299 Mbits/sec [ 5] 0.0-10.1 sec418 MBytes347 Mbits/sec [ 4] 0.0-10.0 sec408 MBytes342 Mbits/sec [ 5] 0.0-10.0 sec422 MBytes353 Mbits/sec [ 4] 0.0-10.1 sec436 MBytes362 Mbits/sec [ 5] 0.0-10.0 sec416 MBytes348 Mbits/sec [ 4] 0.0-10.0 sec431 MBytes361 Mbits/sec Well, it's nice ? It's too good to be true. I think we're seeing two bugs cancel each other out, resulting in a performance gain. Linux doesn't know how to queue outgoing packets, so it bangs on the mmio that starts the transmit after every packet. mmio batching doesn't know that this mmio register is critical for latency, so it queues it up. The result is that you you get not just mmio batching, but also packet batching! Which dramatically improves performace at the expense of latency. How can I check that ? How can I measure latency ? ping (from guest to host) I have 40 ms instead of 0.09 ms, so Avi you are right. Laurent -- - [EMAIL PROTECTED] --- The best way to predict the future is to invent it. - Alan Kay - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
