Re: [kvm-devel] [PATCH] Make virtio devices multi-function
Ian Kirk wrote: Avi Kivity wrote: For mass storage, we should follow the SCSI model with a single device serving multiple disks, similar to what you suggest. Not sure if the device should have a single queue or one queue per disk. Don't you just end up re-implementing SCSI then, at which point you might as well stick with a 'fake' SCSI device in the guest? A virtio-scsi controller is indeed useful as it can control tapes, media changers, and other fancy stuff in addition to ordinary disks. For disks, I'd like to avoid the overhead of scsi command generation and parsing. -- Any sufficiently difficult bug is indistinguishable from a feature. - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH] Make virtio devices multi-function
Anthony Liguori wrote: BTW, I've never been that convinced that hotplugging devices is as useful as people make it out to be. I also think that's particularly true when it comes to hot adding/removing very large numbers of disks. On the contrary, the more disks you have, the more likely one is to fail, so you'd need to hotreplace it (think a setup with redundancy like zfs). -- Any sufficiently difficult bug is indistinguishable from a feature. - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH] Make virtio devices multi-function
* Anthony Liguori ([EMAIL PROTECTED]) wrote: Logically speaking, virtio is a bus. virtio supports all of the features of a bus (discover, hot add, hot remove). Right now, we map virtio devices directly onto the PCI bus. The problem we're trying to address is limitations of the PCI bus. We have a couple options: First question is if we have a real limitation with multiple busses? 1) add a virtio device that supports multiple disks. we need to reinvent hotplug within this device. 2) add a new PCI virtio transport that supports multiple virtio-blk devices within a single PCI slot 3) add a generic PCI virtio transport that supports multiple virtio devices within a single PCI slot compare and contrast above with HBA and disks (makes most sense from my point of view). for 2 and 3, only difference is whether you want to be able to support nics, balloons, and block devices on same pci slot (at which point it's a bridge, how is it different from 4?) 4) add a generic virtio bridge that supports multiple virtio devices within a single virtio device. #4 may seem strange, but it's no different from a PCI-to-PCI bridge. I like #4 the most, but #2 is probably the most practical. Also, your current patch does not work for hotplug disk. thanks, -chris - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH] Make virtio devices multi-function
It sounds reasonable to expect so. ACPI has objects for devices, not functions (IIRC). So what I dislike about multifunction devices is the fact that a single slot shares an IRQ, and that special code is required in the QEMU drivers (virtio guest capability might not always be present). Actually this is not true. It is possible to have separate IRQ's for different devices residing in the same slot, changes are required to the ACPI tables though. I don't see any need for using them if we can extend PCI slots... Maybe require explicit device/function assignment on the command line? It will be managed anyway. ACPI does support hotplugging of individual functions inside slots, not sure how well does Linux (and other OSes) support that.. should be transparent though. - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH] Make virtio devices multi-function
Anthony Liguori wrote: This patch changes virtio devices to be multi-function devices whenever possible. This increases the number of virtio devices we can support now by a factor of 8. With this patch, I've been able to launch a guest with either 220 disks or 220 network adapters. Does this play well with hotplug? Perhaps we need to allocate a new device on hotplug. (certainly if we have a device with one function, which then gets converted to a multifunction device) -- error compiling committee.c: too many arguments to function - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH] Make virtio devices multi-function
Avi Kivity wrote: Anthony Liguori wrote: This patch changes virtio devices to be multi-function devices whenever possible. This increases the number of virtio devices we can support now by a factor of 8. With this patch, I've been able to launch a guest with either 220 disks or 220 network adapters. Does this play well with hotplug? Perhaps we need to allocate a new device on hotplug. Probably not. I imagine you can only hotplug devices, not individual functions? Regards, Anthony Liguori (certainly if we have a device with one function, which then gets converted to a multifunction device) - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH] Make virtio devices multi-function
Anthony Liguori wrote: Avi Kivity wrote: Anthony Liguori wrote: This patch changes virtio devices to be multi-function devices whenever possible. This increases the number of virtio devices we can support now by a factor of 8. With this patch, I've been able to launch a guest with either 220 disks or 220 network adapters. Does this play well with hotplug? Perhaps we need to allocate a new device on hotplug. Probably not. I imagine you can only hotplug devices, not individual functions? It sounds reasonable to expect so. ACPI has objects for devices, not functions (IIRC). Maybe require explicit device/function assignment on the command line? It will be managed anyway. -- error compiling committee.c: too many arguments to function - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH] Make virtio devices multi-function
On Tue, Apr 22, 2008 at 4:15 PM, Anthony Liguori [EMAIL PROTECTED] wrote:
This patch changes virtio devices to be multi-function devices whenever
possible. This increases the number of virtio devices we can support now by
a factor of 8.
[...]
diff --git a/qemu/hw/virtio.c b/qemu/hw/virtio.c
index 9100bb1..9ea14d3 100644
--- a/qemu/hw/virtio.c
+++ b/qemu/hw/virtio.c
@@ -405,9 +405,18 @@ VirtIODevice *virtio_init_pci(PCIBus *bus, const char
*name,
PCIDevice *pci_dev;
uint8_t *config;
uint32_t size;
+static int devfn = 7;
+
+if ((devfn % 8) == 7)
+ devfn = -1;
+else
+ devfn++;
This code look strange... devfn should be passed to virtio_init_pci by
virtio-{net,blk} init functions, no?
Luca
-
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
___
kvm-devel mailing list
kvm-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH] Make virtio devices multi-function
* Anthony Liguori [EMAIL PROTECTED] [2008-04-22 09:16]: This patch changes virtio devices to be multi-function devices whenever possible. This increases the number of virtio devices we can support now by a factor of 8. With this patch, I've been able to launch a guest with either 220 disks or 220 network adapters. Have you confirmed that the network devices show up? I was playing around with some of the limits last night and while it is easy to get QEMU to create the adapters, so far I've only had a guest see 29 pci nics (e1000). -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 [EMAIL PROTECTED] - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH] Make virtio devices multi-function
Ryan Harper wrote: * Anthony Liguori [EMAIL PROTECTED] [2008-04-22 09:16]: This patch changes virtio devices to be multi-function devices whenever possible. This increases the number of virtio devices we can support now by a factor of 8. With this patch, I've been able to launch a guest with either 220 disks or 220 network adapters. Have you confirmed that the network devices show up? I was playing around with some of the limits last night and while it is easy to get QEMU to create the adapters, so far I've only had a guest see 29 pci nics (e1000). Yup, I had an eth219 Regards, Anthony Liguori - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH] Make virtio devices multi-function
On Tue, Apr 22, 2008 at 05:32:45PM +0300, Avi Kivity wrote: Anthony Liguori wrote: This patch changes virtio devices to be multi-function devices whenever possible. This increases the number of virtio devices we can support now by a factor of 8. With this patch, I've been able to launch a guest with either 220 disks or 220 network adapters. Does this play well with hotplug? Perhaps we need to allocate a new device on hotplug. (certainly if we have a device with one function, which then gets converted to a multifunction device) Would have to change the hotplug code to handle functions... It sounds less hacky to just extend the PCI slots instead of (ab)using multiple functions per-slot. - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH] Make virtio devices multi-function
Marcelo Tosatti wrote: Maybe require explicit device/function assignment on the command line? It will be managed anyway. ACPI does support hotplugging of individual functions inside slots, not sure how well does Linux (and other OSes) support that.. should be transparent though. I think we need to decide what we want to target in terms of upper limits. With a bridge or two, we can probably easily do 128. If we really want to push things, I think we should do a PCI based virtio controller. I doubt a large number of PCI devices is ever going to perform very well b/c of interrupt sharing and some of the assumptions in virtio_pci. If we implement a controller, we can use a single interrupt, but multiplex multiple notifications on that single interrupt. We can also be more aggressive about using shared memory instead of PCI config space which would reduce the overall number of exits. We could easily support a very large number of devices this way. But again, what do we want to target for now? Regards, Anthony Liguori - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH] Make virtio devices multi-function
Anthony Liguori wrote: I think we need to decide what we want to target in terms of upper limits. With a bridge or two, we can probably easily do 128. If we really want to push things, I think we should do a PCI based virtio controller. I doubt a large number of PCI devices is ever going to perform very well b/c of interrupt sharing and some of the assumptions in virtio_pci. If we implement a controller, we can use a single interrupt, but multiplex multiple notifications on that single interrupt. We can also be more aggressive about using shared memory instead of PCI config space which would reduce the overall number of exits. We could easily support a very large number of devices this way. But again, what do we want to target for now? I think that for networking we should keep things as is. I don't see anybody using 100 virtual NICs. For mass storage, we should follow the SCSI model with a single device serving multiple disks, similar to what you suggest. Not sure if the device should have a single queue or one queue per disk. -- error compiling committee.c: too many arguments to function - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH] Make virtio devices multi-function
Avi Kivity wrote: Anthony Liguori wrote: I think we need to decide what we want to target in terms of upper limits. With a bridge or two, we can probably easily do 128. If we really want to push things, I think we should do a PCI based virtio controller. I doubt a large number of PCI devices is ever going to perform very well b/c of interrupt sharing and some of the assumptions in virtio_pci. If we implement a controller, we can use a single interrupt, but multiplex multiple notifications on that single interrupt. We can also be more aggressive about using shared memory instead of PCI config space which would reduce the overall number of exits. We could easily support a very large number of devices this way. But again, what do we want to target for now? I think that for networking we should keep things as is. I don't see anybody using 100 virtual NICs. For mass storage, we should follow the SCSI model with a single device serving multiple disks, similar to what you suggest. Not sure if the device should have a single queue or one queue per disk. My latest thought it to do a virtio-based virtio controller. We could avoid creating one in QEMU unless we detect an abnormally large number of disks or something. Regards, Anthony Liguori - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH] Make virtio devices multi-function
On Tue, Apr 22, 2008 at 11:31:11AM -0500, Anthony Liguori wrote: Avi Kivity wrote: Anthony Liguori wrote: I think we need to decide what we want to target in terms of upper limits. With a bridge or two, we can probably easily do 128. If we really want to push things, I think we should do a PCI based virtio controller. I doubt a large number of PCI devices is ever going to perform very well b/c of interrupt sharing and some of the assumptions in virtio_pci. If we implement a controller, we can use a single interrupt, but multiplex multiple notifications on that single interrupt. We can also be more aggressive about using shared memory instead of PCI config space which would reduce the overall number of exits. We should increase the number of interrupt lines, perhaps to 16. Using shared memory to avoid exits sounds very good idea. We could easily support a very large number of devices this way. But again, what do we want to target for now? I think that for networking we should keep things as is. I don't see anybody using 100 virtual NICs. The target was along the lines of 20 nics + 80 disks. Dan? For mass storage, we should follow the SCSI model with a single device serving multiple disks, similar to what you suggest. Not sure if the device should have a single queue or one queue per disk. My latest thought it to do a virtio-based virtio controller. Why do you dislike multiple disks per virtio-blk controller? As mentioned this seems a natural way forward. - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH] Make virtio devices multi-function
For mass storage, we should follow the SCSI model with a single device serving multiple disks, similar to what you suggest. Not sure if the device should have a single queue or one queue per disk. My latest thought it to do a virtio-based virtio controller. Why do you dislike multiple disks per virtio-blk controller? As mentioned this seems a natural way forward. Logically speaking, virtio is a bus. virtio supports all of the features of a bus (discover, hot add, hot remove). Right now, we map virtio devices directly onto the PCI bus. The problem we're trying to address is limitations of the PCI bus. We have a couple options: 1) add a virtio device that supports multiple disks. we need to reinvent hotplug within this device. 2) add a new PCI virtio transport that supports multiple virtio-blk devices within a single PCI slot 3) add a generic PCI virtio transport that supports multiple virtio devices within a single PCI slot 4) add a generic virtio bridge that supports multiple virtio devices within a single virtio device. #4 may seem strange, but it's no different from a PCI-to-PCI bridge. I like #4 the most, but #2 is probably the most practical. Regards, Anthony Liguori - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH] Make virtio devices multi-function
Avi Kivity wrote: For mass storage, we should follow the SCSI model with a single device serving multiple disks, similar to what you suggest. Not sure if the device should have a single queue or one queue per disk. Don't you just end up re-implementing SCSI then, at which point you might as well stick with a 'fake' SCSI device in the guest? - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH] Make virtio devices multi-function
Marcelo Tosatti wrote: On Tue, Apr 22, 2008 at 05:32:45PM +0300, Avi Kivity wrote: Anthony Liguori wrote: This patch changes virtio devices to be multi-function devices whenever possible. This increases the number of virtio devices we can support now by a factor of 8. With this patch, I've been able to launch a guest with either 220 disks or 220 network adapters. Does this play well with hotplug? Perhaps we need to allocate a new device on hotplug. (certainly if we have a device with one function, which then gets converted to a multifunction device) Would have to change the hotplug code to handle functions... BTW, I've never been that convinced that hotplugging devices is as useful as people make it out to be. I also think that's particularly true when it comes to hot adding/removing very large numbers of disks. I think if you created all virtio devices as multifunction devices, but didn't add additional functions until you ran out of PCI slots, it would be a pretty acceptable solution. Hotplug works just as it does today until you get much higher than 32 devices. Even then, hotplug still works with most of your devices (until you hit the absolute maximum number of devices of course). Regards, Anthony Liguori It sounds less hacky to just extend the PCI slots instead of (ab)using multiple functions per-slot. - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
Re: [kvm-devel] [PATCH] Make virtio devices multi-function
On Tue, Apr 22, 2008 at 02:26:45PM -0300, Marcelo Tosatti wrote: On Tue, Apr 22, 2008 at 11:31:11AM -0500, Anthony Liguori wrote: Avi Kivity wrote: Anthony Liguori wrote: I think we need to decide what we want to target in terms of upper limits. With a bridge or two, we can probably easily do 128. If we really want to push things, I think we should do a PCI based virtio controller. I doubt a large number of PCI devices is ever going to perform very well b/c of interrupt sharing and some of the assumptions in virtio_pci. If we implement a controller, we can use a single interrupt, but multiplex multiple notifications on that single interrupt. We can also be more aggressive about using shared memory instead of PCI config space which would reduce the overall number of exits. We should increase the number of interrupt lines, perhaps to 16. Using shared memory to avoid exits sounds very good idea. We could easily support a very large number of devices this way. But again, what do we want to target for now? I think that for networking we should keep things as is. I don't see anybody using 100 virtual NICs. The target was along the lines of 20 nics + 80 disks. Dan? I've already had people ask for ability to as many as 64 disks and 32 nics with Xen, so to my mind, the more we support the better. 100's if possible. Dan. -- |: Red Hat, Engineering, Boston -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :| - This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone ___ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel
