On 09/19/13 15:56, Olivier Martin wrote:
> Contributed-under: TianoCore Contribution Agreement 1.0
> Signed-off-by: Olivier Martin <olivier.mar...@arm.com>
> ---
> OvmfPkg/VirtioPciDeviceDxe/VirtioPciDevice.c | 653
> +++++++++++++++++++++
> OvmfPkg/VirtioPciDeviceDxe/VirtioPciDevice.h | 246 ++++++++
> OvmfPkg/VirtioPciDeviceDxe/VirtioPciDeviceDxe.inf | 42 ++
> OvmfPkg/VirtioPciDeviceDxe/VirtioPciFunctions.c | 188 ++++++
> 4 files changed, 1129 insertions(+), 0 deletions(-)
> create mode 100644 OvmfPkg/VirtioPciDeviceDxe/VirtioPciDevice.c
> create mode 100644 OvmfPkg/VirtioPciDeviceDxe/VirtioPciDevice.h
> create mode 100644 OvmfPkg/VirtioPciDeviceDxe/VirtioPciDeviceDxe.inf
> create mode 100644 OvmfPkg/VirtioPciDeviceDxe/VirtioPciFunctions.c
Okay, so I'm now skipping most of this patch (to be picked up again,
once I'm finished with 03/10); I'd just like to point out the bug I
think to have found by testing & debugging.
I wanted to boot my usual Fedora 19 guest (virtio-blk disk), using
RHEL-6 qemu-kvm. Unfortunately, the OVMF build with the series runs into
an infinite loop very quickly. Relevant diff between the OVMF debug
logs:
Found Mass Storage device: PciRoot(0x0)/Pci(0x4,0x0)
-InstallProtocolInterface: 964E5B21-6459-11D2-8E39-00A0C969723B 3E9ADB90
-InstallProtocolInterface: CE345171-BA0B-11D2-8E4F-00A0C969723B 3E9ADC20
- BlockSize : 512
- LastBlock : FFFFFF
- Valid efi partition table header
-[...]
+InstallProtocolInterface: FA920010-6785-4941-B6EC-498C579F160A 3E9A3A20
+InstallProtocolInterface: 964E5B21-6459-11D2-8E39-00A0C969723B 3E9A3B88
+InstallProtocolInterface: CE345171-BA0B-11D2-8E4F-00A0C969723B 3E9A3C20
BlockSize : 512
LastBlock : FFFFFF
+[HANGS]
The device under PciRoot(0x0)/Pci(0x4,0x0) is the virtio-blk disk in my
VM.
964E5B21-... is BlockIo.
CE345171-... is DiskIo.
So, the currently in-tree driver installs BlockIo on the virtio-blk
device (using PciIo), and DiskIo is automatically installed.
The patchset first installs the new VirtioDevice protocol
(FA920010-...), then the same two (BlockIo and DiskIo) are installed.
After that point the build with the patchset hangs for me.
The infinite loop is the one in VirtioFlush(). The partition driver
wants to read the beginning of the block device (the last two lines are
extra debug messages):
InstallProtocolInterface: FA920010-6785-4941-B6EC-498C579F160A 3E9A3A20
InstallProtocolInterface: 964E5B21-6459-11D2-8E39-00A0C969723B 3E9A3B88
InstallProtocolInterface: CE345171-BA0B-11D2-8E4F-00A0C969723B 3E9A3C20
BlockSize : 512
LastBlock : FFFFFF
SynchronousRequest: entry, Dev=3E9A3B18 Lba=0 BufferSize=200 RequestIsWrite=0
VirtioFlush: entry
but the loop in VirtioFlush(), waiting for the answer from the
hypervisor, never terminates.
I've written some debug patches for RHEL-6 qemu-kvm so that I can see
the virtio PCI traffic. I'm attaching the debug output produced for both
OVMF builds, unpatched (working) and patched (spinning). I forced off
the working VM when it displayed the grub boot menu.
The problem is that the virtio device configuration is not done
correctly. The common virtio header, including the Device Status field,
is located at the beginning of the PCI IO space, but this patch tries to
access those offsets *after* the common virtio header, in the
device-specific area.
The VirtioPciDeviceWrite() function implements the "WriteDevice" member
of the new protocol for PCI devices. The comment for that member
function says
// Functions to read/write Device Specific headers
which is correct, and the PCI implementation seems correct too:
> +STATIC
> +EFI_STATUS
> +EFIAPI
> +VirtioPciInit (
> + IN OUT VIRTIO_PCI_DEVICE *Device
> + )
> +{
> + EFI_STATUS Status;
> + EFI_PCI_IO_PROTOCOL *PciIo;
> + PCI_TYPE00 Pci;
> +
> + ASSERT (Device != NULL);
> + PciIo = Device->PciIo;
> + ASSERT (PciIo != NULL);
> + ASSERT (PciIo->Pci.Read != NULL);
> +
> + Status = PciIo->Pci.Read (
> + PciIo, // (protocol, device)
> + // handle
> + EfiPciIoWidthUint32, // access width & copy
> + // mode
> + 0, // Offset
> + sizeof (Pci) / sizeof (UINT32), // Count
> + &Pci // target buffer
> + );
> + if (EFI_ERROR (Status)) {
> + return Status;
> + }
> + Device->VirtioDevice.SubSystemDeviceId = Pci.Device.SubsystemID;
> +
> + // Note: We should detect if the Device has MSI-X capability. In case the
> device has
> + // MSI-X capability then the offset would be 24.
> + Device->DeviceSpecificConfigurationOffset = 20;
Set here
> +
> + return EFI_SUCCESS;
> +}
and
> +EFI_STATUS
> +VirtioPciDeviceWrite (
> + IN VIRTIO_DEVICE_PROTOCOL *This,
> + IN UINTN FieldOffset,
> + IN UINTN FieldSize,
> + IN UINT64 Value
> + )
> +{
> + UINTN Count;
> + EFI_PCI_IO_PROTOCOL_WIDTH Width;
> + EFI_PCI_IO_PROTOCOL *PciIo;
> + VIRTIO_PCI_DEVICE *Dev;
> +
> + Dev = VIRTIO_PCI_DEVICE_FROM_VIRTIO_DEVICE (This);
> + PciIo = Dev->PciIo;
> +
> + Count = 1;
> + switch (FieldSize) {
> + case 1:
> + Width = EfiPciIoWidthUint8;
> + break;
> +
> + case 2:
> + Width = EfiPciIoWidthUint16;
> + break;
> +
> + case 8:
> + // The 64bit PCI I/O is broken down into two 32bit writes to prevent
> + // any alignment or width issues.
> + // The UEFI spec says under EFI_PCI_IO_PROTOCOL.Io.Write():
> + //
> + // The I/O operations are carried out exactly as requested. The caller
> + // is responsible for any alignment and I/O width issues which the
> + // bus, device, platform, or type of I/O might require. For example on
> + // some platforms, width requests of EfiPciIoWidthUint64 do not work*/
> + Count = 2;
> + // fall through
> +
> + case 4:
> + Width = EfiPciIoWidthUint32;
> + break;
> +
> + default:
> + ASSERT (FALSE);
> + return EFI_INVALID_PARAMETER;
> + }
> +
> + return PciIo->Io.Write (
> + PciIo,
> + Width,
> + PCI_BAR_IDX0,
> + Dev->DeviceSpecificConfigurationOffset + FieldOffset,
used here.
> + Count,
> + &Value
> + );
> +}
The problem is that VirtioPciSetDeviceStatus() tries to use this
function to set a common (ie. *not* device-specific) field with this
function, via the VIRTIO_CFG_WRITE() macro (which just forwards the
arguments):
> +EFI_STATUS
> +VirtioPciSetDeviceStatus (
> + VIRTIO_DEVICE_PROTOCOL *This,
> + UINT8 DeviceStatus
> + )
> +{
> + VIRTIO_PCI_DEVICE *Dev;
> +
> + Dev = VIRTIO_PCI_DEVICE_FROM_VIRTIO_DEVICE (This);
> +
> + return VIRTIO_CFG_WRITE (Dev, VIRTIO_QUEUE_DEVICE_STATUS_OFFSET, sizeof
> (UINT8), DeviceStatus);
> +}
>
In a nutshell, VirtioPciSetDeviceStatus() breaks the interface contract
of VIRTIO_DEVICE_PROTOCOL.WriteDevice(). The former targets a common
virtio field, but the latter provides access to device-specific fields.
Consequently, the virtio device is never brought up, and the guest code
in VirtioFlush() waits for the hypervisor's reply in vain.
I don't yet know enough about the series to suggest a fix (or to fix it
myself just so I can continue the testing and the review). I'd like to
get good test results before reviewing in more detail.
Would it be okay with you to send a v3 (as a separate, top-level thread)
addressing my v2 comments thus far (including the rewrapping of commit
messages to 74 columns (except any log extracts possibly quoted there),
and including the rewrapping of all new code to 79 columns)?
Fixing this PCI problem might require more intrusive reorganization (I'm
not sure), and if that happens, I'd like to keep a fresh mind for v3.
Apologies for the churn...
Thank you!
Laszlo
2013-09-19 19:36:30.541+0000: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/root/bin/rhel-qemu-wrapper -name ovmf.f19 -S -M rhel6.4.0 -enable-kvm -bios
/home/virt-images/OVMF.fd -m 1024 -smp 1,sockets=1,cores=1,threads=1 -uuid
7b20a9f5-274c-0f58-8183-a89538bca62a -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/ovmf.f19.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive
file=/home/virt-images/ovmf.f19.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=/var/lib/libvirt/images/Edk2-AppPkg.img,if=none,id=drive-virtio-disk1,readonly=on,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1
-drive
file=/var/lib/libvirt/images/sb-keys.img,if=none,id=drive-virtio-disk2,readonly=on,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk2,id=virtio-disk2
-netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:93:83:42,bus=pci.0,addr=0x3,bootindex=2
-chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0
-device usb-tablet,id=input0 -vnc 127.0.0.1:0 -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/9
virtio_map: pci_dev=0x7ff172c1a010 region_num=0 addr=c100 size=20 type=1
virtio_map: pci_dev=0x7ff172c10010 region_num=0 addr=c080 size=40 type=1
virtio_map: pci_dev=0x7ff172c1f1e0 region_num=0 addr=c0e0 size=20 type=1
virtio_map: pci_dev=0x7ff172c01920 region_num=0 addr=c0c0 size=20 type=1
virtio_map: pci_dev=0x7ff172c10810 region_num=0 addr=c040 size=40 type=1
virtio_map: pci_dev=0x7ff172c15320 region_num=0 addr=c000 size=40 type=1
virtio_map: pci_dev=0x7ff172c10010 region_num=0 addr=c080 size=40 type=1
virtio_ioport_write: opaque=0x7ff172c10010 addr=0x00000012 val=0x00000000
virtio_ioport_write: opaque=0x7ff172c10010 addr=0x00000012 val=0x00000001
virtio_ioport_write: opaque=0x7ff172c10010 addr=0x00000012 val=0x00000003
virtio_ioport_read: proxy=0x7ff172c10010 addr=0x00000000 ret=0x71000654
virtio_config_readl: vdev=0x7ff172ab6b20 addr=0x00000000
virtio_config_readl: vdev=0x7ff172ab6b20 addr=0x00000004
virtio_config_readl: vdev=0x7ff172ab6b20 addr=0x00000014
virtio_ioport_write: opaque=0x7ff172c10010 addr=0x0000000e val=0x00000000
virtio_ioport_read: proxy=0x7ff172c10010 addr=0x0000000c ret=0x00000080
virtio_ioport_write: opaque=0x7ff172c10010 addr=0x00000008 val=0x0003e971
virtio_ioport_write: opaque=0x7ff172c10010 addr=0x00000004 val=0x00000000
virtio_ioport_write: opaque=0x7ff172c10010 addr=0x00000012 val=0x00000007
virtio_map: pci_dev=0x7ff172c10810 region_num=0 addr=c040 size=40 type=1
virtio_ioport_write: opaque=0x7ff172c10810 addr=0x00000012 val=0x00000000
virtio_ioport_write: opaque=0x7ff172c10810 addr=0x00000012 val=0x00000001
virtio_ioport_write: opaque=0x7ff172c10810 addr=0x00000012 val=0x00000003
virtio_ioport_read: proxy=0x7ff172c10810 addr=0x00000000 ret=0x71000674
virtio_config_readl: vdev=0x7ff172b98720 addr=0x00000000
virtio_config_readl: vdev=0x7ff172b98720 addr=0x00000004
virtio_config_readl: vdev=0x7ff172b98720 addr=0x00000014
virtio_ioport_write: opaque=0x7ff172c10810 addr=0x0000000e val=0x00000000
virtio_ioport_read: proxy=0x7ff172c10810 addr=0x0000000c ret=0x00000080
virtio_ioport_write: opaque=0x7ff172c10810 addr=0x00000008 val=0x0003e96d
virtio_ioport_write: opaque=0x7ff172c10810 addr=0x00000004 val=0x00000000
virtio_ioport_write: opaque=0x7ff172c10810 addr=0x00000012 val=0x00000007
virtio_map: pci_dev=0x7ff172c15320 region_num=0 addr=c000 size=40 type=1
virtio_ioport_write: opaque=0x7ff172c15320 addr=0x00000012 val=0x00000000
virtio_ioport_write: opaque=0x7ff172c15320 addr=0x00000012 val=0x00000001
virtio_ioport_write: opaque=0x7ff172c15320 addr=0x00000012 val=0x00000003
virtio_ioport_read: proxy=0x7ff172c15320 addr=0x00000000 ret=0x71000674
virtio_config_readl: vdev=0x7ff172ba8920 addr=0x00000000
virtio_config_readl: vdev=0x7ff172ba8920 addr=0x00000004
virtio_config_readl: vdev=0x7ff172ba8920 addr=0x00000014
virtio_ioport_write: opaque=0x7ff172c15320 addr=0x0000000e val=0x00000000
virtio_ioport_read: proxy=0x7ff172c15320 addr=0x0000000c ret=0x00000080
virtio_ioport_write: opaque=0x7ff172c15320 addr=0x00000008 val=0x0003e8e9
virtio_ioport_write: opaque=0x7ff172c15320 addr=0x00000004 val=0x00000000
virtio_ioport_write: opaque=0x7ff172c15320 addr=0x00000012 val=0x00000007
virtio_map: pci_dev=0x7ff172c1a010 region_num=0 addr=c100 size=20 type=1
virtio_map: pci_dev=0x7ff172c1f1e0 region_num=0 addr=c0e0 size=20 type=1
virtio_map: pci_dev=0x7ff172c01920 region_num=0 addr=c0c0 size=20 type=1
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000012 val=0x00000000
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000012 val=0x00000001
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000012 val=0x00000003
virtio_ioport_read: proxy=0x7ff172c1a010 addr=0x00000000 ret=0x711fffe3
virtio_config_readb: vdev=0x7ff173129320 addr=0x00000000
virtio_config_readb: vdev=0x7ff173129320 addr=0x00000001
virtio_config_readb: vdev=0x7ff173129320 addr=0x00000002
virtio_config_readb: vdev=0x7ff173129320 addr=0x00000003
virtio_config_readb: vdev=0x7ff173129320 addr=0x00000004
virtio_config_readb: vdev=0x7ff173129320 addr=0x00000005
virtio_config_readw: vdev=0x7ff173129320 addr=0x00000006
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000012 val=0x00000000
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000012 val=0x00000001
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000012 val=0x00000003
virtio_ioport_read: proxy=0x7ff172c1a010 addr=0x00000000 ret=0x711fffe3
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x0000000e val=0x00000000
virtio_ioport_read: proxy=0x7ff172c1a010 addr=0x0000000c ret=0x00000100
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000008 val=0x0003ca20
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x0000000e val=0x00000001
virtio_ioport_read: proxy=0x7ff172c1a010 addr=0x0000000c ret=0x00000100
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000008 val=0x0003ca1d
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000004 val=0x00010020
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000012 val=0x00000007
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000010 val=0x00000000
virtio_config_readw: vdev=0x7ff173129320 addr=0x00000006
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000010 val=0x00000000
virtio_config_readw: vdev=0x7ff173129320 addr=0x00000006
virtio_config_readw: vdev=0x7ff173129320 addr=0x00000006
virtio_config_readw: vdev=0x7ff173129320 addr=0x00000006
virtio_config_readw: vdev=0x7ff173129320 addr=0x00000006
virtio_config_readw: vdev=0x7ff173129320 addr=0x00000006
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000010 val=0x00000000
virtio_config_readw: vdev=0x7ff173129320 addr=0x00000006
virtio_config_readw: vdev=0x7ff173129320 addr=0x00000006
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000010 val=0x00000000
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000010 val=0x00000000
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000010 val=0x00000000
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000010 val=0x00000000
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000010 val=0x00000000
virtio_config_readw: vdev=0x7ff173129320 addr=0x00000006
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000010 val=0x00000000
virtio_config_readw: vdev=0x7ff173129320 addr=0x00000006
virtio_config_readw: vdev=0x7ff173129320 addr=0x00000006
virtio_config_readw: vdev=0x7ff173129320 addr=0x00000006
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000010 val=0x00000000
virtio_config_readw: vdev=0x7ff173129320 addr=0x00000006
virtio_config_readw: vdev=0x7ff173129320 addr=0x00000006
virtio_ioport_write: opaque=0x7ff172c1a010 addr=0x00000010 val=0x00000000
qemu: terminating on signal 15 from pid 10286
2013-09-19 19:36:36.951+0000: shutting down
2013-09-19 19:38:22.423+0000: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none
/root/bin/rhel-qemu-wrapper -name ovmf.f19 -S -M rhel6.4.0 -enable-kvm -bios
/home/virt-images/OVMF.fd -m 1024 -smp 1,sockets=1,cores=1,threads=1 -uuid
7b20a9f5-274c-0f58-8183-a89538bca62a -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/ovmf.f19.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive
file=/home/virt-images/ovmf.f19.img,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=/var/lib/libvirt/images/Edk2-AppPkg.img,if=none,id=drive-virtio-disk1,readonly=on,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1
-drive
file=/var/lib/libvirt/images/sb-keys.img,if=none,id=drive-virtio-disk2,readonly=on,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,drive=drive-virtio-disk2,id=virtio-disk2
-netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:93:83:42,bus=pci.0,addr=0x3,bootindex=2
-chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0
-device usb-tablet,id=input0 -vnc 127.0.0.1:0 -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/13
virtio_map: pci_dev=0x7f263686c010 region_num=0 addr=c100 size=20 type=1
virtio_map: pci_dev=0x7f2636862010 region_num=0 addr=c080 size=40 type=1
virtio_map: pci_dev=0x7f26368711e0 region_num=0 addr=c0e0 size=20 type=1
virtio_map: pci_dev=0x7f2636853920 region_num=0 addr=c0c0 size=20 type=1
virtio_map: pci_dev=0x7f2636862810 region_num=0 addr=c040 size=40 type=1
virtio_map: pci_dev=0x7f2636867320 region_num=0 addr=c000 size=40 type=1
virtio_map: pci_dev=0x7f2636862010 region_num=0 addr=c080 size=40 type=1
virtio_config_writeb: vdev=0x7f2636708b20 addr=0x00000012 data=0x00000000
virtio_config_writeb: vdev=0x7f2636708b20 addr=0x00000012 data=0x00000001
virtio_config_writeb: vdev=0x7f2636708b20 addr=0x00000012 data=0x00000003
virtio_config_readl: vdev=0x7f2636708b20 addr=0x00000000
virtio_config_readl: vdev=0x7f2636708b20 addr=0x00000000
virtio_config_readl: vdev=0x7f2636708b20 addr=0x00000004
virtio_config_writew: vdev=0x7f2636708b20 addr=0x0000000e data=0x00000000
virtio_config_readw: vdev=0x7f2636708b20 addr=0x0000000c
virtio_config_writel: vdev=0x7f2636708b20 addr=0x00000008 data=0x0003e7ab
virtio_config_writel: vdev=0x7f2636708b20 addr=0x00000004 data=0x00000000
virtio_config_writeb: vdev=0x7f2636708b20 addr=0x00000012 data=0x00000007
virtio_config_writew: vdev=0x7f2636708b20 addr=0x00000010 data=0x00000000
qemu: terminating on signal 15 from pid 10286
2013-09-19 19:38:27.003+0000: shutting down
------------------------------------------------------------------------------
LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99!
1,500+ hours of tutorials including VisualStudio 2012, Windows 8, SharePoint
2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack includes
Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/20/13.
http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk
_______________________________________________
edk2-devel mailing list
edk2-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/edk2-devel
