Re: [Xen-devel] Some Xen pci-passthrough questions
On 10/27/2017 04:16 AM, Sander Eikelenboom wrote: On 27/10/17 00:30, Boris Ostrovsky wrote: On 10/26/2017 04:48 PM, Sander Eikelenboom wrote: - Enabling the xen pt logging in qemu spit out some things, i wonder if they are normal: qemu-system-i386: -serial pty: char device redirected to /dev/pts/16 (label serial0) [00:05.0] xen_pt_realize: Assigning real physical device 08:00.0 to devfn 0x28 [00:05.0] xen_pt_register_regions: IO region 0 registered (size=0x2000 base_addr=0xfe1fe000 type: 0x4) Are these somehow expected / benign (they also occur when pci passthrough is succesful) ?: [00:05.0] xen_pt_config_reg_init: Offset 0x000e mismatch! Emulated=0x0080, host=0x, syncing to 0x0080. [00:05.0] xen_pt_config_reg_init: Offset 0x0010 mismatch! Emulated=0x, host=0xfe1fe004, syncing to 0xfe1fe004. [00:05.0] xen_pt_config_reg_init: Offset 0x0052 mismatch! Emulated=0x, host=0x4803, syncing to 0x0003. [00:05.0] xen_pt_config_reg_init: Offset 0x0072 mismatch! Emulated=0x, host=0x0086, syncing to 0x0080. [00:05.0] xen_pt_config_reg_init: Offset 0x00a4 mismatch! Emulated=0x, host=0x8fc0, syncing to 0x8fc0. [00:05.0] xen_pt_config_reg_init: Offset 0x00b2 mismatch! Emulated=0x, host=0x1012, syncing to 0x1012. [00:05.0] xen_pt_msix_init: get MSI-X table BAR base 0xfe1fe000 [00:05.0] xen_pt_msix_init: table_off = 0x1000, total_entries = 8 [00:05.0] xen_pt_msix_init: table_off = 0x1000, total_entries = 8, PCI_MSIX_ENTRY_SIZE = 0x10, msix->table_offset_adjust = 0, msix->table_base = 0xfe1fe000 [00:05.0] xen_pt_msix_init: Error: Can't map physical MSI-X table: Invalid argument That's mmap() of /dev/mem failing: mmap(NULL, total_entries * PCI_MSIX_ENTRY_SIZE + msix->table_offset_adjust, PROT_READ, MAP_SHARED | MAP_LOCKED, fd, msix->table_base + table_off - msix->table_offset_adjust); Are you running with Craig Bergstrom's patch? Yes sorry for not being clear (and using the log of running with the mmap patch), i specifically meant the "mismatch!" lines, although they also appear in the working case (without the patch) so they probably aren't an issue, but still thought it wouldn't hurt to ask :). I believe these warnings are expected. At least in the case of BAR (offset 0x10) I can see that qemu initializes emulated register to 0 (xen_pt_bar_reg_init()) which would not match what the actual HW has. But I'll let Antony or Stefan confirm that. (I do think the warnings look a bit too alarmist so perhaps they can be softened somewhat) And as far as Craig's patch is concerned we know it's broken and will be reverted. -boris ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Some Xen pci-passthrough questions
On 27/10/17 00:30, Boris Ostrovsky wrote: > On 10/26/2017 04:48 PM, Sander Eikelenboom wrote: >> Hi Boris / Andrew, >> >> In the aftermath of the linux mmap path I have some questions regarding >> pci-passthrough: >> >> - Is pci-passthrough in combination with an auto-ballooning dom0 supposed to >> be a supported combination ? > > I thought it is. I haven't done passthrough recently (and can't do it > right now) but I when I did I am pretty sure I was running with > auto-ballooning dom0. > > Our production does passthrough but they always run with dom0's memory > limited. > Ok, at least it's not a clear "nope", so i will have a go again in a couple of days and report back. >> >> I have used dom0_maxmem settings for dom0 since ages and that works fine >> and stable, >> but while doing some testing around the linux mmap patch i also thought >> to try an auto-ballooning dom0. >> >> That ended up in a crashing PV guest with pci-passthrough and a strange >> error on two HVM's with pci-passthrough, >> about vcpu's (while no configurations where changed). >> >> So if it is supported i probably have some more testing and reporting to >> do ... >> >> >> - While adding some extra logging and enabling the logging on xen pt in qemu, >> i wonder if it wouldn't be beneficial to have at least some basic logging >> permanently enabled ? >> >> - Enabling the xen pt logging in qemu spit out some things, i wonder if they >> are normal: >> >> qemu-system-i386: -serial pty: char device redirected to /dev/pts/16 >> (label serial0) >> [00:05.0] xen_pt_realize: Assigning real physical device 08:00.0 to >> devfn 0x28 >> [00:05.0] xen_pt_register_regions: IO region 0 registered >> (size=0x2000 base_addr=0xfe1fe000 type: 0x4) >> >> Are these somehow expected / benign (they also occur when pci >> passthrough is succesful) ?: >> [00:05.0] xen_pt_config_reg_init: Offset 0x000e mismatch! >> Emulated=0x0080, host=0x, syncing to 0x0080. >> [00:05.0] xen_pt_config_reg_init: Offset 0x0010 mismatch! >> Emulated=0x, host=0xfe1fe004, syncing to 0xfe1fe004. >> [00:05.0] xen_pt_config_reg_init: Offset 0x0052 mismatch! >> Emulated=0x, host=0x4803, syncing to 0x0003. >> [00:05.0] xen_pt_config_reg_init: Offset 0x0072 mismatch! >> Emulated=0x, host=0x0086, syncing to 0x0080. >> [00:05.0] xen_pt_config_reg_init: Offset 0x00a4 mismatch! >> Emulated=0x, host=0x8fc0, syncing to 0x8fc0. >> [00:05.0] xen_pt_config_reg_init: Offset 0x00b2 mismatch! >> Emulated=0x, host=0x1012, syncing to 0x1012. >> >> [00:05.0] xen_pt_msix_init: get MSI-X table BAR base 0xfe1fe000 >> [00:05.0] xen_pt_msix_init: table_off = 0x1000, total_entries = 8 >> [00:05.0] xen_pt_msix_init: table_off = 0x1000, total_entries = 8, >> PCI_MSIX_ENTRY_SIZE = 0x10, msix->table_offset_adjust = 0, >> msix->table_base = 0xfe1fe000 >> [00:05.0] xen_pt_msix_init: Error: Can't map physical MSI-X table: >> Invalid argument > > That's mmap() of /dev/mem failing: > > mmap(NULL, > total_entries * PCI_MSIX_ENTRY_SIZE + > msix->table_offset_adjust, > PROT_READ, > MAP_SHARED | MAP_LOCKED, > fd, > msix->table_base + table_off - msix->table_offset_adjust); > > > Are you running with Craig Bergstrom's patch? Yes sorry for not being clear (and using the log of running with the mmap patch), i specifically meant the "mismatch!" lines, although they also appear in the working case (without the patch) so they probably aren't an issue, but still thought it wouldn't hurt to ask :). > -boris > > >> [00:05.0] xen_pt_msix_size_init: Error: Internal error: Invalid >> xen_pt_msix_init. >> Failed to initialize 12/15, type = 0x1, rc: -22 >> [00:05.0] xen_pt_msi_set_enable: disabling MSI. >> >> This crash seems to indicate the above disabling of MSI isn't handled >> well enough to prevent this from happening: >> *** Error in `/usr/local/lib/xen/bin/qemu-system-i386': corrupted >> size vs. prev_size: 0x55ce13565570 *** >> === Backtrace: = >> /lib/x86_64-linux-gnu/libc.so.6(+0x70bcb)[0x7f700ab7ebcb] >> /lib/x86_64-linux-gnu/libc.so.6(+0x76f96)[0x7f700ab84f96] >> >> >> -- >> Sander > ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Some Xen pci-passthrough questions
>>> On 26.10.17 at 22:48, wrote: > - While adding some extra logging and enabling the logging on xen pt in > qemu, > i wonder if it wouldn't be beneficial to have at least some basic logging > permanently enabled ? For this end everything further down you would better have added the qemu maintainers. I've now put them on Cc. Jan > - Enabling the xen pt logging in qemu spit out some things, i wonder if they > are normal: > > qemu-system-i386: -serial pty: char device redirected to /dev/pts/16 > (label serial0) > [00:05.0] xen_pt_realize: Assigning real physical device 08:00.0 to > devfn 0x28 > [00:05.0] xen_pt_register_regions: IO region 0 registered > (size=0x2000 base_addr=0xfe1fe000 type: 0x4) > > Are these somehow expected / benign (they also occur when pci > passthrough is succesful) ?: > [00:05.0] xen_pt_config_reg_init: Offset 0x000e mismatch! > Emulated=0x0080, host=0x, syncing to 0x0080. > [00:05.0] xen_pt_config_reg_init: Offset 0x0010 mismatch! > Emulated=0x, host=0xfe1fe004, syncing to 0xfe1fe004. > [00:05.0] xen_pt_config_reg_init: Offset 0x0052 mismatch! > Emulated=0x, host=0x4803, syncing to 0x0003. > [00:05.0] xen_pt_config_reg_init: Offset 0x0072 mismatch! > Emulated=0x, host=0x0086, syncing to 0x0080. > [00:05.0] xen_pt_config_reg_init: Offset 0x00a4 mismatch! > Emulated=0x, host=0x8fc0, syncing to 0x8fc0. > [00:05.0] xen_pt_config_reg_init: Offset 0x00b2 mismatch! > Emulated=0x, host=0x1012, syncing to 0x1012. > > [00:05.0] xen_pt_msix_init: get MSI-X table BAR base 0xfe1fe000 > [00:05.0] xen_pt_msix_init: table_off = 0x1000, total_entries = 8 > [00:05.0] xen_pt_msix_init: table_off = 0x1000, total_entries = 8, > PCI_MSIX_ENTRY_SIZE = 0x10, msix->table_offset_adjust = 0, msix->table_base > = > 0xfe1fe000 > [00:05.0] xen_pt_msix_init: Error: Can't map physical MSI-X table: > Invalid argument > [00:05.0] xen_pt_msix_size_init: Error: Internal error: Invalid > xen_pt_msix_init. > Failed to initialize 12/15, type = 0x1, rc: -22 > [00:05.0] xen_pt_msi_set_enable: disabling MSI. > > This crash seems to indicate the above disabling of MSI isn't handled > well enough to prevent this from happening: > *** Error in `/usr/local/lib/xen/bin/qemu-system-i386': corrupted > size vs. prev_size: 0x55ce13565570 *** > === Backtrace: = > /lib/x86_64-linux-gnu/libc.so.6(+0x70bcb)[0x7f700ab7ebcb] > /lib/x86_64-linux-gnu/libc.so.6(+0x76f96)[0x7f700ab84f96] > > > -- > Sander > > ___ > Xen-devel mailing list > Xen-devel@lists.xen.org > https://lists.xen.org/xen-devel ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] Some Xen pci-passthrough questions
On 10/26/2017 04:48 PM, Sander Eikelenboom wrote: > Hi Boris / Andrew, > > In the aftermath of the linux mmap path I have some questions regarding > pci-passthrough: > > - Is pci-passthrough in combination with an auto-ballooning dom0 supposed to > be a supported combination ? I thought it is. I haven't done passthrough recently (and can't do it right now) but I when I did I am pretty sure I was running with auto-ballooning dom0. Our production does passthrough but they always run with dom0's memory limited. > > I have used dom0_maxmem settings for dom0 since ages and that works fine > and stable, > but while doing some testing around the linux mmap patch i also thought > to try an auto-ballooning dom0. > > That ended up in a crashing PV guest with pci-passthrough and a strange > error on two HVM's with pci-passthrough, > about vcpu's (while no configurations where changed). > > So if it is supported i probably have some more testing and reporting to > do ... > > > - While adding some extra logging and enabling the logging on xen pt in qemu, > i wonder if it wouldn't be beneficial to have at least some basic logging > permanently enabled ? > > - Enabling the xen pt logging in qemu spit out some things, i wonder if they > are normal: > > qemu-system-i386: -serial pty: char device redirected to /dev/pts/16 > (label serial0) > [00:05.0] xen_pt_realize: Assigning real physical device 08:00.0 to > devfn 0x28 > [00:05.0] xen_pt_register_regions: IO region 0 registered > (size=0x2000 base_addr=0xfe1fe000 type: 0x4) > > Are these somehow expected / benign (they also occur when pci passthrough > is succesful) ?: > [00:05.0] xen_pt_config_reg_init: Offset 0x000e mismatch! > Emulated=0x0080, host=0x, syncing to 0x0080. > [00:05.0] xen_pt_config_reg_init: Offset 0x0010 mismatch! > Emulated=0x, host=0xfe1fe004, syncing to 0xfe1fe004. > [00:05.0] xen_pt_config_reg_init: Offset 0x0052 mismatch! > Emulated=0x, host=0x4803, syncing to 0x0003. > [00:05.0] xen_pt_config_reg_init: Offset 0x0072 mismatch! > Emulated=0x, host=0x0086, syncing to 0x0080. > [00:05.0] xen_pt_config_reg_init: Offset 0x00a4 mismatch! > Emulated=0x, host=0x8fc0, syncing to 0x8fc0. > [00:05.0] xen_pt_config_reg_init: Offset 0x00b2 mismatch! > Emulated=0x, host=0x1012, syncing to 0x1012. > > [00:05.0] xen_pt_msix_init: get MSI-X table BAR base 0xfe1fe000 > [00:05.0] xen_pt_msix_init: table_off = 0x1000, total_entries = 8 > [00:05.0] xen_pt_msix_init: table_off = 0x1000, total_entries = 8, > PCI_MSIX_ENTRY_SIZE = 0x10, msix->table_offset_adjust = 0, msix->table_base > = 0xfe1fe000 > [00:05.0] xen_pt_msix_init: Error: Can't map physical MSI-X table: > Invalid argument That's mmap() of /dev/mem failing: mmap(NULL, total_entries * PCI_MSIX_ENTRY_SIZE + msix->table_offset_adjust, PROT_READ, MAP_SHARED | MAP_LOCKED, fd, msix->table_base + table_off - msix->table_offset_adjust); Are you running with Craig Bergstrom's patch? -boris > [00:05.0] xen_pt_msix_size_init: Error: Internal error: Invalid > xen_pt_msix_init. > Failed to initialize 12/15, type = 0x1, rc: -22 > [00:05.0] xen_pt_msi_set_enable: disabling MSI. > > This crash seems to indicate the above disabling of MSI isn't handled > well enough to prevent this from happening: > *** Error in `/usr/local/lib/xen/bin/qemu-system-i386': corrupted > size vs. prev_size: 0x55ce13565570 *** > === Backtrace: = > /lib/x86_64-linux-gnu/libc.so.6(+0x70bcb)[0x7f700ab7ebcb] > /lib/x86_64-linux-gnu/libc.so.6(+0x76f96)[0x7f700ab84f96] > > > -- > Sander ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
[Xen-devel] Some Xen pci-passthrough questions
Hi Boris / Andrew, In the aftermath of the linux mmap path I have some questions regarding pci-passthrough: - Is pci-passthrough in combination with an auto-ballooning dom0 supposed to be a supported combination ? I have used dom0_maxmem settings for dom0 since ages and that works fine and stable, but while doing some testing around the linux mmap patch i also thought to try an auto-ballooning dom0. That ended up in a crashing PV guest with pci-passthrough and a strange error on two HVM's with pci-passthrough, about vcpu's (while no configurations where changed). So if it is supported i probably have some more testing and reporting to do ... - While adding some extra logging and enabling the logging on xen pt in qemu, i wonder if it wouldn't be beneficial to have at least some basic logging permanently enabled ? - Enabling the xen pt logging in qemu spit out some things, i wonder if they are normal: qemu-system-i386: -serial pty: char device redirected to /dev/pts/16 (label serial0) [00:05.0] xen_pt_realize: Assigning real physical device 08:00.0 to devfn 0x28 [00:05.0] xen_pt_register_regions: IO region 0 registered (size=0x2000 base_addr=0xfe1fe000 type: 0x4) Are these somehow expected / benign (they also occur when pci passthrough is succesful) ?: [00:05.0] xen_pt_config_reg_init: Offset 0x000e mismatch! Emulated=0x0080, host=0x, syncing to 0x0080. [00:05.0] xen_pt_config_reg_init: Offset 0x0010 mismatch! Emulated=0x, host=0xfe1fe004, syncing to 0xfe1fe004. [00:05.0] xen_pt_config_reg_init: Offset 0x0052 mismatch! Emulated=0x, host=0x4803, syncing to 0x0003. [00:05.0] xen_pt_config_reg_init: Offset 0x0072 mismatch! Emulated=0x, host=0x0086, syncing to 0x0080. [00:05.0] xen_pt_config_reg_init: Offset 0x00a4 mismatch! Emulated=0x, host=0x8fc0, syncing to 0x8fc0. [00:05.0] xen_pt_config_reg_init: Offset 0x00b2 mismatch! Emulated=0x, host=0x1012, syncing to 0x1012. [00:05.0] xen_pt_msix_init: get MSI-X table BAR base 0xfe1fe000 [00:05.0] xen_pt_msix_init: table_off = 0x1000, total_entries = 8 [00:05.0] xen_pt_msix_init: table_off = 0x1000, total_entries = 8, PCI_MSIX_ENTRY_SIZE = 0x10, msix->table_offset_adjust = 0, msix->table_base = 0xfe1fe000 [00:05.0] xen_pt_msix_init: Error: Can't map physical MSI-X table: Invalid argument [00:05.0] xen_pt_msix_size_init: Error: Internal error: Invalid xen_pt_msix_init. Failed to initialize 12/15, type = 0x1, rc: -22 [00:05.0] xen_pt_msi_set_enable: disabling MSI. This crash seems to indicate the above disabling of MSI isn't handled well enough to prevent this from happening: *** Error in `/usr/local/lib/xen/bin/qemu-system-i386': corrupted size vs. prev_size: 0x55ce13565570 *** === Backtrace: = /lib/x86_64-linux-gnu/libc.so.6(+0x70bcb)[0x7f700ab7ebcb] /lib/x86_64-linux-gnu/libc.so.6(+0x76f96)[0x7f700ab84f96] -- Sander ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel