[qubes-users] MSI-x support in domU
I have a Mellanox 10G adapter in my Qubes box. When I spin up a domU with the device passed-thru, I am unable to load the device driver due to what appears to be a lack of support for MSI-x interrupt mapping. There was some chatter a while back regarding MSI support: https://www.google.com/url?q=https%3A%2F%2Fwww.qubes-os.org%2Fnews%2F2017%2F10%2F18%2Fmsi-support%2F=D=1=AFQjCNGXTUp6QSX9Fb0v5Q6hyVc0i6NwfQ One might hope that this would enable MSI-x support as well, but apparently, not. My guess is that someone added a kluge to permit writing to the MSI enable regions in PCI config space for a device, neglecting to add the kluges for MSI-x as well. I find that I need to flag the mellanox as 'permissive' or else I get errors flagged on the dom0 console log regarding attempts to write to PCI config space. Once I do so, I am still unable to load the device driver for the ConnectX 4-LX device (mlx5_core). It fails when it attempts to allocate the set of MSIX vectors sized based on the # of CPUs online. The driver assumes MSIx are available. No fall-back to MSI. No fall-back to INT A/ INT B. Are there some other magic knobs I need to tweak? MSIx and Xen does raise some interesting issues. I would like to have to option of spinning up a domU with, say, 20 VCPU and, knowing that the Mellanox will assign queues to MSIx and I can assign MSIx to CPUs, I would like to have dom0 bind the vCPU to real CPU so that the interrupt mapping works correctly. This would be for some network performance work I have to do occasionally. I am also keen to enable the VF devices in the adapter (using some domU instance to enable) so that these VF instances can be passed to other domU instances. Also want to see if I can get hardware offload working and OVS working in qubes. Just for fun. Q: if a domU kernel enables VF devices in a mapped PF device instance, will the dom0 kernel discover the VF devices? IE: what is the mechanism whereby a kernel discovers the need for a bus-walk? This has to work correctly for Xen, no? -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/cdfd36f3-2087-4f34-98fb-732f77fb21ed%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [qubes-users] HP Z8 G4
> On 4 May 2018, at 20:00, awokd <aw...@danwin1210.me> wrote: > > On Fri, May 4, 2018 5:16 am, Damon Permezel wrote: > >> Still, that device, the X722 one using the i40e driver, is useless for >> networking as the TX interrupt never fires, and the driver times out and >> resets. > > I have a USB2 controller in one of my machines that behaves similarly. Try > putting the problem device in a "sys-net2" and experimenting with adding > kernelopts like "pci=nomsi" or "irqpoll”. Added a new ethernet card. Added “pci=nomsi” to get it working. Otherwise, it was not getting interrupts. > >> GUI seems sluggish as well, though, even though `top` shows >> pretty much 100% idle. > > Check /var/log/Xorg.* logs to make sure it's not software rendering. > Thanks for that. -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/DCD8B7CB-A0FE-4D1F-889F-2F23CD07683E%40me.com. For more options, visit https://groups.google.com/d/optout. signature.asc Description: Message signed with OpenPGP
[qubes-users] HCL - HP Z8 G4
Need to set kernelopts to include “pci=nomsi” in sys-net in order for network to work. One of the built-in NICs cannot be reset when sys-net starts. I cannot find recent instructions to use the nVidia drivers, so I am still using the nouveau one. -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/D1AB0E07-3E4F-4BA3-9A66-BD1E39609B69%40me.com. For more options, visit https://groups.google.com/d/optout. Qubes-HCL-HP-HP_Z8_G4_Workstation-20180507-165244.yml Description: Binary data signature.asc Description: Message signed with OpenPGP
[qubes-users] HP Z8 G4
Sort of a progress report. I will eventually try and put together something more meaningful for the HCL. HP Z8 G4 Dual “Intel Silver 4114”, hyperthreading disabled. 64GB memory. nVidia. PS3 style keyboard and mouse. Storage still sucks. Currently running off an enterprise NAS 4T drive on a SAS controller. Will migrate to NVMe/PCIe3 and SSD. Mellanox 10G dual adapter. “native” dual 1G ethernet: i219-LM X722 With the correct BIOS options set to enable all the requisite hypervisor functions, installs and boots. Install failed once due to a Fedora anaconda hang problem, but then succeeded. sys-net initially failed, due to one of the two 1G ethernet devices not having FLR or bus reset, but there is an option in the template editor to not require that. Still, that device, the X722 one using the i40e driver, is useless for networking as the TX interrupt never fires, and the driver times out and resets. The other ethernet port, which uses the e1000e driver, works. I assume the 10G will work, but have yet to try. That will (eventually) give me access to the different VLANs I have. The system for the brief time I have used it seems really sluggish, but I am sure that is the slow NAS drive: 5400RPM. Not sure if 3G or 6G connection. GUI seems sluggish as well, though, even though `top` shows pretty much 100% idle. The box has a TPM. IIRC, I asked for a 2.0 TPM. No time yet to do the AEM stuff. -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/24ABAD0B-E06B-4D23-B122-4F05640C190A%40me.com. For more options, visit https://groups.google.com/d/optout. signature.asc Description: Message signed with OpenPGP
[qubes-users] HP Z8 G4 4.0 install
Fixed the BIOS issues. Now no warnings re missing features. Now, however, I am stuck on “Preparing transaction from installation source”. It did not ask me about whonix or about usb. Just skipped to install. I find that there are some old closed bugs reported against fedora re the “Preparing …” issue. -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/BCE32740-6273-4572-ACF9-964BB002A847%40me.com. For more options, visit https://groups.google.com/d/optout. signature.asc Description: Message signed with OpenPGP
[qubes-users] Re: HP Z8 G4 issues
Actually, I just discovered where they hid the options. Under a “security” TAB in the BIOS. I’ll have another go at 4.0. > On 2 May 2018, at 12:27, Damon Permezel <perme...@me.com> wrote: > > The dual “silver xeon 4114” CPUs in my spiffy new HP Z8 claim support for: > >vPro, HT, VT-x, VT-d, VT-x with EPT. > > On install of 4.0, I am presented with a list of missing features: > > HVM/VT-x/AMD-v, IOMMU/VT-d/AMD-Vi, HAP/SLAT/EPT/RVI, Interrupt Remapping > > Interrupt remapping is part of VT-d. > > I guess these are turned off in the BIOS and as the BIOS will not let me turn > them on, I am hosed. > > Damn. I specced this machine specifically to run Qubes, and was prepared to > do some level or dorking about to get it to work, but did not consider that > someone would lock out features in the BIOS. > > However, having said that, when I boot up some linux on it, lscpu tells me > that at least some of these are enabled. > > > root@z8:~# lscpu > Architecture:x86_64 > CPU op-mode(s): 32-bit, 64-bit > Byte Order: Little Endian > CPU(s): 20 > On-line CPU(s) list: 0-19 > Thread(s) per core: 1 > Core(s) per socket: 10 > Socket(s): 2 > NUMA node(s):2 > Vendor ID: GenuineIntel > CPU family: 6 > Model: 85 > Model name: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz > Stepping:4 > CPU MHz: 800.048 > CPU max MHz: 3000. > CPU min MHz: 800. > BogoMIPS:4400.00 > Virtualisation: VT-x > L1d cache: 32K > L1i cache: 32K > L2 cache:1024K > L3 cache:14080K > NUMA node0 CPU(s): 0-9 > NUMA node1 CPU(s): 10-19 > Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca > cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx > pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl > xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl > vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic > movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm > 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin mba > tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep > bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap > clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 > xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local ibpb ibrs stibp > dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke > > > root@z8:~# dmesg | grep -e DMAR -e IOMMU > root@z8:~# > > Looks like the VT-d is not enabled. > -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/01F188A4-C38F-4670-A29E-5AE2124112EE%40me.com. For more options, visit https://groups.google.com/d/optout. signature.asc Description: Message signed with OpenPGP
[qubes-users] HP Z8 G4 issues
The dual “silver xeon 4114” CPUs in my spiffy new HP Z8 claim support for: vPro, HT, VT-x, VT-d, VT-x with EPT. On install of 4.0, I am presented with a list of missing features: HVM/VT-x/AMD-v, IOMMU/VT-d/AMD-Vi, HAP/SLAT/EPT/RVI, Interrupt Remapping Interrupt remapping is part of VT-d. I guess these are turned off in the BIOS and as the BIOS will not let me turn them on, I am hosed. Damn. I specced this machine specifically to run Qubes, and was prepared to do some level or dorking about to get it to work, but did not consider that someone would lock out features in the BIOS. However, having said that, when I boot up some linux on it, lscpu tells me that at least some of these are enabled. root@z8:~# lscpu Architecture:x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 20 On-line CPU(s) list: 0-19 Thread(s) per core: 1 Core(s) per socket: 10 Socket(s): 2 NUMA node(s):2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz Stepping:4 CPU MHz: 800.048 CPU max MHz: 3000. CPU min MHz: 800. BogoMIPS:4400.00 Virtualisation: VT-x L1d cache: 32K L1i cache: 32K L2 cache:1024K L3 cache:14080K NUMA node0 CPU(s): 0-9 NUMA node1 CPU(s): 10-19 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin mba tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local ibpb ibrs stibp dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke root@z8:~# dmesg | grep -e DMAR -e IOMMU root@z8:~# Looks like the VT-d is not enabled. -- You received this message because you are subscribed to the Google Groups "qubes-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to qubes-users+unsubscr...@googlegroups.com. To post to this group, send email to qubes-users@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/qubes-users/E6DBED38-F218-4ECB-BD27-06DB7925458F%40me.com. For more options, visit https://groups.google.com/d/optout.
