For posterity, I sorted this by speaking with ASRock Rack and getting an as-yet unreleased bios that has the ARI forwarding option. Enabled this and all working now.
On Fri, 1 Mar 2019 at 13:20, Dominik Csapak <[email protected]> wrote: > On 01.03.19 14:13, Mark Adams wrote: > > On Fri, 1 Mar 2019 at 12:52, Dominik Csapak <[email protected]> > wrote: > > > >> On 01.03.19 13:37, Mark Adams wrote: > >>> Hi All, > >>> > >>> I'm trying this out, based on the wiki post and the forum posts: > >>> > >>> > >> > https://forum.proxmox.com/threads/amd-s7150-mxgpu-with-proxmox-ve-5-x.50464/ > >>> > >>> https://pve.proxmox.com/wiki/MxGPU_with_AMD_S7150_under_Proxmox_VE_5.x > >>> > >>> However I'm having issues getting the gim driver working. Was just > >>> wondering if the Proxmox staff member that tested this out came across > >> this > >>> particular issue, or if anyone else had any insights. > >> > >> Hi, i am the one that tested this. > >> > > > > Hi Dominik, Thanks for getting back to me so quickly. > > > > Hi, no problem > > > > >> > >>> > >>> My hardware is an ASRock EPYCD8-2T motherboard (SR-IOV enabled in bios) > >> and > >>> an AMD S7150. Proxmox is 5.3-11. > >>> > >>> When running the modprobe of gim, it crashes out with the following: > >>> > >>> [Fri Mar 1 12:31:49 2019] gim info:(enable_sriov:299) Enable SRIOV > >>> [Fri Mar 1 12:31:49 2019] gim info:(enable_sriov:300) Enable SRIOV vfs > >>> count = 16 > >>> [Fri Mar 1 12:31:49 2019] pci 0000:61:02.0: [1002:692f] type 7f class > >>> 0xffffff > >>> [Fri Mar 1 12:31:49 2019] pci 0000:61:02.0: unknown header type 7f, > >>> ignoring device > >>> [Fri Mar 1 12:31:50 2019] gim error:(enable_sriov:311) Fail to enable > >>> sriov, status = fffffffb > >>> [Fri Mar 1 12:31:50 2019] gim error:(set_new_adapter:668) Failed to > >>> properly enable SRIOV > >>> [Fri Mar 1 12:31:50 2019] gim info:(gim_probe:91) AMD GIM probe: > >> pf_count > >>> = 1 > >>> > >> > >> mhmm i cannot really remember if that exact error message occured, but > >> you have to enable several things in the bios > >> > >> AMD-Vi/VT-d > >> SR-IOV > >> ARI > >> and possibly above-4g-decoding > >> > >> also make sure you enable the 'legacy' or non uefi oprom for > >> that card > >> > >> on our supermicro board we could select the oprom for each pcie port > >> separately > >> > > > > It's the same with this ASRock Rack board. I've set the oprom to legacy > for > > PCIE slot 1, but it doesn't seem to make any difference. I've also tried > > other slots but that doesn't make a difference either. > > > > The only thing I can't find, is any option relating to ARI. Do you recall > > at all what the option was called? I think the supermicro and asrock > boards > > are pretty similar when it comes to options, but maybe this board is > > missing ARI. > > With ARI i mean Alternative Routing-ID Interpretation, an PCI Extension[1] > > > > > Also I have ACS enabled but that doesn't help either. > > > > > >> names may be different in your bios, > >> or some options may not exists at all > >> > >> at last, a different pcie port may be necessary, depending on how > >> the mainboard is wired (with epyc all pcie ports should go to the > >> cpu, but i do not know about your specific board) > >> > >> if all else fails, i would open an issue on github for the gim project > >> and ask there if anything is known > >> > > > > Thanks I will do that. > > > > ok, if i remember anything else, i will answer here on the list > > 1: > > https://pcisig.com/sites/default/files/specification_documents/ECN-alt-rid-interpretation-070604.pdf > > > _______________________________________________ > pve-user mailing list > [email protected] > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user _______________________________________________ pve-user mailing list [email protected] https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
