Hi All,
Adding some additional information, as for custom configuration it now works.
I did have some trouble first with getting the VM boot, as it apparently needs
specific settings for the pci bus/slot etc. I took over the config as is from
an VM where it worked with before, before I installed cloudstack. Now I have
found that out, I am stuck with the following:
When starting the VM, I see in syslog on the host/hypervisor that it is
enabling the device for passthrough:
Feb 15 16:20:48 gc-hs1 kernel: [18145.274514] vfio-pci 0000:81:00.2: enabling
device (0000 -> 0002)
I also can see it in the log of the VM, it uses the parameters:
-device
virtio-net-pci,netdev=hostnet0,id=net0,mac=1e:00:34:00:00:f6,bus=pci.0,addr=0x3
\
When doing virsh edit on the domain, I get:
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/>
</source>
<alias name='hostdev0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0c'
function='0x1'/>
</hostdev>
Only thing I never added was driver name, I am a bit suspicious on that but
probably has it's reasons. It worked on stand alone virsh+qemu setup but there
I never had to give up the driver.
Is there some additional configuration that is required?
For now I test with 1 device, but eventually there should be 4 Mellanox VF's
being exposed/passed to the VM.
Thank you yet again,
JeanPaul
________________________________
From: Wei ZHOU <ustcweiz...@gmail.com>
Sent: Wednesday, February 15, 2023 10:42 AM
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: Re: PCI-E Passthrough
You are welcome. Good to know it works.
-Wei
On Wednesday, 15 February 2023, JeanPaul van der Mijle <
jp.vandermi...@hyperai.ai> wrote:
> Hi All,
>
> Nevermind, figured out, it is very important on the alphabetic order when
> signing the request.
>
> Thanks anyways!
>
> JeanPaul
> ________________________________
> From: JeanPaul van der Mijle <jp.vandermi...@hyperai.ai>
> Sent: Wednesday, February 15, 2023 10:05 AM
> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> Subject: Re: PCI-E Passthrough
>
> Hi Wei, All,
>
> I am currently stuck with the API call, I manage to sign requests as
> listUsers and plain updateVirtualMachine with id works, but as soon as I
> add the extraconfig parameter, it keeps complaining it is not correct. This
> is so far with GET request only, I have not yet tried POST, as I haven't
> taken a look if I should do every value with post (command, id,
> extraconfig, response and signature?) or just the extraconfig and how
> signing would go then.
>
> I have tried to sign the request with: "command=updatevirtualmachine&
> id=xxxx&extraconfig=urlencodedconfig"
> both fully lowercase, as well lowercase only the command.
>
> I also tried without adding extraconfig to the signature generation but
> they all end up with:
>
> ["errorcode"]=>
> int(401)
> ["errortext"]=>
> string(58) "unable to verify user credentials and/or request signature"
>
> Currently using PHP to do the signatures and it looks like as following:
>
> <?php
> $baseurl = "http://localhost:8080/client/api?";;
>
> $response = "response=json";
>
> $vmid = "id=6f695b14-71b4-4deb-bc4c-851580324f91";
>
> $apikey = "apikey=<apikey>";
> $secretkey = "<secret>";
> $configxml = <<<EOD
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> function='0x0' multifunction='on'/>
> </hostdev>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0xc1' slot='0x00' function='0x2'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x09' slot='0x00'
> function='0x0' multifunction='on'/>
> </hostdev>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x81' slot='0x00' function='0x6'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> function='0x1'/>
> </hostdev>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0xc1' slot='0x00' function='0x6'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x09' slot='0x00'
> function='0x1'/>
> </hostdev>
> <memballoon model='virtio'>
> <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> function='0x0'/>
> </memballoon>
> <rng model='virtio'>
> <backend model='random'>/dev/urandom</backend>
> <address type='pci' domain='0x0000' bus='0x05' slot='0x00'
> function='0x0'/>
> </rng>
> EOD;
>
> $extraconfig = "extraconfig=" . urlencode($configxml);
>
> $command = "command=updateVirtualMachine&".$vmid;
>
> $hash = hash_hmac("sha1",strtolower($apikey . "&" . $command) ."&" .
> $extraconfig . "&" . $response,$secretkey, true);
> $base64encoded = base64_encode($hash);
> $signature = "signature=" . urlencode($base64encoded);
>
> $link = $baseurl . $apikey . "&" . $command . "&" . $extraconfig . "&"
> .$response . "&" . $signature;
>
> $cURLConnection = curl_init();
>
> curl_setopt($cURLConnection, CURLOPT_URL, $link);
> curl_setopt($cURLConnection, CURLOPT_RETURNTRANSFER, true);
>
> $json_output = curl_exec($cURLConnection);
> curl_close($cURLConnection);
>
> $jsonArrayResponse = json_decode($json_output,true);
>
> var_dump($jsonArrayResponse);
>
>
> The URL end up looking like this:
>
> http://localhost:8080/client/api?apikey=<Apikey>&command=
> updateVirtualMachine&id=<vmid>&extraconfig=%3Chostdev+mode%
> 3D%27subsystem%27+type%3D%27pci%27+managed%3D%27yes%27%
> 3E%0A++++++%3Csource%3E%0A++++++++%3Caddress+domain%3D%
> 270x0000%27+bus%3D%270x81%27+slot%3D%270x00%27+function%3D%
> 270x2%27%2F%3E%0A++++++%3C%2Fsource%3E%0A++++++%
> 3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%
> 270x07%27+slot%3D%270x00%27+function%3D%270x0%27+
> multifunction%3D%27on%27%2F%3E%0A++++%3C%2Fhostdev%3E%0A++
> ++%3Chostdev+mode%3D%27subsystem%27+type%3D%27pci%
> 27+managed%3D%27yes%27%3E%0A++++++%3Csource%3E%0A++++++++%
> 3Caddress+domain%3D%270x0000%27+bus%3D%270xc1%27+slot%3D%
> 270x00%27+function%3D%270x2%27%2F%3E%0A++++++%3C%2Fsource%
> 3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%
> 27+bus%3D%270x09%27+slot%3D%270x00%27+function%3D%270x0%
> 27+multifunction%3D%27on%27%2F%3E%0A++++%3C%2Fhostdev%3E%
> 0A++++%3Chostdev+mode%3D%27subsystem%27+type%3D%27pci%
> 27+managed%3D%27yes%27%3E%0A++++++%3Csource%3E%0A++++++++%
> 3Caddress+domain%3D%270x0000%27+bus%3D%270x81%27+slot%3D%
> 270x00%27+function%3D%270x6%27%2F%3E%0A++++++%3C%2Fsource%
> 3E%0A++++++%3Caddress+type%3D%27pci%27+domain%3D%270x0000%
> 27+bus%3D%270x07%27+slot%3D%270x00%27+function%3D%270x1%
> 27%2F%3E%0A++++%3C%2Fhostdev%3E%0A++++%3Chostdev+mode%3D%
> 27subsystem%27+type%3D%27pci%27+managed%3D%27yes%27%3E%0A++
> ++++%3Csource%3E%0A++++++++%3Caddress+domain%3D%270x0000%
> 27+bus%3D%270xc1%27+slot%3D%270x00%27+function%3D%270x6%
> 27%2F%3E%0A++++++%3C%2Fsource%3E%0A++++++%3Caddress+type%3D%
> 27pci%27+domain%3D%270x0000%27+bus%3D%270x09%27+slot%3D%
> 270x00%27+function%3D%270x1%27%2F%3E%0A++++%3C%2Fhostdev%
> 3E%0A++++%3Cmemballoon+model%3D%27virtio%27%3E%0A++++++%
> 3Caddress+type%3D%27pci%27+domain%3D%270x0000%27+bus%3D%
> 270x04%27+slot%3D%270x00%27+function%3D%270x0%27%2F%3E%0A+
> +++%3C%2Fmemballoon%3E%0A++++%3Crng+model%3D%27virtio%27%3E%
> 0A++++++%3Cbackend+model%3D%27random%27%3E%2Fdev%
> 2Furandom%3C%2Fbackend%3E%0A++++++%3Caddress+type%3D%27pci%
> 27+domain%3D%270x0000%27+bus%3D%270x05%27+slot%3D%270x00%
> 27+function%3D%270x0%27%2F%3E%0A++++%3C%2Frng%3E&response=
> json&signature=<signature>
>
> Thanks,
>
> JeanPaul
> ________________________________
> From: JeanPaul van der Mijle <jp.vandermi...@hyperai.ai>
> Sent: Sunday, February 12, 2023 6:41 PM
> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> Subject: Re: PCI-E Passthrough
>
> Hi Wei,
>
> Thanks, going to take a deeper look into this.
>
> I assume I will have to add the list as following, to allow the XML below
> to be accepted by the API when I submit this encoded?
>
> I set this to be allowed for KVM: domain,devices,hostdev,source,
> address,memballoon,rng,backend
>
> It should reflect this XML part:
>
> <domain>
> <device>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x81' slot='0x00' function='0x2'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> function='0x0' multifunction='on'/>
> </hostdev>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0xc1' slot='0x00' function='0x2'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x09' slot='0x00'
> function='0x0' multifunction='on'/>
> </hostdev>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x81' slot='0x00' function='0x6'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> function='0x1'/>
> </hostdev>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0xc1' slot='0x00' function='0x6'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x09' slot='0x00'
> function='0x1'/>
> </hostdev>
> <memballoon model='virtio'>
> <address type='pci' domain='0x0000' bus='0x04' slot='0x00'
> function='0x0'/>
> </memballoon>
> <rng model='virtio'>
> <backend model='random'>/dev/urandom</backend>
> <address type='pci' domain='0x0000' bus='0x05' slot='0x00'
> function='0x0'/>
> </rng>
> </device>
> </domain>
>
> I only need to figure out if domain and device are required or not but I
> will see when I test it out.
>
> Thanks so far!
>
> JeanPaul
> ________________________________
> From: Wei ZHOU <ustcweiz...@gmail.com>
> Sent: Sunday, February 12, 2023 5:23 PM
> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> Subject: Re: PCI-E Passthrough
>
> I think you need to add additional configuration to vm.
> Refer to
> https://www.shapeblue.com/cloudstack-feature-first-look-
> enable-sending-of-arbitrary-configuration-data-to-vms/
>
> -Wei
>
>
>
> On Sunday, 12 February 2023, JeanPaul van der Mijle <
> jp.vandermi...@hyperai.ai> wrote:
>
> > Hi all,
> >
> > Does anyone have a good guide on adding a PCI-E card (A Mellanox
> Connect-X
> > 5 VF interface) to cloudstack?
> > I had this all running before in stock Qemu, and all settings are still
> > present for the card and VF's, I just need to know how the management
> sees
> > that I have additional cards that I have available to assign to VMs.
> >
> > I tried adding:
> > pci.devices=100GE,81:00.2|100GE,81:00.3|100GE,81:00.4|
> > 100GE,81:00.5|100GE,81:00.6|100GE,81:00.7|100GE,81:01.0|
> > 100GE,81:01.1|100GE,c1:00.2|100GE,c1:00.3|100GE,c1:00.4|
> > 100GE,c1:00.5|100GE,c1:00.6|100GE,c1:00.7|100GE,c1:01.0|100GE,c1:01.1
> >
> > However, after restarting the agent, I noticed that it added escape
> > characters:
> > pci.devices=100GE,81\:00.2|100GE,81\:00.3|100GE,81\:00.4|
> > 100GE,81\:00.5|100GE,81\:00.6|100GE,81\:00.7|100GE,81\:01.0|
> > 100GE,81\:01.1|100GE,c1\:00.2|100GE,c1\:00.3|100GE,c1\:00.4|
> > 100GE,c1\:00.5|100GE,c1\:00.6|100GE,c1\:00.7|100GE,c1\:01.0|
> 100GE,c1\:01.1
> >
> > Not sure if this is fine, but so far it was the only thing I could find
> > back on google, but this is 10 years ago. I suspect this is no longer
> > sufficient.
> > After restarting the agent, I do not see any changes unfortunately.
> >
> > Thanks!
> >
> > JeanPaul
> >
> >
>
