Re: [Xen-devel] pci-passthrough loses msi-x interrupts ability after domain destroy

[Pasi Kärkkäinen]

Hi,

On Fri, Sep 22, 2017 at 03:25:00PM -0400, Konrad Rzeszutek Wilk wrote:
> On Fri, Sep 22, 2017 at 09:35:40AM +0200, Sander Eikelenboom wrote:
> > On 22/09/17 04:09, Christopher Clark wrote:
> > > On Thu, Sep 21, 2017 at 1:27 PM, Sander Eikelenboom
> > >  wrote:
> > >>
> > >> On Thu, September 21, 2017, 10:39:52 AM, Roger Pau Monné wrote:
> > >>
> > >>> On Wed, Sep 20, 2017 at 03:50:35PM -0400, Jérôme Oufella wrote:
> > 
> >  I'm using PCI pass-through to map a PCIe (intel i210) controller into
> >  a HVM domain. The system uses xen-pciback to hide the appropriate PCI
> >  device from Dom0.
> > 
> >  When creating the HVM domain after an hypervisor cold boot, the HVM
> >  domain can access and use the PCIe controller without problem.
> > 
> >  However, if the HVM domain is destroyed then restarted, it won't be
> >  able to use the pass-through PCI device anymore. The PCI device is
> >  seen and can be mapped, however, the interrupts will not be passed to
> >  the HVM domain anymore (this is visible under a Linux guest as
> >  /proc/interrupts counters remain 0). The behavior on a Windows10 guest
> >  is the same.
> > 
> >  A few interesting hints I noticed:
> > 
> >  - On Dom0, 'lspci -vv' on that PCIe device between the "working" and
> >  the "muted interrupts" states, I noted a difference between the
> >  MSI-X caps:
> > 
> >  - Capabilities: [70] MSI-X: Enable- Count=5 Masked- <-- IRQs will work 
> >  if domain started
> >  + Capabilities: [70] MSI-X: Enable- Count=5 Masked+ <-- IRQs won't 
> >  work if domain started
> >  ^^^
> > >>
> > >>> IMHO it seems that either your device is not able to perform a reset
> > >>> successfully, or Linux is not correctly performing such reset. I don't
> > >>> think there's a lot that can be done from the Xen side.
> > >>
> > >> Unfortunately for a lot of pci-devices a simple reset as performed by 
> > >> default isn't enough,
> > >> but also almost none support a real pci FLR.
> > >>
> > >> In the distant past Konrad has made a patchset that implemented a bus 
> > >> reset and
> > >> reseting config space. (It piggy backed on already existing libxl 
> > >> mechanism of
> > >> trying to call on a syfs "do_flr" attribute which triggers pciback to 
> > >> perform
> > >> the busreset and rewrite of config space for the device.
> > >>
> > >> I use that patchset ever since for my pci-passtrough needs and it works 
> > >> pretty
> > >> well. I can shutdown an restart VM's with pci devices passed trhough 
> > >> (also AMD
> > >> Radeon graphic cards).
> > > 
> > > Just to confirm the utility of that piece of work: OpenXT also uses an
> > > extended version of that same patch to perform device reset for
> > > passthrough.
> > > 
> > > I've attached a copy of that OpenXT patch to this message and it can
> > > also be obtained from our git repository:
> > > https://github.com/OpenXT/xenclient-oe/blob/f8d3b282a87231d9ae717b13d506e8e7e28c78c4/recipes-kernel/linux/4.9/patches/thorough-reset-interface-to-pciback-s-sysfs.patch
> > > This version creates a sysfs node named "reset_device" and the OpenXT
> > > libxl toolstack is patched to use that node instead of "do_flr".
> > 
> > Nice to hear there are more users of this patch. On #xen on IRC there were 
> > from time to time
> > also users who tried pci-passtrough and ran into this issue (and probably 
> > abandonning the idea
> > since having to restart your host before being able to use your pass 
> > throughed device again
> > defies much of the use case).
> >  
> > > Konrad's original work encountered pushback on upstream acceptance at
> > > the time it was developed. I'm not sure I've found where that
> > > discussion ended. Is there any prospect of a more comprehensive reset
> > > mechanism being accepted into xen-pciback, or elsewhere in the kernel?
> > 
> > Yeah it was nacked by David Vrabel and the discussion somewhat bleeded to 
> > death. 
> > >From what i remember the main issue was with the naming, since it doesn't 
> > >do a FLR,
> > the sysfs hook shouldn't be called "do_flr".
> > 
> > Some other perhaps minor issues i can think of are:
> > - No way to excempt pci-devices from this new way of resetting them.
> >   Perhaps there could be pci devices/topologies were this way of
> >   resetting causes more problems than it solves and could cause a
> >   regression. Unfortunately auto detecting what works doesn't seem to
> >   be possible. On the other hand (though only with my n=10) i haven't 
> > encountered
> >   such a device yet.
> > 
> > - The communication path between libxl and the kernel via sysfs.
> >   I think the preference was for a:
> >   a) having it use a more common used Xen communication channel or
> >   b) having it all self-contained in pci-back. (from my memory and the 
> > openxt patch description
> >  there could be some 

Re: [Xen-devel] pci-passthrough loses msi-x interrupts ability after domain destroy

[Jan Beulich]

>>> On 25.09.17 at 18:10,  wrote:
> I tried the various device reset patches posted on this discussion
> (do_flr, Christopher's "more thorough" reset_device) but without luck.
> 
> After reset, I could notice that lspci shows the device's Masked
> state has been cleared but a newly started guest will still fail
> to receive any interrupts.

And that's even with most recent Xen and qemu (there were some
changes by Roger here a few weeks ago)? If so, I'm afraid is all
will remain guesswork without someone (you?) who is able to
see the issue also debugging it.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] pci-passthrough loses msi-x interrupts ability after domain destroy

[Jérôme Oufella]

Hi Jan and al,

- On Sep 21, 2017, at 9:12 AM, Jan Beulich jbeul...@suse.com wrote:

> And did you verify that the OS actually makes an attempt to clear
> this mask-all flag? If such an attempt doesn't have the intended
> effect, finding the problem location in the code and sending a
> fix can't be that difficult. If otoh the guest doesn't do this, then
> we'd need to figure out whether we leave the device in a wrong
> state after de-assigning it from the original guest instance (albeit,
> as Roger said, the reset the device is supposed to go through
> would be expected to clear it).

> I can certainly see an OS not
> necessarily expecting the bit to be set when first gaining control
> of the device. For this, look at the lspci output for the device in
> Dom0 between shutting down and then restarting the guest.
> 
> Jan

Thanks for the good hints and reset patches.

I tried the various device reset patches posted on this discussion
(do_flr, Christopher's "more thorough" reset_device) but without luck.

After reset, I could notice that lspci shows the device's Masked
state has been cleared but a newly started guest will still fail
to receive any interrupts.

However, I just found a sequence that makes the device survive a guest
destruction: issuing a remove_slot, unbind, bind, new_slot on that
specific device in pciback's sysfs corner, then re-creating the
domain. This allows interrupts flowing back into both windows and
linux domains, without involving an additional reset step.

It's a quick fix for me, but I'm wondering what path differs between
the two methods, and how what it leads to could become part of the
standard reset path.

Regards,
Jerome

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] pci-passthrough loses msi-x interrupts ability after domain destroy

[Ross Philipson]

>
> ​[snip]​
>
>
> > So i think David's NACK was mostly for the patchset having some hackish
> cosmetics.
>
> He didn't like 'do_flr' which made sense as the patchset did not do FLR.
> It made a bus-reset
> for more than one device (if those devices were assigned to pciback).
>

​When I first wrote this, FLR was not yet implemented in the kernel so I
was actually adding FLR support; thus the name do_flr. So that is just
cargo from years ago :)​



>
> >
> > On the upside one can conclude that this patchset is now pretty well
> tested over the years ;)
> >
> > Since David has left, perhaps Jurgen/Boris/Konrad could express their
> views (again) ?
> > (CC'ed them as well)
>
> I've asked Govinda (CC-ed) to refresh the patchset against the lastest
> kernel and
> repost it and see where it goes.
>
> >
> > > As noted in the original LKML threads, vfio has similar relevant pci
> > > device reset retry logic. (Thanks to Rich Persaud for this pointer:)
> > > http://elixir.free-electrons.com/linux/v4.14-rc1/source/
> drivers/vfio/pci/vfio_pci.c#L1353
> > >
> > > libvirt also performs similar reset logic, using a direct low level
> > > interface to config space (Thanks to Marek for this pointer, libvirt
> > > is used by Qubes:)
> > > https://github.com/libvirt/libvirt/blob/master/src/util/virpci.c#L929
> > > I thinks this indicates that it would be possible to extend libxl to
> > > do something similar, but that seems less satisfactory compared to
> > > performing the work in a kernel-provided implementation.
> > >
> > > Is there a way forward to providing this functionality within Xen
> > > software or Linux> Christopher
> > > --
> > >
> > > openxt.org
> > >
> >
>
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
>



-- 
Ross Philipson
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] pci-passthrough loses msi-x interrupts ability after domain destroy

[Roger Pau Monné]

On Fri, Sep 22, 2017 at 09:35:40AM +0200, Sander Eikelenboom wrote:
> - Not an issue back then when the patch was made, but as the question earlier 
> to Roger,
>   the hypervisor seems to grow more interference with pci devices with the 
> PVH dom0 work.
>   If and hoow does that relate to pci-back and pci-passthrough and (the 
> location of) resetting mechanisms ?

pci-back will still be required in order to do pci-passthrough to PV
guests. The aim is to use the new pci-emulation layer to perform
pci-passthrough to both HVM and PVH guests, deprecating the
passthrough code inside of QEMU.

I would prefer to have the reset mechanism inside of Xen, but I'm not
sure how feasible it is to put this code inside of Xen because I
haven't looked at it yet.

Roger.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] pci-passthrough loses msi-x interrupts ability after domain destroy

[Konrad Rzeszutek Wilk]

On Fri, Sep 22, 2017 at 09:35:40AM +0200, Sander Eikelenboom wrote:
> On 22/09/17 04:09, Christopher Clark wrote:
> > On Thu, Sep 21, 2017 at 1:27 PM, Sander Eikelenboom
> >  wrote:
> >>
> >> On Thu, September 21, 2017, 10:39:52 AM, Roger Pau Monné wrote:
> >>
> >>> On Wed, Sep 20, 2017 at 03:50:35PM -0400, Jérôme Oufella wrote:
> 
>  I'm using PCI pass-through to map a PCIe (intel i210) controller into
>  a HVM domain. The system uses xen-pciback to hide the appropriate PCI
>  device from Dom0.
> 
>  When creating the HVM domain after an hypervisor cold boot, the HVM
>  domain can access and use the PCIe controller without problem.
> 
>  However, if the HVM domain is destroyed then restarted, it won't be
>  able to use the pass-through PCI device anymore. The PCI device is
>  seen and can be mapped, however, the interrupts will not be passed to
>  the HVM domain anymore (this is visible under a Linux guest as
>  /proc/interrupts counters remain 0). The behavior on a Windows10 guest
>  is the same.
> 
>  A few interesting hints I noticed:
> 
>  - On Dom0, 'lspci -vv' on that PCIe device between the "working" and
>  the "muted interrupts" states, I noted a difference between the
>  MSI-X caps:
> 
>  - Capabilities: [70] MSI-X: Enable- Count=5 Masked- <-- IRQs will work 
>  if domain started
>  + Capabilities: [70] MSI-X: Enable- Count=5 Masked+ <-- IRQs won't work 
>  if domain started
>  ^^^
> >>
> >>> IMHO it seems that either your device is not able to perform a reset
> >>> successfully, or Linux is not correctly performing such reset. I don't
> >>> think there's a lot that can be done from the Xen side.
> >>
> >> Unfortunately for a lot of pci-devices a simple reset as performed by 
> >> default isn't enough,
> >> but also almost none support a real pci FLR.
> >>
> >> In the distant past Konrad has made a patchset that implemented a bus 
> >> reset and
> >> reseting config space. (It piggy backed on already existing libxl 
> >> mechanism of
> >> trying to call on a syfs "do_flr" attribute which triggers pciback to 
> >> perform
> >> the busreset and rewrite of config space for the device.
> >>
> >> I use that patchset ever since for my pci-passtrough needs and it works 
> >> pretty
> >> well. I can shutdown an restart VM's with pci devices passed trhough (also 
> >> AMD
> >> Radeon graphic cards).
> > 
> > Just to confirm the utility of that piece of work: OpenXT also uses an
> > extended version of that same patch to perform device reset for
> > passthrough.
> > 
> > I've attached a copy of that OpenXT patch to this message and it can
> > also be obtained from our git repository:
> > https://github.com/OpenXT/xenclient-oe/blob/f8d3b282a87231d9ae717b13d506e8e7e28c78c4/recipes-kernel/linux/4.9/patches/thorough-reset-interface-to-pciback-s-sysfs.patch
> > This version creates a sysfs node named "reset_device" and the OpenXT
> > libxl toolstack is patched to use that node instead of "do_flr".
> 
> Nice to hear there are more users of this patch. On #xen on IRC there were 
> from time to time
> also users who tried pci-passtrough and ran into this issue (and probably 
> abandonning the idea
> since having to restart your host before being able to use your pass 
> throughed device again
> defies much of the use case).
>  
> > Konrad's original work encountered pushback on upstream acceptance at
> > the time it was developed. I'm not sure I've found where that
> > discussion ended. Is there any prospect of a more comprehensive reset
> > mechanism being accepted into xen-pciback, or elsewhere in the kernel?
> 
> Yeah it was nacked by David Vrabel and the discussion somewhat bleeded to 
> death. 
> >From what i remember the main issue was with the naming, since it doesn't do 
> >a FLR,
> the sysfs hook shouldn't be called "do_flr".
> 
> Some other perhaps minor issues i can think of are:
> - No way to excempt pci-devices from this new way of resetting them.
>   Perhaps there could be pci devices/topologies were this way of
>   resetting causes more problems than it solves and could cause a
>   regression. Unfortunately auto detecting what works doesn't seem to
>   be possible. On the other hand (though only with my n=10) i haven't 
> encountered
>   such a device yet.
> 
> - The communication path between libxl and the kernel via sysfs.
>   I think the preference was for a:
>   a) having it use a more common used Xen communication channel or
>   b) having it all self-contained in pci-back. (from my memory and the openxt 
> patch description
>  there could be some locking issue when trying to implement it this way,
>  but the vfio guys had that solved for there reset implementation if i
>  from one of the comments in there source code (patches by Alex Williamson
>  if i remember correctly).
> 
> - Not an issue back then when 

Re: [Xen-devel] pci-passthrough loses msi-x interrupts ability after domain destroy

[Roger Pau Monné]

On Thu, Sep 21, 2017 at 10:27:01PM +0200, Sander Eikelenboom wrote:
> Roger: 
> I follow your PVH (dom0) patches shallowly, from my understanding it will 
> result 
> in Xen having more inteference with the handling of PCI devices ?

Yes, that's correct.

> If that's correct will this also impact the resetting logic, or will most 
> stay 
> in  the dom0 kernel/pciback ?

It's not clear, IMHO it would be better to handle the reset logic in
Xen itself, but I haven't looked into it to know whether that's
something feasible or not.

Roger.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] pci-passthrough loses msi-x interrupts ability after domain destroy

[Sander Eikelenboom]

On 22/09/17 04:09, Christopher Clark wrote:
> On Thu, Sep 21, 2017 at 1:27 PM, Sander Eikelenboom
>  wrote:
>>
>> On Thu, September 21, 2017, 10:39:52 AM, Roger Pau Monné wrote:
>>
>>> On Wed, Sep 20, 2017 at 03:50:35PM -0400, Jérôme Oufella wrote:

 I'm using PCI pass-through to map a PCIe (intel i210) controller into
 a HVM domain. The system uses xen-pciback to hide the appropriate PCI
 device from Dom0.

 When creating the HVM domain after an hypervisor cold boot, the HVM
 domain can access and use the PCIe controller without problem.

 However, if the HVM domain is destroyed then restarted, it won't be
 able to use the pass-through PCI device anymore. The PCI device is
 seen and can be mapped, however, the interrupts will not be passed to
 the HVM domain anymore (this is visible under a Linux guest as
 /proc/interrupts counters remain 0). The behavior on a Windows10 guest
 is the same.

 A few interesting hints I noticed:

 - On Dom0, 'lspci -vv' on that PCIe device between the "working" and
 the "muted interrupts" states, I noted a difference between the
 MSI-X caps:

 - Capabilities: [70] MSI-X: Enable- Count=5 Masked- <-- IRQs will work if 
 domain started
 + Capabilities: [70] MSI-X: Enable- Count=5 Masked+ <-- IRQs won't work if 
 domain started
 ^^^
>>
>>> IMHO it seems that either your device is not able to perform a reset
>>> successfully, or Linux is not correctly performing such reset. I don't
>>> think there's a lot that can be done from the Xen side.
>>
>> Unfortunately for a lot of pci-devices a simple reset as performed by 
>> default isn't enough,
>> but also almost none support a real pci FLR.
>>
>> In the distant past Konrad has made a patchset that implemented a bus reset 
>> and
>> reseting config space. (It piggy backed on already existing libxl mechanism 
>> of
>> trying to call on a syfs "do_flr" attribute which triggers pciback to perform
>> the busreset and rewrite of config space for the device.
>>
>> I use that patchset ever since for my pci-passtrough needs and it works 
>> pretty
>> well. I can shutdown an restart VM's with pci devices passed trhough (also 
>> AMD
>> Radeon graphic cards).
> 
> Just to confirm the utility of that piece of work: OpenXT also uses an
> extended version of that same patch to perform device reset for
> passthrough.
> 
> I've attached a copy of that OpenXT patch to this message and it can
> also be obtained from our git repository:
> https://github.com/OpenXT/xenclient-oe/blob/f8d3b282a87231d9ae717b13d506e8e7e28c78c4/recipes-kernel/linux/4.9/patches/thorough-reset-interface-to-pciback-s-sysfs.patch
> This version creates a sysfs node named "reset_device" and the OpenXT
> libxl toolstack is patched to use that node instead of "do_flr".

Nice to hear there are more users of this patch. On #xen on IRC there were from 
time to time
also users who tried pci-passtrough and ran into this issue (and probably 
abandonning the idea
since having to restart your host before being able to use your pass throughed 
device again
defies much of the use case).
 
> Konrad's original work encountered pushback on upstream acceptance at
> the time it was developed. I'm not sure I've found where that
> discussion ended. Is there any prospect of a more comprehensive reset
> mechanism being accepted into xen-pciback, or elsewhere in the kernel?

Yeah it was nacked by David Vrabel and the discussion somewhat bleeded to 
death. 
From what i remember the main issue was with the naming, since it doesn't do a 
FLR,
the sysfs hook shouldn't be called "do_flr".

Some other perhaps minor issues i can think of are:
- No way to excempt pci-devices from this new way of resetting them.
  Perhaps there could be pci devices/topologies were this way of
  resetting causes more problems than it solves and could cause a
  regression. Unfortunately auto detecting what works doesn't seem to
  be possible. On the other hand (though only with my n=10) i haven't 
encountered
  such a device yet.

- The communication path between libxl and the kernel via sysfs.
  I think the preference was for a:
  a) having it use a more common used Xen communication channel or
  b) having it all self-contained in pci-back. (from my memory and the openxt 
patch description
 there could be some locking issue when trying to implement it this way,
 but the vfio guys had that solved for there reset implementation if i
 from one of the comments in there source code (patches by Alex Williamson
 if i remember correctly).

- Not an issue back then when the patch was made, but as the question earlier 
to Roger,
  the hypervisor seems to grow more interference with pci devices with the PVH 
dom0 work.
  If and hoow does that relate to pci-back and pci-passthrough and (the 
location of) resetting mechanisms ?


So i think David's 

Re: [Xen-devel] pci-passthrough loses msi-x interrupts ability after domain destroy

[Pasi Kärkkäinen]

Hi,

On Thu, Sep 21, 2017 at 07:09:12PM -0700, Christopher Clark wrote:
> On Thu, Sep 21, 2017 at 1:27 PM, Sander Eikelenboom
>  wrote:
> >
> > On Thu, September 21, 2017, 10:39:52 AM, Roger Pau Monné wrote:
> >
> >> On Wed, Sep 20, 2017 at 03:50:35PM -0400, Jérôme Oufella wrote:
> >>>
> >>> I'm using PCI pass-through to map a PCIe (intel i210) controller into
> >>> a HVM domain. The system uses xen-pciback to hide the appropriate PCI
> >>> device from Dom0.
> >>>
> >>> When creating the HVM domain after an hypervisor cold boot, the HVM
> >>> domain can access and use the PCIe controller without problem.
> >>>
> >>> However, if the HVM domain is destroyed then restarted, it won't be
> >>> able to use the pass-through PCI device anymore. The PCI device is
> >>> seen and can be mapped, however, the interrupts will not be passed to
> >>> the HVM domain anymore (this is visible under a Linux guest as
> >>> /proc/interrupts counters remain 0). The behavior on a Windows10 guest
> >>> is the same.
> >>>
> >>> A few interesting hints I noticed:
> >>>
> >>> - On Dom0, 'lspci -vv' on that PCIe device between the "working" and
> >>> the "muted interrupts" states, I noted a difference between the
> >>> MSI-X caps:
> >>>
> >>> - Capabilities: [70] MSI-X: Enable- Count=5 Masked- <-- IRQs will work if 
> >>> domain started
> >>> + Capabilities: [70] MSI-X: Enable- Count=5 Masked+ <-- IRQs won't work 
> >>> if domain started
> >>> ^^^
> >
> >> IMHO it seems that either your device is not able to perform a reset
> >> successfully, or Linux is not correctly performing such reset. I don't
> >> think there's a lot that can be done from the Xen side.
> >
> > Unfortunately for a lot of pci-devices a simple reset as performed by 
> > default isn't enough,
> > but also almost none support a real pci FLR.
> >
> > In the distant past Konrad has made a patchset that implemented a bus reset 
> > and
> > reseting config space. (It piggy backed on already existing libxl mechanism 
> > of
> > trying to call on a syfs "do_flr" attribute which triggers pciback to 
> > perform
> > the busreset and rewrite of config space for the device.
> >
> > I use that patchset ever since for my pci-passtrough needs and it works 
> > pretty
> > well. I can shutdown an restart VM's with pci devices passed trhough (also 
> > AMD
> > Radeon graphic cards).
> 
> Just to confirm the utility of that piece of work: OpenXT also uses an
> extended version of that same patch to perform device reset for
> passthrough.
> 
> I've attached a copy of that OpenXT patch to this message and it can
> also be obtained from our git repository:
> https://github.com/OpenXT/xenclient-oe/blob/f8d3b282a87231d9ae717b13d506e8e7e28c78c4/recipes-kernel/linux/4.9/patches/thorough-reset-interface-to-pciback-s-sysfs.patch
> This version creates a sysfs node named "reset_device" and the OpenXT
> libxl toolstack is patched to use that node instead of "do_flr".
> 
> Konrad's original work encountered pushback on upstream acceptance at
> the time it was developed. I'm not sure I've found where that
> discussion ended. Is there any prospect of a more comprehensive reset
> mechanism being accepted into xen-pciback, or elsewhere in the kernel?
> 
> As noted in the original LKML threads, vfio has similar relevant pci
> device reset retry logic. (Thanks to Rich Persaud for this pointer:)
> http://elixir.free-electrons.com/linux/v4.14-rc1/source/drivers/vfio/pci/vfio_pci.c#L1353
> 
> libvirt also performs similar reset logic, using a direct low level
> interface to config space (Thanks to Marek for this pointer, libvirt
> is used by Qubes:)
> https://github.com/libvirt/libvirt/blob/master/src/util/virpci.c#L929
> I thinks this indicates that it would be possible to extend libxl to
> do something similar, but that seems less satisfactory compared to
> performing the work in a kernel-provided implementation.
> 
> Is there a way forward to providing this functionality within Xen
> software or Linux?
> 

Adding Konrad to CC ..


-- Pasi

> Christopher
> --
> 
> openxt.org

> From d686351d8ea4a1ea1d755d0a10f6f14d1c870911 Mon Sep 17 00:00:00 2001
> From: Kyle Temkin 
> Date: Wed, 8 Apr 2015 00:58:24 -0400
> Subject: [PATCH] Add thorough reset interface to pciback's sysfs.
> 
> 
> SHORT DESCRIPTION:
> 
> Adds an interface that allows "more thorough" resets to be performed
> on devices which don't support Function Level Resets (FLRs). This
> interface should allow the toolstack to ensure that a PCI device is in a
> known state prior to passing it through to a VM.
> 
> 
> LONG DESCRIPTION:
> 
> 
> 

Re: [Xen-devel] pci-passthrough loses msi-x interrupts ability after domain destroy

[Christopher Clark]

On Thu, Sep 21, 2017 at 1:27 PM, Sander Eikelenboom
 wrote:
>
> On Thu, September 21, 2017, 10:39:52 AM, Roger Pau Monné wrote:
>
>> On Wed, Sep 20, 2017 at 03:50:35PM -0400, Jérôme Oufella wrote:
>>>
>>> I'm using PCI pass-through to map a PCIe (intel i210) controller into
>>> a HVM domain. The system uses xen-pciback to hide the appropriate PCI
>>> device from Dom0.
>>>
>>> When creating the HVM domain after an hypervisor cold boot, the HVM
>>> domain can access and use the PCIe controller without problem.
>>>
>>> However, if the HVM domain is destroyed then restarted, it won't be
>>> able to use the pass-through PCI device anymore. The PCI device is
>>> seen and can be mapped, however, the interrupts will not be passed to
>>> the HVM domain anymore (this is visible under a Linux guest as
>>> /proc/interrupts counters remain 0). The behavior on a Windows10 guest
>>> is the same.
>>>
>>> A few interesting hints I noticed:
>>>
>>> - On Dom0, 'lspci -vv' on that PCIe device between the "working" and
>>> the "muted interrupts" states, I noted a difference between the
>>> MSI-X caps:
>>>
>>> - Capabilities: [70] MSI-X: Enable- Count=5 Masked- <-- IRQs will work if 
>>> domain started
>>> + Capabilities: [70] MSI-X: Enable- Count=5 Masked+ <-- IRQs won't work if 
>>> domain started
>>> ^^^
>
>> IMHO it seems that either your device is not able to perform a reset
>> successfully, or Linux is not correctly performing such reset. I don't
>> think there's a lot that can be done from the Xen side.
>
> Unfortunately for a lot of pci-devices a simple reset as performed by default 
> isn't enough,
> but also almost none support a real pci FLR.
>
> In the distant past Konrad has made a patchset that implemented a bus reset 
> and
> reseting config space. (It piggy backed on already existing libxl mechanism of
> trying to call on a syfs "do_flr" attribute which triggers pciback to perform
> the busreset and rewrite of config space for the device.
>
> I use that patchset ever since for my pci-passtrough needs and it works pretty
> well. I can shutdown an restart VM's with pci devices passed trhough (also AMD
> Radeon graphic cards).

Just to confirm the utility of that piece of work: OpenXT also uses an
extended version of that same patch to perform device reset for
passthrough.

I've attached a copy of that OpenXT patch to this message and it can
also be obtained from our git repository:
https://github.com/OpenXT/xenclient-oe/blob/f8d3b282a87231d9ae717b13d506e8e7e28c78c4/recipes-kernel/linux/4.9/patches/thorough-reset-interface-to-pciback-s-sysfs.patch
This version creates a sysfs node named "reset_device" and the OpenXT
libxl toolstack is patched to use that node instead of "do_flr".

Konrad's original work encountered pushback on upstream acceptance at
the time it was developed. I'm not sure I've found where that
discussion ended. Is there any prospect of a more comprehensive reset
mechanism being accepted into xen-pciback, or elsewhere in the kernel?

As noted in the original LKML threads, vfio has similar relevant pci
device reset retry logic. (Thanks to Rich Persaud for this pointer:)
http://elixir.free-electrons.com/linux/v4.14-rc1/source/drivers/vfio/pci/vfio_pci.c#L1353

libvirt also performs similar reset logic, using a direct low level
interface to config space (Thanks to Marek for this pointer, libvirt
is used by Qubes:)
https://github.com/libvirt/libvirt/blob/master/src/util/virpci.c#L929
I thinks this indicates that it would be possible to extend libxl to
do something similar, but that seems less satisfactory compared to
performing the work in a kernel-provided implementation.

Is there a way forward to providing this functionality within Xen
software or Linux?

Christopher
--

openxt.org
From d686351d8ea4a1ea1d755d0a10f6f14d1c870911 Mon Sep 17 00:00:00 2001
From: Kyle Temkin 
Date: Wed, 8 Apr 2015 00:58:24 -0400
Subject: [PATCH] Add thorough reset interface to pciback's sysfs.


SHORT DESCRIPTION:

Adds an interface that allows "more thorough" resets to be performed
on devices which don't support Function Level Resets (FLRs). This
interface should allow the toolstack to ensure that a PCI device is in a
known state prior to passing it through to a VM.


LONG DESCRIPTION:


From Konrad Rzeszutek Wilk's original post to xen-devel and the LKML:

  The life-cycle of a PCI device in Xen pciback is complex
  and is constrained by the PCI generic locking mechanism.

  It starts with the device being binded to us - for which
  we do a device function reset (and done via SysFS
  so the PCI lock is held)

  If 

Re: [Xen-devel] pci-passthrough loses msi-x interrupts ability after domain destroy

[Sander Eikelenboom]

Thursday, September 21, 2017, 10:39:52 AM, you wrote:

> On Wed, Sep 20, 2017 at 03:50:35PM -0400, Jérôme Oufella wrote:
>> Hi Xen-devel, 
>> 
>> I'm using PCI pass-through to map a PCIe (intel i210) controller into 
>> a HVM domain. The system uses xen-pciback to hide the appropriate PCI 
>> device from Dom0. 
>> 
>> When creating the HVM domain after an hypervisor cold boot, the HVM 
>> domain can access and use the PCIe controller without problem. 
>> 
>> However, if the HVM domain is destroyed then restarted, it won't be 
>> able to use the pass-through PCI device anymore. The PCI device is 
>> seen and can be mapped, however, the interrupts will not be passed to 
>> the HVM domain anymore (this is visible under a Linux guest as 
>> /proc/interrupts counters remain 0). The behavior on a Windows10 guest 
>> is the same. 
>> 
>> A few interesting hints I noticed: 
>> 
>> - On Dom0, 'lspci -vv' on that PCIe device between the "working" and 
>> the "muted interrupts" states, I noted a difference between the 
>> MSI-X caps: 
>> 
>> - Capabilities: [70] MSI-X: Enable- Count=5 Masked- <-- IRQs will work if 
>> domain started 
>> + Capabilities: [70] MSI-X: Enable- Count=5 Masked+ <-- IRQs won't work if 
>> domain started
>> ^^^

> IMHO it seems that either your device is not able to perform a reset
> successfully, or Linux is not correctly performing such reset. I don't
> think there's a lot that can be done from the Xen side.

Unfortunately for a lot of pci-devices a simple reset as performed by default 
isn't enough, 
but also almost none support a real pci FLR.
 
In the distant past Konrad has made a patchset that implemented a bus reset and
reseting config space. (It piggy backed on already existing libxl mechanism of 
trying to call on a syfs "do_flr" attribute which triggers pciback to perform 
the busreset and rewrite of config space for the device.

I use that patchset ever since for my pci-passtrough needs and it works pretty 
well. I can shutdown an restart VM's with pci devices passed trhough (also AMD 
Radeon graphic cards).

Jérôme:
Although your mileage may vary, you could try the attached patch, it's
for a > 4.9 linux kernel's pciback, although it probably also would apply with 
for a > minimal adjustments (mostly non matching line numbers) to an some what 
earlier kernel. 

If it works for you as well, perhaps it deserves a mention on the Xen wiki in 
the 
pci-passtrough section.

Roger: 
I follow your PVH (dom0) patches shallowly, from my understanding it will 
result 
in Xen having more inteference with the handling of PCI devices ? 
If that's correct will this also impact the resetting logic, or will most stay 
in  the dom0 kernel/pciback ?

--
Sander

> Thanks, Roger.



pciback_do_flr_4_9.patch
Description: Binary data
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] pci-passthrough loses msi-x interrupts ability after domain destroy

[Jan Beulich]

>>> On 20.09.17 at 21:50,  wrote:
> - On Dom0, 'lspci -vv' on that PCIe device between the "working" and 
> the "muted interrupts" states, I noted a difference between the 
> MSI-X caps: 
> 
> - Capabilities: [70] MSI-X: Enable- Count=5 Masked- <-- IRQs will work if 
> domain started 
> + Capabilities: [70] MSI-X: Enable- Count=5 Masked+ <-- IRQs won't work if 
> domain started
> ^^^

And did you verify that the OS actually makes an attempt to clear
this mask-all flag? If such an attempt doesn't have the intended
effect, finding the problem location in the code and sending a
fix can't be that difficult. If otoh the guest doesn't do this, then
we'd need to figure out whether we leave the device in a wrong
state after de-assigning it from the original guest instance (albeit,
as Roger said, the reset the device is supposed to go through
would be expected to clear it). I can certainly see an OS not
necessarily expecting the bit to be set when first gaining control
of the device. For this, look at the lspci output for the device in
Dom0 between shutting down and then restarting the guest.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] pci-passthrough loses msi-x interrupts ability after domain destroy

[Roger Pau Monné]

On Wed, Sep 20, 2017 at 03:50:35PM -0400, Jérôme Oufella wrote:
> Hi Xen-devel, 
> 
> I'm using PCI pass-through to map a PCIe (intel i210) controller into 
> a HVM domain. The system uses xen-pciback to hide the appropriate PCI 
> device from Dom0. 
> 
> When creating the HVM domain after an hypervisor cold boot, the HVM 
> domain can access and use the PCIe controller without problem. 
> 
> However, if the HVM domain is destroyed then restarted, it won't be 
> able to use the pass-through PCI device anymore. The PCI device is 
> seen and can be mapped, however, the interrupts will not be passed to 
> the HVM domain anymore (this is visible under a Linux guest as 
> /proc/interrupts counters remain 0). The behavior on a Windows10 guest 
> is the same. 
> 
> A few interesting hints I noticed: 
> 
> - On Dom0, 'lspci -vv' on that PCIe device between the "working" and 
> the "muted interrupts" states, I noted a difference between the 
> MSI-X caps: 
> 
> - Capabilities: [70] MSI-X: Enable- Count=5 Masked- <-- IRQs will work if 
> domain started 
> + Capabilities: [70] MSI-X: Enable- Count=5 Masked+ <-- IRQs won't work if 
> domain started
> ^^^

IMHO it seems that either your device is not able to perform a reset
successfully, or Linux is not correctly performing such reset. I don't
think there's a lot that can be done from the Xen side.

Thanks, Roger.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel