RE: [PATCH] intel-iommu: Turn off translations at shutdown
> -Original Message- > On Sun, Nov 10, 2019 at 10:24 AM Deepa Dinamani > > I've posted the v2 without the conditional for now: > > https://lore.kernel.org/patchwork/patch/1151225/ > > > > As a side topic, I'm trying to support https://www.linuxboot.org/. I > > have a couple of more such cleanups coming. The VMM live updates and > > linuxboot seem to have contradicting requirements and they both use > > kexec. So kexec_in_progress doesn't seem like a sufficient indicator > > to distinguish between the two. Do you already have an idea on how to > > distiguish between them? Does a separate sys_reboot() command > > parameter sound ok? Or, we could use the flags in the sys_kexec_load() > > depending on how the live update feature is implemented. > > Also, the AMD driver disables iommu at shutdown already. So the live update > feature is already broken on AMD. > Hi Deepa, I think you may not need to consider too much VMM live update here (although it would be good to consider possible future features), after all it is an on-going effort, we are still not quite sure what exact modifications it needs. The VMM live update itself will figure out what is the best way to modify the code. Thanks, Jason > -Deepa ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH] intel-iommu: Turn off translations at shutdown
> -Original Message- > > > > For VMM live update case, we should be able to detect and bypass > > > > the shutdown that Deepa introduced here, so keep IOMMU still > operating? > > > > > > Is that a 'yes' to Deepa's "if someone wants to make it conditional, > > > we can do that" ? > > > > Yes, I think so. Thanks! > > Are these changes already part of the kernel like avoiding shutdown of the > passthrough devices? device_shutdown() doesn't seem to be doing anything > selectively as of now. > No, it is not in upstream yet. It is an on-going effort, which tries to kexec reboot the host while keeping IOMMU still working. Thanks, Jason > Thanks, > Deepa ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH] intel-iommu: Turn off translations at shutdown
On Sun, Nov 10, 2019 at 10:24 AM Deepa Dinamani wrote: > > On Fri, Nov 8, 2019 at 2:48 PM Deepa Dinamani wrote: > > > > > > > For VMM live update case, we should be able to detect and bypass > > > > > the shutdown that Deepa introduced here, so keep IOMMU still > > > > > operating? > > > > > > > > Is that a 'yes' to Deepa's "if someone wants to make it conditional, we > > > > can do that" ? > > > > > > Yes, I think so. Thanks! > > > > Are these changes already part of the kernel like avoiding shutdown of > > the passthrough devices? device_shutdown() doesn't seem to be doing > > anything selectively as of now. > > I've posted the v2 without the conditional for now: > https://lore.kernel.org/patchwork/patch/1151225/ > > As a side topic, I'm trying to support https://www.linuxboot.org/. I > have a couple of more such cleanups coming. The VMM live updates and > linuxboot seem to have contradicting requirements and they both use > kexec. So kexec_in_progress doesn't seem like a sufficient indicator > to distinguish between the two. Do you already have an idea on how to > distiguish between them? Does a separate sys_reboot() command > parameter sound ok? Or, we could use the flags in the sys_kexec_load() > depending on how the live update feature is implemented. Also, the AMD driver disables iommu at shutdown already. So the live update feature is already broken on AMD. -Deepa ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH] intel-iommu: Turn off translations at shutdown
On Fri, Nov 8, 2019 at 2:48 PM Deepa Dinamani wrote: > > > > > For VMM live update case, we should be able to detect and bypass > > > > the shutdown that Deepa introduced here, so keep IOMMU still operating? > > > > > > Is that a 'yes' to Deepa's "if someone wants to make it conditional, we > > > can do that" ? > > > > Yes, I think so. Thanks! > > Are these changes already part of the kernel like avoiding shutdown of > the passthrough devices? device_shutdown() doesn't seem to be doing > anything selectively as of now. I've posted the v2 without the conditional for now: https://lore.kernel.org/patchwork/patch/1151225/ As a side topic, I'm trying to support https://www.linuxboot.org/. I have a couple of more such cleanups coming. The VMM live updates and linuxboot seem to have contradicting requirements and they both use kexec. So kexec_in_progress doesn't seem like a sufficient indicator to distinguish between the two. Do you already have an idea on how to distiguish between them? Does a separate sys_reboot() command parameter sound ok? Or, we could use the flags in the sys_kexec_load() depending on how the live update feature is implemented. -Deepa ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH] intel-iommu: Turn off translations at shutdown
Hi, On 11/9/19 6:28 AM, Deepa Dinamani wrote: + x86_platform.iommu_shutdown = intel_iommu_shutdown; How about moving it to detect_intel_iommu() in drivers/iommu/dmar.c? And Ok, makes sense to move it along with the init handler. make sure that it's included with CONFIG_X86_64. You mean CONFIG_X86 like the init that is already there? #ifdef CONFIG_X86 if (!ret) x86_init.iommu.iommu_init = intel_iommu_init; #endif Yes. Also, change the title to "iommu/vt-d: Turn off ..." Best regards, baolu ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH] intel-iommu: Turn off translations at shutdown
> > > For VMM live update case, we should be able to detect and bypass > > > the shutdown that Deepa introduced here, so keep IOMMU still operating? > > > > Is that a 'yes' to Deepa's "if someone wants to make it conditional, we > > can do that" ? > > Yes, I think so. Thanks! Are these changes already part of the kernel like avoiding shutdown of the passthrough devices? device_shutdown() doesn't seem to be doing anything selectively as of now. Thanks, Deepa ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH] intel-iommu: Turn off translations at shutdown
> > + x86_platform.iommu_shutdown = intel_iommu_shutdown; > > How about moving it to detect_intel_iommu() in drivers/iommu/dmar.c? And Ok, makes sense to move it along with the init handler. > make sure that it's included with CONFIG_X86_64. You mean CONFIG_X86 like the init that is already there? #ifdef CONFIG_X86 if (!ret) x86_init.iommu.iommu_init = intel_iommu_init; #endif Thanks, -Deepa ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH] intel-iommu: Turn off translations at shutdown
> -Original Message- > From: David Woodhouse > Sent: Friday, November 8, 2019 3:54 PM > To: Deepa Dinamani ; j...@8bytes.org; linux- > ker...@vger.kernel.org > Cc: iommu@lists.linux-foundation.org; Zeng, Jason ; > Tian, Kevin > Subject: Re: [PATCH] intel-iommu: Turn off translations at shutdown > > On Thu, 2019-11-07 at 12:59 -0800, Deepa Dinamani wrote: > > The intel-iommu driver assumes that the iommu state is > > cleaned up at the start of the new kernel. > > But, when we try to kexec boot something other than the > > Linux kernel, the cleanup cannot be relied upon. > > Hence, cleanup before we go down for reboot. > > > > Keeping the cleanup at initialization also, in case BIOS > > leaves the IOMMU enabled. > > > > I considered turning off iommu only during kexec reboot, > > but a clean shutdown seems always a good idea. But if > > someone wants to make it conditional, we can do that. > > This is going to break things for the VMM live update scheme that Jason > presented at KVM Forum, isn't it? > > In that case we rely on the IOMMU still operating during the > transition. For VMM live update case, we should be able to detect and bypass the shutdown that Deepa introduced here, so keep IOMMU still operating? Thanks, Jason ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
RE: [PATCH] intel-iommu: Turn off translations at shutdown
> -Original Message- > From: David Woodhouse > Sent: Friday, November 8, 2019 4:57 PM > To: Zeng, Jason ; Deepa Dinamani > ; j...@8bytes.org; linux-ker...@vger.kernel.org > Cc: iommu@lists.linux-foundation.org; Tian, Kevin > Subject: Re: [PATCH] intel-iommu: Turn off translations at shutdown > > On Fri, 2019-11-08 at 08:47 +, Zeng, Jason wrote: > > > -Original Message- > > > From: David Woodhouse > > > Sent: Friday, November 8, 2019 3:54 PM > > > To: Deepa Dinamani ; j...@8bytes.org; > linux- > > > ker...@vger.kernel.org > > > Cc: iommu@lists.linux-foundation.org; Zeng, Jason > ; > > > Tian, Kevin > > > Subject: Re: [PATCH] intel-iommu: Turn off translations at shutdown > > > > > > On Thu, 2019-11-07 at 12:59 -0800, Deepa Dinamani wrote: > > > > The intel-iommu driver assumes that the iommu state is > > > > cleaned up at the start of the new kernel. > > > > But, when we try to kexec boot something other than the > > > > Linux kernel, the cleanup cannot be relied upon. > > > > Hence, cleanup before we go down for reboot. > > > > > > > > Keeping the cleanup at initialization also, in case BIOS > > > > leaves the IOMMU enabled. > > > > > > > > I considered turning off iommu only during kexec reboot, > > > > but a clean shutdown seems always a good idea. But if > > > > someone wants to make it conditional, we can do that. > > > > > > This is going to break things for the VMM live update scheme that Jason > > > presented at KVM Forum, isn't it? > > > > > > In that case we rely on the IOMMU still operating during the > > > transition. > > > > For VMM live update case, we should be able to detect and bypass > > the shutdown that Deepa introduced here, so keep IOMMU still operating? > > Is that a 'yes' to Deepa's "if someone wants to make it conditional, we > can do that" ? Yes, I think so. Thanks! ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH] intel-iommu: Turn off translations at shutdown
On Fri, 2019-11-08 at 08:47 +, Zeng, Jason wrote: > > -Original Message- > > From: David Woodhouse > > Sent: Friday, November 8, 2019 3:54 PM > > To: Deepa Dinamani ; j...@8bytes.org; linux- > > ker...@vger.kernel.org > > Cc: iommu@lists.linux-foundation.org; Zeng, Jason ; > > Tian, Kevin > > Subject: Re: [PATCH] intel-iommu: Turn off translations at shutdown > > > > On Thu, 2019-11-07 at 12:59 -0800, Deepa Dinamani wrote: > > > The intel-iommu driver assumes that the iommu state is > > > cleaned up at the start of the new kernel. > > > But, when we try to kexec boot something other than the > > > Linux kernel, the cleanup cannot be relied upon. > > > Hence, cleanup before we go down for reboot. > > > > > > Keeping the cleanup at initialization also, in case BIOS > > > leaves the IOMMU enabled. > > > > > > I considered turning off iommu only during kexec reboot, > > > but a clean shutdown seems always a good idea. But if > > > someone wants to make it conditional, we can do that. > > > > This is going to break things for the VMM live update scheme that Jason > > presented at KVM Forum, isn't it? > > > > In that case we rely on the IOMMU still operating during the > > transition. > > For VMM live update case, we should be able to detect and bypass > the shutdown that Deepa introduced here, so keep IOMMU still operating? Is that a 'yes' to Deepa's "if someone wants to make it conditional, we can do that" ? smime.p7s Description: S/MIME cryptographic signature ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH] intel-iommu: Turn off translations at shutdown
On Thu, 2019-11-07 at 12:59 -0800, Deepa Dinamani wrote: > The intel-iommu driver assumes that the iommu state is > cleaned up at the start of the new kernel. > But, when we try to kexec boot something other than the > Linux kernel, the cleanup cannot be relied upon. > Hence, cleanup before we go down for reboot. > > Keeping the cleanup at initialization also, in case BIOS > leaves the IOMMU enabled. > > I considered turning off iommu only during kexec reboot, > but a clean shutdown seems always a good idea. But if > someone wants to make it conditional, we can do that. This is going to break things for the VMM live update scheme that Jason presented at KVM Forum, isn't it? In that case we rely on the IOMMU still operating during the transition. smime.p7s Description: S/MIME cryptographic signature ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH] intel-iommu: Turn off translations at shutdown
Hi, On 11/8/19 5:27 AM, Deepa Dinamani wrote: On Thu, Nov 7, 2019 at 12:59 PM Deepa Dinamani wrote: +static void intel_iommu_shutdown(void) + if (no_iommu || dmar_disabled) + return; This check is actually not required here, as the handler is only installed after these have been checked in intel_iommu_init. I can remove this in the next version of the patch, but I'll wait a few days for comments. This is probably still necessary if moving to detect_intel_iommu(). Best regards, baolu -Deepa ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH] intel-iommu: Turn off translations at shutdown
Hi, On 11/8/19 4:59 AM, Deepa Dinamani wrote: The intel-iommu driver assumes that the iommu state is cleaned up at the start of the new kernel. But, when we try to kexec boot something other than the Linux kernel, the cleanup cannot be relied upon. Hence, cleanup before we go down for reboot. Keeping the cleanup at initialization also, in case BIOS leaves the IOMMU enabled. I considered turning off iommu only during kexec reboot, but a clean shutdown seems always a good idea. But if someone wants to make it conditional, we can do that. Tested that before, the info message 'DMAR: Translation was enabled for but we are not in kdump mode' would be reported for each iommu. The message will not appear when the DMA-remapping is not enabled on entry to the kernel. Signed-off-by: Deepa Dinamani --- drivers/iommu/intel-iommu.c | 22 ++ 1 file changed, 22 insertions(+) diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c index fe8097078669..f0636b263722 100644 --- a/drivers/iommu/intel-iommu.c +++ b/drivers/iommu/intel-iommu.c @@ -4764,6 +4764,26 @@ static void intel_disable_iommus(void) iommu_disable_translation(iommu); } +static void intel_iommu_shutdown(void) +{ + struct dmar_drhd_unit *drhd; + struct intel_iommu *iommu = NULL; + + if (no_iommu || dmar_disabled) + return; + + down_write(&dmar_global_lock); + + /* Disable PMRs explicitly here. */ + for_each_iommu(iommu, drhd) + iommu_disable_protect_mem_regions(iommu); + + /* Make sure the IOMMUs are switched off */ + intel_disable_iommus(); + + up_write(&dmar_global_lock); +} + static inline struct intel_iommu *dev_to_intel_iommu(struct device *dev) { struct iommu_device *iommu_dev = dev_to_iommu_device(dev); @@ -5013,6 +5033,8 @@ int __init intel_iommu_init(void) } up_write(&dmar_global_lock); + x86_platform.iommu_shutdown = intel_iommu_shutdown; How about moving it to detect_intel_iommu() in drivers/iommu/dmar.c? And make sure that it's included with CONFIG_X86_64. Best regards, baolu + #if defined(CONFIG_X86) && defined(CONFIG_SWIOTLB) /* * If the system has no untrusted device or the user has decided ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH] intel-iommu: Turn off translations at shutdown
On Thu, Nov 7, 2019 at 12:59 PM Deepa Dinamani wrote: > +static void intel_iommu_shutdown(void) > + if (no_iommu || dmar_disabled) > + return; This check is actually not required here, as the handler is only installed after these have been checked in intel_iommu_init. I can remove this in the next version of the patch, but I'll wait a few days for comments. -Deepa ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
[PATCH] intel-iommu: Turn off translations at shutdown
The intel-iommu driver assumes that the iommu state is cleaned up at the start of the new kernel. But, when we try to kexec boot something other than the Linux kernel, the cleanup cannot be relied upon. Hence, cleanup before we go down for reboot. Keeping the cleanup at initialization also, in case BIOS leaves the IOMMU enabled. I considered turning off iommu only during kexec reboot, but a clean shutdown seems always a good idea. But if someone wants to make it conditional, we can do that. Tested that before, the info message 'DMAR: Translation was enabled for but we are not in kdump mode' would be reported for each iommu. The message will not appear when the DMA-remapping is not enabled on entry to the kernel. Signed-off-by: Deepa Dinamani --- drivers/iommu/intel-iommu.c | 22 ++ 1 file changed, 22 insertions(+) diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c index fe8097078669..f0636b263722 100644 --- a/drivers/iommu/intel-iommu.c +++ b/drivers/iommu/intel-iommu.c @@ -4764,6 +4764,26 @@ static void intel_disable_iommus(void) iommu_disable_translation(iommu); } +static void intel_iommu_shutdown(void) +{ + struct dmar_drhd_unit *drhd; + struct intel_iommu *iommu = NULL; + + if (no_iommu || dmar_disabled) + return; + + down_write(&dmar_global_lock); + + /* Disable PMRs explicitly here. */ + for_each_iommu(iommu, drhd) + iommu_disable_protect_mem_regions(iommu); + + /* Make sure the IOMMUs are switched off */ + intel_disable_iommus(); + + up_write(&dmar_global_lock); +} + static inline struct intel_iommu *dev_to_intel_iommu(struct device *dev) { struct iommu_device *iommu_dev = dev_to_iommu_device(dev); @@ -5013,6 +5033,8 @@ int __init intel_iommu_init(void) } up_write(&dmar_global_lock); + x86_platform.iommu_shutdown = intel_iommu_shutdown; + #if defined(CONFIG_X86) && defined(CONFIG_SWIOTLB) /* * If the system has no untrusted device or the user has decided -- 2.17.1 ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu