Re: [PATCH 0/3] irqchip: GIC kexec/kdump improvement and workarounds
On Wed, 14 Mar 2018, Marc Zyngier wrote: > On 14/03/18 17:11, Thomas Gleixner wrote: > > Makes sense. Do we have any indicator that tells us that a particular irq > > chip is missing something in the init code or do we have to rely on crash > > reports? > A way to work out what is potentially missing would be to make sure that > whatever we're removing from machine_kexec_mask_interrupts, we can find > it in the irqchip init code. Not an easy task, and certainly not perfect > (patches 1 and 2 in this series have no equivalent in the kexec code). > > There is still another category of "reset" stuff that belongs to the > teardown path, and that's for things that may have an impact on the > secondary kernel. > > The case I have in mind is that of the GIC LPI pending tables. These are > allocated to the GIC, which can write pending bits at any time. Think of > it as a DMA engine. At the moment we enter the secondary kernel, we must > make sure the GIC has already been shut down, as the table memory will > be reallocated. Yes, you surely need to prevent that. Thanks, tglx
Re: [PATCH 0/3] irqchip: GIC kexec/kdump improvement and workarounds
On 14/03/18 17:11, Thomas Gleixner wrote: > On Wed, 14 Mar 2018, Mark Rutland wrote: >> On Tue, Mar 13, 2018 at 06:35:07PM +, Marc Zyngier wrote: >>> On 13/03/18 17:51, Mark Rutland wrote: On Tue, Mar 13, 2018 at 05:21:00PM +, Marc Zyngier wrote: > As kexec and kdump are getting used a bit more intensively, I've been > made aware of a number of shortcomings. > > The main gripe is from folks trying to launch a kdump kernel from > within an interrupt handler. If using EOImode==1, things work as > expected. If using EOImode==0 (such as in a guest), the secondary > kernel hangs as the previous interrupt hasn't been EOI'd, and the > active priority is still set. The first two patches are addressing > this situation for both GICv2 and GICv3 by reseting the APRs to their > default value. As a more general thing, if irqchip drivers have state that needs to be reset in their init code, can we live all this irqchip reset to the crashdump kernel, and kill machine_kexec_mask_interrupts() entirely? >>> >>> We could, once we know for sure that all the potential irqchips have >>> been fixed. Or we could just remove it immediately, and see what breaks. >> >> I would be very tempted to do the latter. > > Makes sense. Do we have any indicator that tells us that a particular irq > chip is missing something in the init code or do we have to rely on crash > reports? A way to work out what is potentially missing would be to make sure that whatever we're removing from machine_kexec_mask_interrupts, we can find it in the irqchip init code. Not an easy task, and certainly not perfect (patches 1 and 2 in this series have no equivalent in the kexec code). There is still another category of "reset" stuff that belongs to the teardown path, and that's for things that may have an impact on the secondary kernel. The case I have in mind is that of the GIC LPI pending tables. These are allocated to the GIC, which can write pending bits at any time. Think of it as a DMA engine. At the moment we enter the secondary kernel, we must make sure the GIC has already been shut down, as the table memory will be reallocated. For that particular case, I've started looking at some "reset" API that an irqchip to register with, and get called back on kexec/kdump. Not completely dissimilar to the shutdown method that some IOMMU drivers use to gracefully stop in the same circumstances. Thanks, M. -- Jazz is not dead. It just smells funny...
Re: [PATCH 0/3] irqchip: GIC kexec/kdump improvement and workarounds
On Wed, 14 Mar 2018, Mark Rutland wrote: > On Tue, Mar 13, 2018 at 06:35:07PM +, Marc Zyngier wrote: > > On 13/03/18 17:51, Mark Rutland wrote: > > > On Tue, Mar 13, 2018 at 05:21:00PM +, Marc Zyngier wrote: > > >> As kexec and kdump are getting used a bit more intensively, I've been > > >> made aware of a number of shortcomings. > > >> > > >> The main gripe is from folks trying to launch a kdump kernel from > > >> within an interrupt handler. If using EOImode==1, things work as > > >> expected. If using EOImode==0 (such as in a guest), the secondary > > >> kernel hangs as the previous interrupt hasn't been EOI'd, and the > > >> active priority is still set. The first two patches are addressing > > >> this situation for both GICv2 and GICv3 by reseting the APRs to their > > >> default value. > > > > > > As a more general thing, if irqchip drivers have state that needs to be > > > reset in their init code, can we live all this irqchip reset to the > > > crashdump kernel, and kill machine_kexec_mask_interrupts() entirely? > > > > We could, once we know for sure that all the potential irqchips have > > been fixed. Or we could just remove it immediately, and see what breaks. > > I would be very tempted to do the latter. Makes sense. Do we have any indicator that tells us that a particular irq chip is missing something in the init code or do we have to rely on crash reports? Thanks, tglx
Re: [PATCH 0/3] irqchip: GIC kexec/kdump improvement and workarounds
On Tue, Mar 13, 2018 at 06:35:07PM +, Marc Zyngier wrote: > On 13/03/18 17:51, Mark Rutland wrote: > > On Tue, Mar 13, 2018 at 05:21:00PM +, Marc Zyngier wrote: > >> As kexec and kdump are getting used a bit more intensively, I've been > >> made aware of a number of shortcomings. > >> > >> The main gripe is from folks trying to launch a kdump kernel from > >> within an interrupt handler. If using EOImode==1, things work as > >> expected. If using EOImode==0 (such as in a guest), the secondary > >> kernel hangs as the previous interrupt hasn't been EOI'd, and the > >> active priority is still set. The first two patches are addressing > >> this situation for both GICv2 and GICv3 by reseting the APRs to their > >> default value. > > > > As a more general thing, if irqchip drivers have state that needs to be > > reset in their init code, can we live all this irqchip reset to the > > crashdump kernel, and kill machine_kexec_mask_interrupts() entirely? > > We could, once we know for sure that all the potential irqchips have > been fixed. Or we could just remove it immediately, and see what breaks. I would be very tempted to do the latter. Thanks, Mark.
Re: [PATCH 0/3] irqchip: GIC kexec/kdump improvement and workarounds
On 13/03/18 17:51, Mark Rutland wrote: > On Tue, Mar 13, 2018 at 05:21:00PM +, Marc Zyngier wrote: >> As kexec and kdump are getting used a bit more intensively, I've been >> made aware of a number of shortcomings. >> >> The main gripe is from folks trying to launch a kdump kernel from >> within an interrupt handler. If using EOImode==1, things work as >> expected. If using EOImode==0 (such as in a guest), the secondary >> kernel hangs as the previous interrupt hasn't been EOI'd, and the >> active priority is still set. The first two patches are addressing >> this situation for both GICv2 and GICv3 by reseting the APRs to their >> default value. > > As a more general thing, if irqchip drivers have state that needs to be > reset in their init code, can we live all this irqchip reset to the > crashdump kernel, and kill machine_kexec_mask_interrupts() entirely? We could, once we know for sure that all the potential irqchips have been fixed. Or we could just remove it immediately, and see what breaks. > That would avoid some work (including pointer chasing on potentially > corrupt memory) in the kernel that crashed, making it more likely that > we get to the crashkernel intact... Seems perfectly sensible to me. M. -- Jazz is not dead. It just smells funny...
Re: [PATCH 0/3] irqchip: GIC kexec/kdump improvement and workarounds
On Tue, Mar 13, 2018 at 05:21:00PM +, Marc Zyngier wrote: > As kexec and kdump are getting used a bit more intensively, I've been > made aware of a number of shortcomings. > > The main gripe is from folks trying to launch a kdump kernel from > within an interrupt handler. If using EOImode==1, things work as > expected. If using EOImode==0 (such as in a guest), the secondary > kernel hangs as the previous interrupt hasn't been EOI'd, and the > active priority is still set. The first two patches are addressing > this situation for both GICv2 and GICv3 by reseting the APRs to their > default value. As a more general thing, if irqchip drivers have state that needs to be reset in their init code, can we live all this irqchip reset to the crashdump kernel, and kill machine_kexec_mask_interrupts() entirely? That would avoid some work (including pointer chasing on potentially corrupt memory) in the kernel that crashed, making it more likely that we get to the crashkernel intact... Thanks, Mark
[PATCH 0/3] irqchip: GIC kexec/kdump improvement and workarounds
As kexec and kdump are getting used a bit more intensively, I've been made aware of a number of shortcomings. The main gripe is from folks trying to launch a kdump kernel from within an interrupt handler. If using EOImode==1, things work as expected. If using EOImode==0 (such as in a guest), the secondary kernel hangs as the previous interrupt hasn't been EOI'd, and the active priority is still set. The first two patches are addressing this situation for both GICv2 and GICv3 by reseting the APRs to their default value. The last patch is introduced to workaround an annoying shortcoming on some GICv3 implementations, where LPIs cannot be disabled once they've been enabled. This completely kills the whole kexec story, as the secondary kernel will not be able to configure LPIs, and may experience random single bit corruption due to interrupts being made pending in the now reallocated pending tables. Fun! This is quite annoying in those (limited) situations where you'd like Linux to act as your bootloader. For this particular use case, we introduce a kernel command line option "irqchip.gicv3_nolpi=1", which will force the kernel to ignore LPIs altogether, leaving them intact to the secondary kernel to enjoy. This has proved to be quite useful on my Chromebook. I'd welcome any testing and reviewing. If nobody fundamentaly disagrees with this, I plan to get it merged in 4.17. Marc Zyngier (3): irqchip/gic-v2: Reset APRn registers at boot time irqchip/gic-v3: Reset APgRn registers at boot time irqchip/gic-v3: Allow LPIs to be disabled from the command line Documentation/admin-guide/kernel-parameters.txt | 8 + arch/arm/include/asm/arch_gicv3.h | 41 +++-- drivers/irqchip/irq-gic-v3.c| 33 +++- drivers/irqchip/irq-gic.c | 17 ++ 4 files changed, 82 insertions(+), 17 deletions(-) -- 2.14.2
